Data synchronization method, system and equipment based on embedded database and medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a data synchronization method, system and device based on an embedded database and a medium, and the method comprises the steps: directly obtaining a corresponding log record before writing as a synchronization data source when a transaction change occurs in the database; determining a synchronization priority according to the data type of the log record, and caching and packaging the time series data with the common priority in batches; for the to-be-synchronized data, determining a consistent snapshot based on the log serial number, and constructing a synchronous data unit; sending the information to a cloud; and receiving and executing a data merging instruction issued by the cloud to update local data. According to the invention, high-reliability data synchronization with lower delay, higher network and storage efficiency and zero interference to services is realized on the Internet of Things terminal with limited resources.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a data synchronization method, system, device, and medium based on an embedded database. Background Technology

[0002] In IoT applications, massive amounts of terminal devices generate mixed time-series and relational data that needs to be reliably synchronized to the cloud. Traditional synchronization methods typically convert database changes into independent data packets for transmission on resource-constrained embedded devices. This approach has significant drawbacks: synchronized data acquisition requires additional computational overhead, increasing CPU load; the uniform synchronization strategy cannot distinguish between critical configurations and massive amounts of sensor data, resulting in low network efficiency; the synchronization process may briefly block local data read / write operations on the device, affecting real-time performance; and the offline caching mechanism is independent of database storage management, leading to insufficient reliability. Therefore, there is an urgent need for a data synchronization method that deeply collaborates with the embedded database kernel, is highly efficient, and does not impact business operations. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a data synchronization method, system, device and medium based on an embedded database to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a data synchronization method based on an embedded database, comprising: In response to transactional changes in the data in the embedded database, the corresponding log record before writing is directly obtained as the data source to be synchronized. Based on the data types involved in the log records before writing, determine their synchronization priority and processing strategy; for time-series data determined to be of ordinary priority, cache them until the preset batch conditions are met, and then package them into a synchronization batch of data; For each log record or batch of data to be synchronized before writing, a consistent log sequence number snapshot is determined, and the corresponding synchronization data unit is constructed based on the snapshot. The synchronized data unit is sent to the cloud server; Receive and execute the data merging instruction issued by the cloud server to update local data.

[0005] In an optional implementation, when sending the synchronization data unit to the cloud server, the method further includes: If a network interruption causes a transmission failure, the corresponding pre-write log record will be written to a specific storage area reserved in the embedded database flash memory and managed by the database storage engine.

[0006] In an optional implementation, the storage engine of the embedded database is configured as follows: The log-as-data converged storage architecture is adopted, which physically mixes the log records before the transaction is written with the corresponding data pages and stores them in the same flash erase block; Fixed-size micropages are used as the smallest unit for data organization, writing, and management.

[0007] In an optional implementation, the synchronization priority and processing strategy are determined based on the data type involved in the pre-write log record, including: Parse the pre-write log records to identify the data table type to which they belong; If the log record before writing belongs to a relational data table, it is determined to be of high priority and immediate synchronization is triggered; If the log record before writing belongs to a time-series data table, it is determined to be of normal priority and cached. When the cached log records belonging to the time-series data table before writing meet the preset batch conditions, the cached records are packaged into a synchronous batch for processing. The preset batch conditions include at least one of the following: reaching a preset time window threshold; the accumulated amount of data to be synchronized reaching a preset data volume threshold; and the accumulated number of log records before writing reaching a preset quantity threshold.

[0008] In an optional implementation, for each pre-write log record or synchronization batch data to be synchronized, a consistent log sequence number snapshot is determined, and a corresponding synchronization data unit is constructed based on the snapshot, including: Record the current log sequence number of the database when the construction of the synchronization data unit begins, as the snapshot version number of the synchronization data unit; Filter out log records that have not yet been synchronized and whose log sequence number is less than or equal to the snapshot version number. The original content of one or more selected log records before writing is directly used as the load data of the synchronization data unit; Generate header information containing the snapshot version number, and combine it with the load data to form a complete synchronization data unit; Based on the type of the load data, select the corresponding compression algorithm to compress the complete synchronization data unit; During the construction of the synchronous data unit, the database continues to process new transactions and generate higher log sequence numbers.

[0009] In an optional implementation, after sending the synchronization data unit to the cloud server, the method further includes: Receive synchronization data units from the device, the synchronization data units containing log records before writing filtered based on the device log sequence number snapshot; Multiple synchronized data units from the same device within a predetermined time window are merged and sorted according to their log sequence numbers; for each data change after sorting, the hash value of the key field of the data row involved is calculated; the calculated hash value is compared with the current hash value of the corresponding data row stored in the cloud; if they are inconsistent, it is determined that there is a data conflict. If a data conflict is detected, a data merging instruction frame is generated using a cloud-first data strategy. The data update content in the data merging instruction frame is designed to ensure that the data on the device side is consistent with the data on the cloud side. The data merging instruction frame is sent to the corresponding device.

[0010] In an optional implementation, receiving and executing a data merging instruction issued by the cloud server to update local data includes: Receive a data merging instruction frame sent by the cloud server, the instruction frame containing a unique instruction identifier; Check if the instruction identifier exists in the locally maintained set of processed instruction identifiers; If it does not exist, the instruction identifier is added to the set, and the data update operation defined in the data merging instruction frame is executed to modify the data snapshot in the local database; If it exists, discard the data merging instruction frame.

[0011] Secondly, the present invention provides a data synchronization system based on an embedded database, comprising: The synchronization triggering module is used to respond to transactional changes in data in the embedded database by directly obtaining the corresponding log record before writing as the data source to be synchronized. The strategy determination module is used to determine the synchronization priority and processing strategy based on the data types involved in the log records before writing; for time-series data determined to be of ordinary priority, it is cached until the preset batch conditions are met, and then packaged into a synchronization batch of data. The unit construction module is used to determine a consistent log sequence number snapshot for each pre-write log record or synchronization batch data to be synchronized, and to construct the corresponding synchronization data unit based on the snapshot. The unit sending module is used to send the synchronization data unit to the cloud server; The instruction execution module is used to receive and execute data merging instructions issued by the cloud server to update local data.

[0012] Thirdly, a device is provided, comprising: Memory, used to store data synchronization programs based on embedded databases; A processor is configured to implement the steps of the embedded database-based data synchronization method provided in the first aspect when executing the embedded database-based data synchronization program.

[0013] Fourthly, a computer-readable medium is provided, on which a data synchronization program based on an embedded database is stored, wherein when the data synchronization program based on the embedded database is executed by a processor, the steps of the data synchronization method based on an embedded database provided in the first aspect are implemented.

[0014] The beneficial effects of this invention are as follows: The data synchronization method, system, device, and medium based on embedded databases provided by this invention eliminate the overhead of computational differences and significantly reduce CPU load by directly using the pre-write log of the database kernel as the synchronization source. The intelligent hierarchical synchronization strategy based on data types reduces the number of handshakes under narrowband networks, improving transmission efficiency. The use of the database MVCC mechanism to achieve lock-free snapshot construction ensures that the synchronization process does not affect the device's real-time data acquisition and querying. Incorporating offline data awaiting transmission into the database micro-page management system significantly improves offline cache reliability. Ultimately, highly reliable data synchronization with lower synchronization latency, higher network and storage efficiency, and zero interference with business operations is achieved on resource-constrained IoT terminals. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0017] Figure 2 This is another illustrative flowchart of a method according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of a network disconnection resumption queue according to an embodiment of the present invention.

[0019] Figure 4 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0023] The data synchronization method based on an embedded database provided in this embodiment of the invention is executed by a computer device, and correspondingly, the data synchronization system based on an embedded database runs in the computer device.

[0024] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a data synchronization system based on an embedded database. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0025] like Figure 1 As shown, the method includes: S1. In response to transactional changes in the data in the embedded database, directly obtain the corresponding log record before writing as the data source to be synchronized; S2. Based on the data types involved in the log records before writing, determine their synchronization priority and processing strategy; for time-series data determined to be of ordinary priority, cache them until the preset batch conditions are met, and then package them into a synchronization batch of data; S3. For each log record or batch of data to be synchronized before writing, determine a consistent log sequence number snapshot, and construct the corresponding synchronization data unit based on the snapshot; S4. Send the synchronized data unit to the cloud server; S5. Receive and execute the data merging instruction issued by the cloud server to update the local data.

[0026] Please refer to Figure 2 The present invention will be described in detail below.

[0027] 1. System Overall Architecture The system consists of three horizontal and two vertical modules: the three horizontal modules are an ultra-lightweight storage engine, a dual-mode storage layer, and an SQL / TSML parser; the two vertical modules are a log-data synchronization bus and a flash wear leveling adapter. All code is statically linked into a single ELF file, with the trimmed text segment ≤384 kB; during startup, only the first 4 kB of pages are mapped, and subsequent segments are loaded based on page faults, resulting in a cold start time of ≤18 ms until the first SQL statement returns.

[0028] 2. Ultra-lightweight storage engine 2.1 Page-Log Fusion Format: The traditional 4 kB page is compressed into a 512 B "micro page", with a micro page header of only 6 B; WAL and data pages share the same 64 kB erase block, writing data is writing to the log, eliminating double writing.

[0029] 2.2 Lock-free concurrency model: Single writer single log sequence number (LSN) is adopted. Readers read through snapshot version number MVCC without row locks; transaction commit only appends 8 B commit records to achieve "zero disk flush" commit.

[0030] 2.3 Memory layout: B+-tree nodes and log buffers use the same 32 kB static RAM, which is dynamically partitioned using the buddy algorithm to avoid heap fragmentation; the cursor object pool is reused, with a maximum of 16 objects, ensuring that the peak RAM is ≤48 kB.

[0031] 3. Time-series-relational dual-mode storage layer 3.1 Time Series Table: Column differential + XOR compression is used, with a floating-point value compression rate ≤ 0.5 B / point within the same acquisition period; an implicit time index is automatically created, with the primary key being...<device_id, timestamp> The insertion complexity is O(1).

[0032] 3.2 Relationship Tables: Supports standard INTEGER / REAL / TEXT / BLOB; foreign keys and CHECK constraints can be trimmed, reducing code size by 42 kB when disabled.

[0033] 3.3 Cross-mode JOIN: The timing table and relational table are merged in one go according to the time range by the "time window push-down" operator, avoiding secondary scanning of flash memory.

[0034] 4. Differential compression transmission protocol 4.1 Frame format: Uses a 3-B header (frame length + compression flag + CRC5), and the payload adopts LZ4-HC level compression; additional XOR+Leading-Zero encoding is applied to the sensor floating-point array, which improves the compression rate by 30%.

[0035] 4.2 Uplink process: After the local transaction is committed, the synchronization bus immediately encapsulates the LSN interval into a DIFF frame and sends it to the cloud topic / diff / {device_id} via MQTT-TLS; QoS=1, and after 3 retransmissions on failure, it is written to the local "disconnection queue".

[0036] 4.3 Downlink process: The cloud conflict resolution result is sent to / merge / {device_id} in the form of a MERGE frame; after receiving the MERGE frame, the edge device replays it in the lock-free state machine, updates the local snapshot, and sends back an ACK; if the ACK is lost, the cloud retransmits it after 5 seconds to ensure at least one semantic response.

[0037] 5. End-to-Cloud Lock-Free Merging Algorithm 5.1 Define the window: use 1 second as the time bucket, and the transactions in the bucket are monotonically increased by LSN; sort by bucket number in the cloud to eliminate out-of-order.

[0038] 5.2 Conflict Detection: Calculate the 64-bit hash only for the primary key or unique index column. If the hashes in the cloud and edge are inconsistent, the "cloud priority" strategy is adopted to write back.

[0039] 5.3 Replay Buffer: A 64-line circular MERGE buffer is maintained at the edge. Duplicate frames are discarded upon receipt to ensure idempotency. No write lock is acquired during the replay process. The application status is determined by comparing the LSN.

[0040] 6. Disconnection resume mechanism 6.1 Local Queue: When the network is disconnected, DIFF frames are written to a NOR Flash circular queue, with the head and tail of the queue each occupying 4 bytes. The queue can be recovered after power failure. The queue length is configurable, with a default of 512 frames, approximately 256 kB.

[0041] 6.2 Fast Retransmission: After the network is restored, the synchronization bus delivers data in batches according to LSN order. The delivery rate is adaptively adjusted by the ACK interval returned by the MQTT agent to prevent sudden increases in cellular module current.

[0042] 6.3 Consistency Check: After the batch delivery is completed, a CHECKSUM frame is sent. The cloud returns the latest global LSN, and the edge compares it with the local maximum LSN. If the difference is not zero, it is retransmitted to ensure eventual consistency.

[0043] 7. Startup and Upgrade 7.1 Image Layout: The firmware is divided into three areas: Bootloader, database kernel, and metadata page; during the upgrade, only the kernel area is replaced, while the metadata page remains backward compatible, and the upgrade time is less than 1 second.

[0044] 7.2 Quick self-test: Page check and verification are completed within 4 ms after power-on; if bit flipping is detected, a spare micropage is used to replace it to ensure that the database is readable and writable.

[0045] Actual resource consumption was tested on an STM32L4R5 (120 MHz, 640 kB Flash, 520 kB RAM) platform: Resident ROM 384 kB, RAM peak 48 kB; Continuous insertion of 1000 records per second, CPU usage at 18%; The 512-frame network outage queue can cache 8 hours and 20 minutes of data (1-second cycle). When connected to the Alibaba Cloud IoT platform, the average end-to-end synchronization latency is 2.3 seconds, and the 99th percentile is 2.9 seconds.

[0046] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0047] S101. Storage Engine Configuration for Embedded Databases The embedded database in this embodiment runs on a resource-constrained IoT terminal MCU. Its storage engine is deeply customized for the characteristics of NOR Flash, with the core being a converged storage architecture of "logs as data" and "micro-page" management.

[0048] Converged storage architecture: This database abandons the traditional design of storing the Write-Ahead Log (WAL) separately from the actual data pages. A physical flash erase block (e.g., 64KB in size) is formatted as a contiguous sequence of micropages. When a transaction needs to modify data, the storage engine does not write to the log separately and then update the data page in another location. Instead, it directly writes the modified content (including metadata and the changed data) as a complete WAL record, sequentially writing it to the next available micropage of the current erase block. This WAL record itself contains the latest state of the data, achieving physical fusion of the log and the data. This design eliminates two flash writes to the same data, reducing the write amplification factor to close to 1 and significantly extending the flash lifespan.

[0049] Micropage Management Unit: The basic unit for managing flash memory in databases is the fixed-size micropage, such as 512 bytes. This size is much smaller than the 4KB page of a traditional database, and its advantages are: Hardware compatibility: Better fits the limited RAM capacity (typically tens of KB) and Flash programming unit of the MCU.

[0050] Reduced waste: Fine-grained writing reduces storage waste caused by unused space within a page.

[0051] Fine-grained control: Provides a natural and manageable storage unit for subsequent synchronous data caching.

[0052] The storage engine maintains the allocation state of micropages and manages the cyclic use of erase blocks through a wear leveling algorithm.

[0053] S102. Direct Acquisition of Synchronous Data Source The synchronization module is a tightly integrated component of the database kernel, rather than an external, independent process. Its process for obtaining the synchronization data source is as follows: Triggering and Capturing: When an application submits a transaction via the database API (e.g., inserting a sensor record), the storage engine internally generates the corresponding WAL record and prepares to write it to the micropage. At this moment, the storage engine proactively sends this soon-to-be-persistent WAL record (in-memory format) to the acquisition interface of the synchronization module via an internal callback interface. This process occurs almost in real-time, before the data is written to flash memory.

[0054] Zero-overhead principle: Since the synchronization module receives WAL records natively generated by the storage engine without any secondary processing, it eliminates the entire process of comparing the data before and after the data state, calculating and encapsulating the differences (DIFF) that is required in traditional methods. This achieves "zero computational overhead" for synchronous data acquisition.

[0055] Data content: The obtained WAL record contains all the information needed to replay the data changes, such as table identifier, operation type (insert / update / delete), row primary key, and the changed column data values.

[0056] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0057] S201. Data Table Type Analysis The WAL record contains a unique identifier (Table ID) for the target table. The synchronization module internally maintains a metadata cache table that records the table type (such as "relational table" or "time-series table") and other attributes corresponding to each Table ID. When a WAL record is received, the decision engine can quickly identify the table type to which the record belongs by querying this cache table. This process is completed in memory with extremely low overhead.

[0058] S202. High-priority data processing (relational data) If the parsed result is a relational data table (e.g., the device_config table for storage device configuration parameters), the WAL record is marked as high priority. The decision engine immediately places it into a high-priority send queue. This queue has a "send when full" characteristic; once data is enqueued, a network send thread is immediately triggered to encapsulate it into an independent synchronous data unit and attempt to send it, aiming to synchronize critical metadata changes to the cloud with minimal latency.

[0059] S203. Normal Priority Data Processing and Batch Packaging of Time Series Data If the parsing result is a time-series data table (e.g., the sensor_temp table storing temperature readings), the record is marked as normal priority and temporarily stored in a batch buffer.

[0060] Cache management: The batch cache is implemented in memory as a circular buffer to store multiple time-series WAL records in an orderly manner.

[0061] Batch condition monitoring: A separate condition checker continuously monitors the cache area to check whether any of the following preset batch trigger conditions are met: Time condition: The time accumulated since the last batch was sent has reached the window threshold of 1 second.

[0062] Data volume condition: The total size of the raw data of all WAL records in the buffer has reached 32KB (approximately 64 micropages of 512 bytes each).

[0063] Quantity condition: The number of accumulated WAL records in the cache has reached 100.

[0064] Batch Packaging: Once any of the above conditions are met, the condition checker triggers the packaging operation. The decision engine extracts all time-series WAL records in the current buffer as a whole, preparing to construct a synchronous batch of data. Subsequently, the buffer is cleared to receive new records.

[0065] This tiered strategy ensures the real-time nature of critical configurations while aggregating massive amounts of sensor data with tolerable short delays into larger data packets for transmission. This significantly reduces the number of connections and transmissions in low-bandwidth, high-power cellular networks (such as NB-IoT), thereby optimizing overall network efficiency and device power consumption.

[0066] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0067] S301. Anchored snapshot version number When a synchronized data unit needs to be built (whether triggered by a high-priority record or met by batch conditions), the build engine first atomically queries the database storage engine to obtain the current global log sequence number (LSN), for example, LSN=1050. This LSN value is recorded as the snapshot version number (Snapshot LSN) of the data unit, representing the anchor point in time for data consistency.

[0068] S302. Filtering data based on snapshots The build engine then filters from the pool of pending WAL records (including high-priority queues or newly packaged time-series batches). The filtering logic is as follows: only records whose own LSN is less than or equal to the snapshot version number (i.e., ≤1050) and whose "synchronized" status is marked as "false" are selected. This step is quickly accomplished using a lightweight hash table with LSN as the key. All selected records logically constitute a consistent data snapshot at LSN=1050.

[0069] S303. Encapsulate raw log data The engine constructs all selected WAL records, concatenating them in their original binary format order, and uses this as the payload for the data units to be sent. This process does not involve any decoding or reassembly, preserving the most compact form of the data.

[0070] S304. Generate Synchronization Data Unit The engine creates a concise frame header, whose key fields must include the snapshot version number (Snapshot LSN: 1050) obtained in step S301, and may also include metadata such as payload length and frame type. This frame header is then concatenated with the payload data generated in step S303 to form a complete, uncompressed synchronization data unit.

[0071] S305. Dynamic Compression Processing Before sending, the compression module selects the optimal compression strategy based on the characteristics of the payload data: Dedicated floating-point compression: If the data unit consists purely of a batch of continuous floating-point sensor data from the timing table, then a dedicated floating-point compression algorithm based on differential, XOR and leading zero coding can be used to achieve an extremely high compression ratio (as described in the documentation, <0.5 bytes per point on average).

[0072] General streaming compression: If the data unit is of mixed type (such as including relation table changes), the general LZ4-HC algorithm is used for compression, achieving a balance between compression speed and ratio. The compressed data and the frame header (the frame header itself is not compressed) are finally combined to form the packet body to be sent.

[0073] Lock-free concurrency guarantee: Crucially, throughout the execution of steps S301 to S305 (i.e., from obtaining the snapshot LSN to completing packet construction), the database's main thread continues to run normally, freely committing new transactions (e.g., generating new WAL records with LSN=1051, 1052…). The synchronization module always processes a snapshot of data at a specific point in the past (LSN=1050), therefore requiring no read or write locks on the database, achieving true zero business disruption.

[0074] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0075] S401. Data Transmission and Confirmation The synchronous sending thread publishes the constructed synchronous data unit to a specific topic in the cloud (e.g., / diff / {device_id}) via the MQTT protocol (based on TLS secure transmission), and sets the Quality of Service (QoS) to 1 (at least once). After sending, the thread waits for a publish acknowledgment packet (PUBACK) returned by the MQTT broker. If an acknowledgment is received within the timeout period (e.g., 3 seconds), the sending is considered successful, and the "synchronized" status of the corresponding WAL record is marked as "true".

[0076] S402. Network Interruption Detection and Persistent Storage If transmission fails (e.g., no PUBACK is received within a timeout period, or the underlying network interface reports a disconnection), the synchronization module immediately initiates the network disconnection handling procedure: Identification data: Mark the original WAL record (or record set) corresponding to the synchronization data unit that failed to be sent this time as "requires persistent retransmission".

[0077] Kernel storage: The synchronization module calls a dedicated storage engine interface wal_append_to_pending_area() to submit these marked WAL records to the database kernel.

[0078] Writing to the "Pending Synchronization Area": ​​After receiving these WAL records, the database storage engine does not write them to the regular data area, but instead writes them to a special logical area on flash memory that is pre-allocated and formatted—the "Pending Synchronization Persistence Area." This area: Physical structure: It consists of continuous, fixed micropages (e.g., a total of 256 micropages, with a capacity of 128KB).

[0079] Management Mode: It is entirely managed by the database's micropage allocator and wear leveling algorithm, existing as a special "table" or "log stream". Data is sequentially filled into micropages in an append-only manner, and after a page is full, it is overwritten from the head in a loop (after ensuring that older data has been successfully retransmitted).

[0080] Power loss safety: By utilizing the micro-page write mechanism of database transactions, each write ensures the integrity and recoverability of data after a power outage, avoiding the risk of metadata corruption caused by power loss in self-built queues.

[0081] S403. Network Recovery and Intelligent Retransmission After the device network monitoring module detects network recovery, it notifies the synchronization module. The queue structure for resuming data transmission after a network outage is as follows: Figure 3 As shown.

[0082] Sequential Read: The synchronization module calls the storage engine interface wal_read_from_pending_area() to read the previously saved WAL records from the "persistent area to be synchronized" in ascending order of LSN.

[0083] Reconstruction and Transmission: Using these WAL records, go through steps S302 to S305 (filtering, constructing, and compressing) again to generate new synchronization data units, and retransmit them through step S501.

[0084] Adaptive rate: The retransmission rate is dynamically adjusted based on the MQTT ACK response time to avoid congestion or current surges in weak network environments.

[0085] Area reclamation: Once a WAL record is confirmed to be received by the cloud, the micropages it occupies in the "Pending Synchronization Persistent Area" can be marked as reclaimable space.

[0086] This mechanism transforms the volatile communication queue into part of non-volatile storage managed by a highly reliable database kernel, greatly improving the robustness of the synchronization system in harsh network environments and unexpected power outages.

[0087] The cloud server acts as the coordinator and arbitrator for data synchronization. Its core responsibility is to reliably receive device data and efficiently resolve potential data conflicts. The cloud synchronization service implementation process in this embodiment is as follows: 1. Data reception and preprocessing The cloud service receives synchronization data units from the device by subscribing to the MQTT topic / diff / {device_id}. Upon receiving the data, the service first verifies and decompresses it (if applicable), and then parses out key metadata from the frame header, particularly the device's snapshot version number (LSN) and the list of WAL records. Each WAL record specifies its target table, row key, and changes.

[0088] 2. Merge Sort Based on Time Buckets To address the issue of out-of-order data packets caused by network jitter, cloud services introduce a "time bucket" mechanism for preprocessing: Merge: The service uses a 1-second time window (time bucket) to temporarily store all synchronized data units from the same device within the same time period into the same processing batch.

[0089] Sorting: Within a time bucket, the service globally sorts all data change operations to be applied based on the Snapshot LSN carried by each synchronized data unit and the LSN of the WAL record within that unit, resulting in a strictly increasing sequence of operations by LSN. This step eliminates out-of-order operations and provides a consistent operational baseline for subsequent conflict detection.

[0090] 3. Lightweight hash collision detection For the sorted sequence of operations, the cloud does not directly compare it with the complete row data in the database, but uses an efficient hash comparison method for collision detection: Calculate the device-side change hash: For each data update or insert operation, the cloud service calculates a 64-bit hash value (e.g., using the CityHash64 algorithm) based on the row primary key (or unique index column) in the WAL record. This hash value represents the "expected state identifier" of the data row that the device is attempting to modify.

[0091] Get the current hash in the cloud: The cloud service queries the current primary key hash value of the corresponding data row from its own state database.

[0092] Comparison and Conflict Determination: The calculated "expected hash" on the device side is compared with the "current hash" obtained from the cloud. If they match, it indicates that the data in this row has not been updated or modified by other sources in the cloud since the device snapshot, and there is no conflict. If they do not match, it is determined to be a conflict, meaning that the data in this row was updated in the cloud before the data synchronization on the device arrived (possibly due to network packets from other systems or from a later snapshot of the same device that arrived earlier).

[0093] 4. Conflict resolution and instruction generation A simplified arbitration strategy prioritizing cloud data is adopted. If there are no conflicts: the cloud service applies the changes in the device's WAL record to the cloud master database and prepares a simple confirmation command frame.

[0094] If a conflict is detected: the cloud service will use its currently stored data as the standard. It will generate a special MERGE Frame. The payload of this frame contains the primary key of the conflicting row and the complete data of that row currently in the cloud (or a reverse operation instruction). Its purpose is to explicitly instruct the device to overwrite / update its local data to a state consistent with the cloud. The header of the instruction frame contains a globally unique command ID and the device's original Snapshot LSN, among other information.

[0095] 5. Issuance of instructions Finally, the cloud service publishes the generated merge command frame (or acknowledgment frame) to the specific topic subscribed to by the device (e.g., / merge / {device_id}) via the MQTT protocol, with the service quality also set to QoS 1 to ensure reliable command delivery. The device will then complete the synchronization loop accordingly.

[0096] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0097] S501. Command Frame Reception and Parsing The MQTT client on the device subscribes to the topic ` / merge / {device_id}`. When it receives a data packet from the cloud, the instruction processing module first performs integrity verification and parsing. The parsed data merging instruction frame typically contains: Frame header: Embedded unique command identifier (Command ID), such as a monotonically increasing sequence number or UUID generated by the cloud.

[0098] Load: Contains specific operational instructions. This could be: Confirmation command: Contains only the command ID and the original Snapshot LSN, indicating that the change has been successfully applied in the cloud.

[0099] Merge / correct instructions: In addition to the instruction ID, they also include the target table name, row primary key, and the complete data row (or explicit update opcode) used to update the device-side data snapshot.

[0100] S502. Idempotency Check To prevent the same instruction from being executed repeatedly due to network retransmission, the instruction processing module maintains a circular buffer (e.g., with a capacity of 64 records) in memory to identify processed instructions.

[0101] Query operation: After the instruction ID (such as CMD_12345) is parsed, the processing module immediately performs a fast search in the circular buffer.

[0102] Decision and Branch: If it exists: This indicates that the instruction has already been processed. The processing module will directly discard the current instruction frame without performing any further operations and return a "duplicate instruction ignored" status to the application layer, ending the process.

[0103] If it does not exist: proceed with the execution process.

[0104] S503. Lock-free data snapshot update For new instructions that need to be executed, the processing module executes them sequentially: Status recording: First, append the instruction ID (CMD_12345) to the circular buffer to mark it as "processed".

[0105] Data merging operation: Obtaining the transaction context: The instruction processing module starts a short-lived transaction in the database kernel.

[0106] Application Change: Based on the instruction payload, directly update or overwrite the target data row in the database. This operation is performed internally within the database and is managed by the MVCC mechanism.

[0107] Generate a compensation log: To maintain the continuity of the device-side WAL log stream and provide a basis for possible subsequent synchronization, the database will generate a special WAL record for this "cloud-driven" change. This record is marked as "originating from cloud merge," and its LSN will increment normally, but it will be recognized by the synchronization module and usually excluded from subsequent uplink synchronization data sources to avoid update loops.

[0108] Commit Transaction: Commit the transaction after completing the data modification. Since this operation only modifies a small amount of data explicitly specified by the cloud command, and the database kernel's MVCC mechanism ensures read-write isolation, this process will not block other data acquisition or query transactions that are currently being performed on the device.

[0109] S504. Final State Synchronization Upon successful execution, the instruction processing module can send a brief confirmation receipt to the cloud (optional). Simultaneously, the local database status is synchronized with the cloud, preparing for the next data synchronization cycle.

[0110] This implementation achieves strong idempotency guarantees through instruction ID caching, ensuring operational security under unreliable network conditions; and by performing updates within database transactions, it seamlessly and securely integrates cloud instructions into the device's local data management system, ensuring data consistency and system robustness.

[0111] In some embodiments, the embedded database-based data synchronization system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the embedded database-based data synchronization system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Data synchronization functionality based on an embedded database.

[0112] In this embodiment, the data synchronization system based on the embedded database can be divided into multiple functional modules according to the functions it performs, such as... Figure 4 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0113] The synchronization triggering module is used to respond to transactional changes in data in the embedded database by directly obtaining the corresponding log record before writing as the data source to be synchronized. The strategy determination module is used to determine the synchronization priority and processing strategy based on the data types involved in the log records before writing; for time-series data determined to be of ordinary priority, it is cached until the preset batch conditions are met, and then packaged into a synchronization batch of data. The unit construction module is used to determine a consistent log sequence number snapshot for each pre-write log record or synchronization batch data to be synchronized, and to construct the corresponding synchronization data unit based on the snapshot. The unit sending module is used to send the synchronization data unit to the cloud server; The instruction execution module is used to receive and execute data merging instructions issued by the cloud server to update local data.

[0114] Figure 5The data synchronization method based on an embedded database provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0115] The device 500 may include a processor 510, a memory 520, and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0116] The memory 520 can be used to store execution instructions of the processor 510. The memory 520 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the device 500 is able to perform some or all of the steps in the above method embodiments.

[0117] The processor 510 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 520, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 510 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0118] The communication unit 530 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0119] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0120] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0121] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0122] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0123] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0125] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A method for data synchronization based on an embedded database, characterized in that, The method comprises: in response to transactional changes in data in the embedded database, directly obtaining corresponding pre-write log records as a data source to be synchronized; determining the synchronization priority and processing strategy of the pre-write log records according to the type of data involved in the pre-write log records; for time series data determined as ordinary priority, caching until a preset batch condition is met, and then packaging as a synchronization batch data; for each pre-write log record or synchronization batch data to be synchronized, determining a consistent log sequence number snapshot, and constructing a corresponding synchronization data unit according to the snapshot; sending the synchronization data unit to the cloud server; receiving and executing the data merging instructions issued by the cloud server to update the local data.

2. The method of claim 1, wherein, When sending the synchronization data unit to the cloud server, the method further comprises: if network interruption causes sending failure, write the corresponding pre-write log record into a specific storage area reserved in the embedded database flash memory and managed by the database storage engine.

3. The method of claim 1, wherein, The storage engine of the embedded database is configured to: adopt a log-data fusion storage architecture to physically mix the pre-write log records of transactions and corresponding data pages in the same flash erase block; and use fixed-size micro-pages as the smallest unit of data organization, writing and management.

4. The method of claim 1, wherein, determining the synchronization priority and processing strategy of the pre-write log records according to the type of data involved in the pre-write log records comprises: parsing the pre-write log record to identify the type of data table it belongs to; if the pre-write log record belongs to a relational data table, it is determined as high priority and immediate synchronization is triggered; if the pre-write log record belongs to a time series data table, it is determined as ordinary priority and is cached; when the cached pre-write log records belonging to the time series data table meet a preset batch condition, the cached records are packaged as a synchronization batch for processing; the preset batch condition includes at least one of the following: reaching a preset time window threshold; the accumulated amount of data to be synchronized reaching a preset data amount threshold; the number of pre-write log records reaching a preset number threshold.

5. The method of claim 1, wherein, for each pre-write log record or synchronization batch data to be synchronized, determining a consistent log sequence number snapshot, and constructing a corresponding synchronization data unit according to the snapshot comprises: recording the current log sequence number of the database when starting to construct the synchronization data unit as the snapshot version number of the synchronization data unit; filtering out pre-write log records with log sequence numbers less than or equal to the snapshot version number and not yet synchronized; directly using the original content of one or more filtered pre-write log records as the payload data of the synchronization data unit; generating header information containing the snapshot version number and combining it with the payload data to form a complete synchronization data unit; according to the type of the payload data, selecting a corresponding compression algorithm to compress the complete synchronization data unit; wherein, during the construction of the synchronization data unit, the database continues to process new transactions and generates higher log sequence numbers.

6. The method of claim 1, wherein, After sending the synchronization data unit to the cloud server, the method further comprises: receiving the synchronization data unit from the device end, wherein the synchronization data unit contains the write-ahead log record filtered based on the device end log sequence number snapshot; merging and sorting the multiple synchronization data units from the same device end within a predetermined time window according to the log sequence numbers carried by the synchronization data units; calculating the key field hash value of the data row involved for each data change after sorting; comparing the calculated hash value with the current hash value of the corresponding data row stored in the cloud; if the values are inconsistent, it is determined that there is a data conflict; if a data conflict is detected, a data merge instruction frame is generated using the cloud data priority strategy, wherein the data update content in the data merge instruction frame is used to make the device end data consistent with the cloud data; sending the data merge instruction frame to the corresponding device end.

7. The method of claim 1, wherein, receiving and executing the data merge instruction issued by the cloud server to update the local data, comprising: receiving the data merge instruction frame issued by the cloud server, wherein the instruction frame contains a unique instruction identifier; checking whether the instruction identifier exists in the set of processed instruction identifiers maintained locally; if not, adding the instruction identifier to the set and executing the data update operation defined in the data merge instruction frame to modify the data snapshot in the local database; if so, discarding the data merge instruction frame.

8. An embedded database based data synchronization system, characterized by, comprising: a synchronization trigger module for directly obtaining the corresponding write-ahead log record as a synchronization data source in response to the transactional change of data in the embedded database; a strategy determination module for determining the synchronization priority and processing strategy of the write-ahead log record according to the data type involved in the write-ahead log record; for time series data determined as ordinary priority, the write-ahead log record is cached until the preset batch condition is met, and then packaged as a synchronization batch data; a unit construction module for determining a consistent log sequence number snapshot for each write-ahead log record or synchronization batch data to be synchronized, and constructing a corresponding synchronization data unit according to the snapshot; a unit sending module for sending the synchronization data unit to the cloud server; an instruction execution module for receiving and executing the data merge instruction issued by the cloud server to update the local data.

9. An embedded database-based data synchronization apparatus, characterized by comprising: comprising: a memory for storing an embedded database-based data synchronization program; a processor for implementing the steps of the embedded database-based data synchronization method when executing the embedded database-based data synchronization program.

10. A computer readable medium having stored thereon a computer program, characterized in that, The readable medium stores an embedded database-based data synchronization program, and the embedded database-based data synchronization program is executed by the processor to implement the steps of the embedded database-based data synchronization method according to any one of claims 1-7.

Images