A data updating method and device, electronic equipment and storage medium

Abstract

The application provides a data updating method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a batch updating task and generating an updating plan in combination with a current lake table snapshot; determining the storage medium and path of the updated file based on a preset storage routing strategy; writing the updated file into a Staging temporary storage path of the corresponding medium and completing integrity and conflict double checking; and formally submitting the file and generating a new lake table snapshot after the checking is passed. According to the embodiment of the application, the file storage medium allocation can be uniformly controlled in the batch updating process, and the abnormalities such as file defects and concurrent updating conflicts can be intercepted in advance, so that the updating operation has atomicity, the inconsistent data problem that part of the medium is successfully written and part of the medium fails to be written is avoided, the data updating consistency is improved, and the integrity of the lake table snapshot version and the data reading and writing accuracy are stably maintained.

Description

Technical Field

[0001] This invention relates to the field of data update technology, and in particular to a data update method, apparatus, electronic device, and storage medium. Background Technology

[0002] Modern data lakes generally adopt a lake-table multi-media hierarchical storage architecture, relying on open lake-table formats such as Iceberg, Hudi, and Delta Lake to uniformly manage metadata. Based on routing rules such as hot / cold data, partitions, and tenants, hot data is stored on high-speed SSD media, warm data is stored on ordinary mechanical disks, and cold archived data is moved to low-cost object storage. Data files across media are associated by a unified metadata catalog and manifest file, balancing query performance and storage costs, and supporting data update operations such as stream and batch writing, partitioned hierarchical migration, etc.

[0003] Existing multi-media update mechanisms have significant consistency defects: Current update processes mostly use the completion of single-media file writing as the visibility boundary. When writing across media or migrating between cold and hot media, there is a lack of global atomic coordination, which can easily lead to half-updated data where writing to one type of media is successful while writing to other media is interrupted. At the same time, the metadata Manifest and the commit sequence of multi-media data files cannot be fully aligned. Concurrent updates and media failures can easily lead to problems such as mismatch between metadata and physical files and cross-media snapshot version disorder, which undermine transaction atomicity and global data consistency. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data update method, apparatus, electronic device, and storage medium to improve data update consistency.

[0005] According to one aspect of the present invention, a data update method is provided, applied to a cross-cloud storage medium, wherein the cross-cloud storage medium includes multiple storage media, the method comprising: Obtain target batch update tasks, wherein the target batch update tasks include target batch update files and target update types for each target batch update file; Based on the target batch update task and the current lake table snapshot of the data lake, a target update plan is generated. The target update plan includes at least the target update partition involved in the target batch update task and the original storage medium where the target batch update file is located. Based on the target batch update task and the preset storage routing strategy, the target storage medium and / or target storage path of each target updated file are determined; wherein, the target updated file is the file obtained by updating the target batch update file according to the target update type. The target updated file is stored in a pre-defined staging path on the target storage medium for the target updated file, and the target updated file is verified, including at least integrity verification and conflict detection. If the verification passes, the target updated file is submitted from the Staging path to the target storage medium, and a corresponding updated lake table snapshot is generated.

[0006] According to another aspect of the present invention, a data update apparatus is provided, applied to a cross-cloud storage medium, the cross-cloud storage medium including multiple storage media, the apparatus comprising: The acquisition module is used to acquire target batch update tasks, wherein the target batch update tasks include target batch update files and target update types for each target batch update file; The generation module is used to generate a target update plan based on the target batch update task and the current lake table snapshot of the data lake. The target update plan includes at least the target update partition involved in the target batch update task, the original storage medium where the target batch update file is located, and the target updated file. The target updated file is a file obtained by updating the target batch update file according to the target update type. The determination module is used to determine the target storage medium for each of the target updated files based on the target update plan and the preset storage routing strategy. The staging module is used to store the target updated file to a pre-defined staging path for the target storage medium of the target updated file, and to verify the target updated file, the verification including at least integrity verification and conflict detection. The submission module is used to submit the target updated file from the staging path to the target storage medium and generate a corresponding updated lake table snapshot if the verification passes.

[0007] According to another aspect of the present invention, an electronic device is provided, comprising: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform any of the data update methods described above.

[0008] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform any of the data update methods described above.

[0009] The embodiments of this invention provide one or more technical solutions that acquire batch update tasks and generate update plans based on the current lake table snapshot. They determine the storage medium and path of the updated files using a preset storage routing strategy, write the updated files to the staging temporary storage path of the corresponding medium, and perform dual integrity and conflict checks. Only after successful checks are the files formally submitted and a new lake table snapshot generated. Applying these embodiments enables unified management of file storage medium allocation during batch updates, proactively intercepting anomalies such as incomplete files and concurrent update conflicts, ensuring the atomicity of update operations, avoiding data inconsistencies where some media are successfully written while others fail, improving data update consistency, and stably maintaining the integrity of lake table snapshot versions and the accuracy of data read and write operations. Attached Figure Description

[0010] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart illustrating the data update method provided by the present invention; Figure 2 A schematic diagram of the architecture of the data update method provided by the present invention; Figure 3 This is another flowchart illustrating the data update method provided by the present invention; Figure 4 A schematic diagram of the data update device provided by the present invention; Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0011] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0012] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0013] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0014] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0015] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0016] Existing data lake table formats typically manage table state through snapshots, manifests, and data file references. Update operations can be implemented using methods such as copy-on-write, merge-on-read, delete file, or rewrite data files. These mechanisms avoid directly modifying existing data files and control the visibility of old and new files through metadata commits.

[0017] In a single storage medium scenario, batch update tasks typically write new files to the table path and then generate a new snapshot via the data lake table commit protocol. If the commit fails, new files not referenced by the snapshot can be cleaned up as orphan files; if the commit succeeds, old files can be cleaned up after the snapshot expires.

[0018] In hybrid cloud or multi-storage media scenarios, platforms typically control file write locations through storage path rules, LocationProviders (components in lake tables used to generate and resolve file storage paths), FileIO (file input / output components), object storage connectors, or HDFS (Hadoop Distributed File System) client configuration. Different partitions or different data files can reside under different schemes (storage protocol identifiers) or buckets (object storage buckets).

[0019] Some systems use external distributed transactions, two-phase commit, global locks, or task-level mutual exclusion to control cross-storage writes. However, these methods often rely on additional services and cannot directly resolve the relationship between data lake table snapshots, old file references, new file visibility, and snapshot cleanup.

[0020] Existing data update methods have the following problems: First, the target storage medium is uncertain: different partitions of the same table may be located on different media such as S3 (Simple Storage Service), HDFS, OSS (Object Storage Service). If the update task only writes according to the default table path, it may write the new file to the wrong medium, causing the data distribution and governance strategy to fail.

[0021] Second, the underlying storage lacks unified transactions: storage systems such as S3, HDFS, and OSS cannot inherently guarantee atomic commits across media. If the underlying file write results are used as the visibility boundary, a half-update state can easily occur where some media writes are successful while others fail.

[0022] Third, directly overwriting old files carries a high risk: if the update task directly overwrites the old data file or partition directory, it may affect query tasks that are still reading the old snapshot, and cause accidental overwriting, data loss or duplicate results during concurrent updates.

[0023] Fourth, concurrent update conflicts are difficult to detect. Multiple batch update, compaction, rewrite, or partition correction tasks may modify the same data file or the same partition at the same time. If the basesnapshot and the reference relationships of the affected files are not checked, it is easy to commit based on the expired plan.

[0024] Fifth, failure remnants are difficult to manage. After an update task writes to staging files on multiple media, if the submission fails or the task exits abnormally, files that are not referenced by snapshots will remain in the underlying storage. A compensation mechanism that can be identified, audited, and delayed in cleaning up is needed.

[0025] Based on this, the present invention provides a data update method, apparatus, electronic device, and storage medium. The data update method provided by the present invention can be applied to any electronic device with data update function, such as a server, computer, or mobile terminal. The following describes the solution of the present invention with reference to the accompanying drawings: Figure 1 A flowchart illustrating the data update method provided by this invention, as shown below. Figure 1As shown, the data update method provided by the present invention may include the following steps: S101. Obtain the target batch update task, wherein the target batch update task includes the target batch update file and the target update type of each target batch update file; S102. Based on the target batch update task and the current lake table snapshot of the data lake, generate a target update plan. The target update plan includes at least the target update partition involved in the target batch update task and the original storage medium where the target batch update file is located. S103. Based on the target batch update task and the preset storage routing strategy, determine the target storage medium and / or target storage path for each target updated file; wherein, the target updated file is the file obtained by updating the target batch update file according to the target update type. S104. Store the target updated file to a preset staging path on the target storage medium for the target updated file, and verify the target updated file. The verification includes at least integrity verification and conflict detection. S105. If the verification passes, the target updated file is submitted from the Staging path to the target storage medium, and a corresponding updated lake table snapshot is generated.

[0026] In this embodiment of the invention, a batch update task is obtained and an update plan is generated based on the current lake table snapshot. The storage medium and path of the updated files are determined according to a preset storage routing strategy. The updated files are written to the staging temporary storage path of the corresponding medium, and integrity and conflict checks are performed. After the checks pass, the files are officially submitted and a new lake table snapshot is generated. Applying this embodiment of the invention enables unified management of file storage medium allocation during batch updates, proactively intercepting anomalies such as incomplete files and concurrent update conflicts, ensuring the atomicity of update operations, avoiding data inconsistencies where some media are successfully written while others fail, improving data update consistency, and stably maintaining the integrity of the lake table snapshot version and the accuracy of data read and write operations.

[0027] The target batch update task can be any type of update task involving multiple files submitted by the user, or it can be a batch file update task generated by the big data engine according to a preset cycle, such as a Merge task or an Overwrite task. The target batch update task may include the target files to be updated and their target update types, as well as the target user information for sending the task, such as the target file identifier and the target user identifier. The target update types can be modification, deletion, insertion, etc.

[0028] Based on the target batch update task, an update plan can be generated through the update plan generation module. This update plan can include the target update partition involved in the target batch update task and the original storage medium where the target batch update files are currently located. The target update partition can specifically be the target partition value, and the original storage medium can be the original storage path or the original storage medium type. In databases / data lakes, data files are typically stored in tables. Tables are the top-level logical containers for storing data in a data lake / database, corresponding to business entities (order tables, user tables, product tables), and consist of multiple rows of data with a fixed structure, defining uniform fields, data types, and constraint rules. A table can contain multiple partitions split according to preset rules, such as splitting the table by time, user type, etc., resulting in multiple partitions. Each partition contains its corresponding partition value. In practical applications, partitions are split according to specified fields. Correspondingly, the partition value is used to identify the field value corresponding to a single data file. For example, if a table is split based on time to obtain multiple partitions, then the partition value of each partition is the specific time. Different partitions of the same table may be stored on different storage media, such as S3, HDFS, OSS, etc. Therefore, the update plan can include multiple target partition values ​​and multiple types of raw storage media.

[0029] The update plan generation module can generate update plans based on the current lake table snapshot. A snapshot is a complete and consistent view of the lake table at a specific point in time. It is the core metadata unit for the lake table to achieve ACID transactions (Atomicity, Consistency, Isolation, Durability), multi-version reads, and time travel. After each write, merge, overwrite, or delete update operation, the system generates a new snapshot, representing the full table state after the transaction is committed. The lake table snapshot contains snapshot metadata and a Manifest List. The snapshot metadata includes the update transaction ID, commit time, operation type (Merge / Overwrite / Append), operator, partition change records, etc. The Manifest List points to a batch of Manifest files, which register all valid data files, partition ownership, storage media, and file verification information under the current version. When reading a snapshot, the big data engine only reads the Manifest bound to it to locate all data in that version, without mixing in expired files from other snapshots.

[0030] In one possible embodiment, the update plan generation module can generate an update plan by parsing the Manifest based on the data lake snapshot. As mentioned above, the Manifest contains all valid data files, partitions, storage media, and file verification information under the data snapshot indication version. Therefore, by parsing the Manifest, information such as the target update partition and original storage media of the target batch update files can be obtained. Simultaneously, the Manifest also includes Data Files and Delete Files. DataFiles are physical files storing the original detailed business data, often in Parquet / ORC format. They are the smallest storage unit for the lake table to carry actual business records, stored on various storage media such as SSDs, HDDs, and archives, and are registered item by item in the Manifest list. DeleteFiles are auxiliary metadata files for the lake table to implement incremental updates and row-level deletions, recording the primary key / position of rows that need to be logically deleted, without directly deleting the original DataFile. Therefore, by parsing the Manifest, the DataFiles and Delete Files involved in the target batch update task can also be obtained and added to the update plan, thereby obtaining the complete update status of the target batch update files.

[0031] In this invention, the target storage medium and target storage path corresponding to the target batch update file can be determined through the storage routing strategy module. This target storage medium / path is the storage medium / path to which the target batch update file needs to be stored after updating. In one possible embodiment, a storage routing strategy can be pre-configured. This strategy may include judgment dimensions and specific storage medium judgment logic for each dimension. For example, the judgment dimensions in the storage routing strategy may include partition, tablespace, tenant, region, hot / cold attribute, media availability, and media capacity information. For example, the order table is partitioned by month; the partition for June 2026 corresponds to high-speed SSD media; the historical partition for 2025 corresponds to low-priced HDD media. The target storage medium is usually determined based on the results of multi-dimensional judgments. If the storage medium results obtained from multi-dimensional judgments are different, the storage medium corresponding to the judgment dimension with the highest priority can be determined as the target storage medium according to the preset dimension priority. For example, when a new order is written in June, the routing engine judges in the following order: Partition: Belongs to the partition for June 2026, prioritizes the hot media pool. Tablespace: Belongs to the hot_ts hot tablespace, locked SSD candidate pool; Tenant: Paid enterprise tenant, archive cold disks prohibited; Region: User Zhejiang, only Zhejiang data center media is selected; Hot / cold attribute: Current month's orders = hot data, SSDs are required; Media availability: Zhejiang data center SSD node 1 is faulty, node 2 is normal; Capacity: Node 2 has 2TB remaining, sufficient. Final routing decision: Write to the SSD media of node 2 in the Zhejiang data center.

[0032] A batch update task may contain multiple target batch update files that may come from different target update partitions. Correspondingly, there may also be multiple target storage media and target storage paths.

[0033] Storage routing policies may change depending on changes in the storage architecture. To ensure that the target batch update task always uses the same storage routing policy, a version identifier (routePolicyVersion) can be set for the storage routing policy. After receiving the target batch update task, the target batch update task is bound to the version number of the latest storage routing policy. In this way, no matter how the version of the storage routing policy changes, the target batch update task will only use the bound storage routing policy to determine the target storage medium, avoiding the situation where the same batch of data is written to different target media or path templates due to policy changes during task execution.

[0034] The target storage medium and target storage path generated by the policy routing module can be stored in the update plan to form complete information on the file situation before the update (storage medium, partition values, etc.), the update situation, and the file situation after the update. For example, the update plan may include fields such as updateJobId (update task ID), tableIdentifier (unique identifier of the data table), baseSnapshotId (current data lake snapshot ID), operationType (operation type), affectedPartitions (partitions involved, i.e., the target update partition), affectedDataFiles (data files involved), affectedDeleteFiles (files involved), outputFileGroups (output file groups), targetStorageMapping (target storage mapping), routePolicyVersion (storage routing policy version), validationPolicy (validation policy), conflictPolicy (conflict detection policy), commitPolicy (commit policy, metadata, snapshot transaction commit rules), and cleanupPolicy (cleanup policy, rules for recycling and cleaning up failed temporary files and expired orphan files). The affectedDataFiles section describes the set of old files that will be replaced or overwritten by the delete file, while the targetStorageMapping section records the target media to which each output file group should be written.

[0035] After updating the target batch update files according to the target update type, the updated files (such as updated datafile, DeleteFile, etc.) are obtained. These updated files are not directly stored in the target storage medium / target storage path, but are first temporarily stored in the Staging path. The old files (old DataFiles) are not directly overwritten, deleted, or moved, nor is the current snapshot modified. The Staging path is the root path across media and is the top-level unified staging directory entry point shared by the entire cluster and all storage media. In one possible embodiment, different Staging paths can be established for different types of target storage media and different batch update task IDs to temporarily store files to be written to different target storage media in the corresponding batch update task. For example, the following Staging root paths can be created for S3, HDFS, and OSS storage media respectively: s3: / / bucket / table / .staging / jobId / , hdfs: / / nameservice / table / .staging / jobId / , oss: / / bucket / table / .staging / jobId / . When the updated file is written to the corresponding staging path, the batch update task ID, partition value, target storage medium, file group number, and number of attempts can be attached. The number of attempts is the number of times the automatic retry will occur after subsequent verification failures, which can be set according to the actual application scenario.

[0036] Verification of the updated file can include integrity checks and conflict checks. Integrity checks include verifying file existence, file size (greater than zero), record count (compliant with the update plan), partition values ​​(consistent with the target partition), schema compatibility, availability of statistics, and file path conforming to the path rules generated by `routePolicyVersion`. For media supporting checksums, checksums or object ETAGTAs can also be verified. Specifically, it checks whether all temporary `DataFile` and `DeleteFile` physical files generated in this transaction actually exist on the storage medium, reads file metadata, determines if the file occupies more than 0 bytes of storage space, and rejects empty files. Before the update task starts, the expected total number of rows to be written (update plan) is calculated; after the file is written, the actual number of data rows in the file is read and compared with the expected value to determine if the record count conforms to the update plan. The partition field values ​​stored in the file are read and matched with the target partition specified in this update. The schema (table structure, field names, field types, field order, nullability) verifies the compatibility between the data structure of the current updated file and the table structure defined in the target lake table. The availability of statistical information refers to the built-in column-level statistics within the DataFile: maximum value, minimum value, and null value count for each column. Therefore, it's possible to verify the completeness and readability of the statistical data for these updated files. Different versions of storage routing policies generate different levels and media directory paths. Therefore, the current routing policy version can be read, and a standard path template can be constructed according to the rules to verify that the actual path of the file in the staging / production directory matches the template perfectly. Checksum refers to calculating a hash value (MD5 / SHA256) based on the entire binary content of the file when writing to it, which is stored in the Manifest. Etag is specific to object storage (OSS, S3). The storage service automatically generates a unique identifier for each file; changes to the file content will change the Etag. Therefore, the physical file can be read, the hash recalculated / the object's Etag can be read, and compared with the pre-stored value in the metadata. Only after all verifications pass can atomic migration from the staging temporary directory to the production media, writing to the Manifest, and generating new snapshots be allowed, thus avoiding consistency issues such as partial updates and metadata / file mismatches from the source.

[0037] Conflict detection involves reading the current table snapshot and manifest before submitting an atomic update task. This includes checking the relationship between the baseSnapshotId and the current snapshot, confirming whether affectedDataFiles are still referenced by valid snapshots, verifying that affected partitions have not been overwritten by other concurrent tasks, and confirming that delete file or rewrite tasks have not modified the same filegroup. If it is found that old files have been replaced, partitions have been overwritten, or the policy version is unacceptable, the commit is rejected and a retry or compensation process begins.

[0038] After conflict detection passes, the files temporarily stored in the staging area are promoted to references in the output files of this commit. New data file or delete file metadata is generated, and a new snapshot is committed using a lake-table-style commit protocol. Upon successful commit, the new snapshot references the updated files located on different storage media such as S3, HDFS, and OSS. Simultaneously, the old data file is not immediately deleted after a successful commit but continues to be referenced by the old snapshot. The system determines the expiration time based on the table's snapshot retention policy, the maximum runtime of read tasks, and compliance requirements. Physical deletion is only allowed after the old snapshot expires and the old file is no longer referenced by any manifest.

[0039] When a commit fails, the current snapshot remains unchanged, and queries still show the old data. Specifically, this invention provides a strategy to prevent accidental overwriting, prohibiting update tasks from directly writing to old data file paths or overwriting old partition directories. For copy-on-write updates, whenever any row of data in a partition needs modification, all old data files in the entire partition are read, and the updated complete data is used to generate a completely new data file. When a new snapshot is generated, the new snapshot's manifest only records these new files and no longer references the original old data files. Queries of the new snapshot show the updated data, while queries of historical old snapshots still read the original old files, without interference. For merge-on-read updates, the system writes a delete file or incremental data file and controls read semantics through sequence numbers or equivalence mechanisms. Specifically, when deleting or modifying data, the original data file is not modified; only two types of auxiliary files are added: a delete file (marking deleted rows) and an incremental data file (storing the newly added / modified records). Each update is assigned an incremental number. During queries, the data is retrieved in sequence, prioritizing incremental files and filtering out old rows marked with delete, ensuring that the latest data is read, thus achieving read semantic control.

[0040] In one possible implementation, if staging write fails, the system deletes or marks the temporary files currently written to the media without proceeding to the commit phase. If verification or conflict detection fails, the system records the reason for the failure and marks the staging files as pending cleanup. If all staging files pass verification, the system prepares to commit the transaction (create a new Manifest, generate a new snapshot); however, if metadata commit fails, the system keeps the old snapshot visible and registers new files not referenced by any snapshot as orphan candidates based on updateJobId.

[0041] For the above-mentioned orphan candidate files, the cleanup module can periodically scan the staging path and update the task audit log to identify files that are beyond the safety window and not referenced by any snapshot (orphan candidate files). Before cleanup, the current snapshot, historical snapshots, and manifest references are checked again to avoid accidentally deleting files that are being referenced by new snapshots or are still being used by read tasks. The cleanup results are written to the audit log.

[0042] Audit logs are used to record the processing status of each batch update task, including the baseSnapshotId, newSnapshotId (new lake table snapshot ID), routePolicyVersion, affected partitions, affected files, target media for new files, verification results, conflict detection results, commit results, and cleanup results for each batch update. Operations personnel can use audit logs to pinpoint the causes of update failures, write errors, and cross-media write delays.

[0043] The data update method provided by this invention can be implemented as an independent update coordination service, or it can be embedded in the batch update task orchestration layer of Spark, Flink, or data lake platforms.

[0044] like Figure 2 As shown, Figure 2 This is a schematic diagram of an architecture for the data update method provided by the present invention. The data update method provided by the present invention may include an update entry layer, a consistency control layer, and a lake table commit layer. The update entry layer (task initiation layer) contains three core modules, which are responsible for receiving and constraining the basic rules of the update task: Batch update task MERGE / Overwrite: This handles the mainstream batch update operations in the data lake, corresponding to two types of update logic: copy-on-write and merge-on-read. Update coordinator planning and scheduling: coordinates the execution sequence of update tasks, calculates update plan information such as the expected number of rows to be written and the target partition; Storage routing policy version locking: Locks the version of the routing policy used in this update, and restricts the generation path specifications for temporary files and official files.

[0045] The consistency control layer (pre-verification and temporary write layer) receives tasks from the entry layer, performs cross-media temporary writes and full-dimensional verification. If verification fails, the submission process is terminated. Specifically, this includes: The update plan affects the following file sets: Identify all old DataFiles, DeleteFiles, and the target partition range involved in this update; Staging writes to the multi-media temporary area: Based on the locked routing policy, a brand new temporary file is written to the unified staging root path across media, and direct overwriting of the original formal file is prohibited; Integrity verification records / partitions / statistics: Performs a full set of file integrity checks, including file existence, file size, number of records, partition values, schema, statistics, path rules, and checksum / ETag. Conflict detection snapshot / file reference: Detects concurrent update conflicts and compares snapshots and file reference relationships to determine whether there are conflicting behaviors of modifying the same partition in parallel.

[0046] The lake table commit layer (transaction commit and fault recovery layer) is entered only after all checks and conflict detections in the consistency control layer have passed. It handles transaction commits and version lifecycle management. Metadata atomic commit makes new snapshot visible: atomically writes to the Manifest and generates a brand new snapshot. The new data is only visible after a successful commit, realizing atomic semantics of all-success or all-failure updates. Failure compensation orphan file cleanup: For two scenarios, verification failure and metadata submission failure, mark the files to be cleaned up as Staging files and register orphan candidate files without snapshot references, and then recycle and delete them in the background. Old snapshots are retained for safe expiration: Historical old snapshots are persistently retained for time travel. Once the snapshot reaches its expiration period and has no references, the old data files that are no longer valid are cleaned up.

[0047] like Figure 3 As shown, Figure 3 Another flowchart illustrating the data update method provided by the present invention may include: Users submit batch update tasks, including MERGE INTO, batch DELETE, batch UPDATE, partition correction, or historical data rewrite.

[0048] The update coordinator reads the current snapshot of the target table, records the baseSnapshotId, and parses the manifest to obtain the affected partitions, old data files, delete files, and the storage media where the files are located, generating a set of files affected by the update plan.

[0049] The storage routing policy module reads the currently available policies, locks the routePolicyVersion, and determines the target media and target staging path for each output file group.

[0050] The update task reads the old file, calculates the update result, and writes the new data file or delete file to the staging path of the target medium such as S3, HDFS, or OSS.

[0051] The integrity verification module performs existence, size, record count, partition value, schema, statistics, and path policy consistency checks on all new files. If any file fails verification, the commit is terminated, the reason for the failure is recorded, and the files already written to the staging file are marked for compensation and cleanup.

[0052] If all checks pass, the conflict detection module rereads the current table metadata and checks whether the base snapshot, affected old files, and affected partitions still meet the commit conditions.

[0053] If concurrent conflicts exist, the submission will be rejected. The system can retry generating an update plan according to the strategy, or proceed to the failure compensation process.

[0054] If there are no conflicts, the metadata submission module generates a new data file / delete file reference and submits a new lake table snapshot, making the new cross-media file visible to queries all at once.

[0055] After a successful commit, the old snapshot remains readable according to the retention policy, and the old files are cleaned up after the snapshot expires; after a failed commit, the old snapshot remains visible, and the new files are treated as orphan candidates and cleaned up later.

[0056] The data update method provided by this invention can achieve the following beneficial effects: Avoid cross-media partial updates: Even if some writes fail in S3, HDFS, or OSS, the new file will not be visible to queries as long as the metadata commit is unsuccessful.

[0057] To prevent accidental overwriting: update tasks do not directly overwrite old data files or old partition directories, but instead write new files and switch via snapshot references.

[0058] Improve concurrency safety: Conflict detection is performed based on the base snapshot and references to affected files to avoid committing based on expired update plans.

[0059] To reduce data redundancy: failed writes to files do not enter a valid snapshot, and old files are cleaned up according to the snapshot lifecycle after a successful commit, thus avoiding the simultaneous reading of new and old data by the current query.

[0060] Adaptable to hybrid cloud and multi-media: It does not require unified transaction capabilities between underlying storage systems, and achieves table-level atomic visibility by using lake table metadata commit.

[0061] Enhanced maintainability: A complete audit chain is formed by updatingJobId, policy version, verification results, conflict results, and cleanup results, which facilitates tracking and compensation.

[0062] Based on the same inventive concept, the present invention also provides a data update device applied to cross-cloud storage media, wherein the cross-cloud storage media includes multiple storage media, such as... Figure 4 As shown, the device 400 may include: The acquisition module 401 is used to acquire target batch update tasks, wherein the target batch update tasks include target batch update files and target update types of each target batch update file; The generation module 402 is used to generate a target update plan based on the target batch update task and the current lake table snapshot of the data lake. The target update plan includes at least the target update partition involved in the target batch update task, the original storage medium where the target batch update file is located, and the target updated file. The target updated file is a file obtained by updating the target batch update file according to the target update type. The determination module 403 is used to determine the target storage medium for each of the target updated files based on the target update plan and the preset storage routing strategy; The staging module 404 is used to store the target updated file to a staging path preset for the target storage medium of the target updated file, and to verify the target updated file, the verification including at least integrity verification and conflict detection. The submission module 405 is used to submit the target updated file from the staging path to the target storage medium and generate a corresponding updated lake table snapshot if the verification passes.

[0063] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention comply with relevant laws and regulations and do not violate public order and good morals.

[0064] An exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of the present invention.

[0065] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0066] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0067] refer to Figure 5 The present invention will now be described in the form of a structural block diagram of an electronic device 500 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0068] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0069] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0070] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, any of the data update methods described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform any of the data update methods described above by any other suitable means (e.g., by means of firmware).

[0071] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0072] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0073] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0074] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0075] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0076] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A data updating method characterized by comprising: Applied to cross-cloud storage media, wherein the cross-cloud storage media includes multiple storage media, the method includes: Obtain target batch update tasks, wherein the target batch update tasks include target batch update files and target update types for each target batch update file; Based on the target batch update task and the current lake table snapshot of the data lake, a target update plan is generated. The target update plan includes at least the target update partition involved in the target batch update task and the original storage medium where the target batch update file is located. Based on the target batch update task and the preset storage routing strategy, the target storage medium and / or target storage path of each target updated file are determined; wherein, the target updated file is the file obtained by updating the target batch update file according to the target update type. The target updated file is stored in a pre-defined staging path on the target storage medium for the target updated file, and the target updated file is verified, including at least integrity verification and conflict detection. If the verification passes, the target updated file is submitted from the Staging path to the target storage medium, and a corresponding updated lake table snapshot is generated.

2. The method according to claim 1, characterized in that, The step of generating a target update plan based on the target batch update task and the current snapshot of the data lake includes: Based on the current lake surface snapshot, obtain the Manifest file of the current lake surface snapshot; The Manifest file is parsed to obtain the target update partition, data file DataFile, deleted file DeleteFile involved in the target batch update task, and the original storage medium where the target batch update file is located, thus forming the target update plan.

3. The method according to claim 2, characterized in that, The target batch update task also includes: target user information; the step of determining the target storage medium and / or target storage path for each target updated file based on the target batch update task and a preset storage routing strategy includes: Bind the current storage routing policy version to the target batch update task; Based on the target user information, target table, target update partition, hot / cold attributes of the target batch update files, and capacity of each storage medium, the target storage medium and / or target storage path of each target updated file are determined according to the current storage routing policy.

4. The method according to claim 2, characterized in that, The step of storing the updated target file to a pre-defined staging path on the target storage medium for the updated target file includes: For the target batch update task, generate the staging root path corresponding to each of the target storage media; Write the updated files of each target and the target batch update task ID into the staging root path of the corresponding target storage medium.

5. The method according to claim 1, characterized in that, The verification of the target updated file includes: Verify whether the target updated file exists in the Staging path, whether the target updated file is not empty, and whether the target storage medium conforms to the storage routing policy; Read the current lake surface snapshot and Manifest file to determine whether the data file DataFile involved in the target batch update task is referenced by a valid snapshot.

6. The method according to claim 5, characterized in that, The step of submitting the updated target file from the staging path to the target storage medium and generating a corresponding updated lake table snapshot includes: Based on the target updated file, generate a new data file DataFile and a new deletion file DeleteFile; Submit an updated lake table snapshot, which references the new data file DataFile and the new deletion file DeleteFile.

7. The method according to claim 6, characterized in that, The method further includes: In the event that the update lake table snapshot metadata submission fails, new files that are not referenced by any snapshot are marked as orphan candidate files, including the new data file DataFile and / or the new deletion file DeleteFile; If the orphan candidate file takes longer than the safety window to generate and is not referenced by any snapshot, the orphan candidate file is cleaned up.

8. A data update device, characterized in that, The device is applied to cross-cloud storage media, which includes multiple storage media, and comprises: The acquisition module is used to acquire target batch update tasks, wherein the target batch update tasks include target batch update files and target update types for each target batch update file; The generation module is used to generate a target update plan based on the target batch update task and the current lake table snapshot of the data lake. The target update plan includes at least the target update partition involved in the target batch update task, the original storage medium where the target batch update file is located, and the target updated file. The target updated file is a file obtained by updating the target batch update file according to the target update type. The determination module is used to determine the target storage medium for each of the target updated files based on the target update plan and the preset storage routing strategy. The staging module is used to store the target updated file to a pre-defined staging path for the target storage medium of the target updated file, and to verify the target updated file, the verification including at least integrity verification and conflict detection. The submission module is used to submit the target updated file from the staging path to the target storage medium and generate a corresponding updated lake table snapshot if the verification passes.

9. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images