Node data synchronization method and apparatus, electronic device, and storage medium

Abstract

The application is suitable for the technical field of data synchronization, and provides a node data synchronization method, which comprises the following steps: when a target node is in a state of waiting for synchronization, obtaining a global content index of data to be synchronized, and obtaining a local content feature set of the target node and a target content feature set of the target node; the local content feature set is determined according to existing data content of the target node, and the target content feature set is determined according to required bearing content of the target node; based on the global content index, difference identification processing is performed on the local content feature set and the target content feature set to obtain a difference identification result; based on the difference identification result, a hierarchical synchronization plan of the target node is generated; and based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing. The method solves the problems of large data transmission volume, long synchronization time, difficult identification of repeated content and insufficient synchronization consistency in the prior art.

Description

Technical Field

[0001] This application belongs to the field of data synchronization technology, and in particular relates to a node data synchronization method, device, electronic device and storage medium. Background Technology

[0002] In video analytics clusters, algorithm mirroring clusters, and edge computing clusters, when nodes are dynamically added, recover from faults, scale horizontally, or upgrade algorithms, it is necessary to synchronize a large number of mirror layers, algorithm models, configuration files, image libraries, personnel libraries, vehicle libraries, and feature indexes. Currently, existing solutions typically rely on pulling from mirror repositories, copying file directories, full synchronization of object storage, or synchronization via scheduled tasks. These solutions suffer from several drawbacks: they can only determine data existence by path or mirror tag, making it difficult to identify data with the same content but different paths; they lack a unified content feature representation for mirror layers, model files, business images, feature files, and index files, hindering cross-data type reuse; the synchronization unit is too coarse, usually based on mirrors, directories, or files; when large files undergo only partial changes, the entire file may still be transmitted; there is a lack of business priority scheduling, failing to guarantee that nodes will synchronize the minimum set of data required to reach a business-ready state first; there is a lack of synchronization consistency protection, potentially leading to issues such as semi-synchronized data being read by business processes, mismatched configuration versions, or incorrectly pointed mirror tags; and feature scanning, difference calculation, compression transmission, verification disk writing, and index updates are executed sequentially, resulting in insufficient overall throughput. Therefore, there is an urgent need for a data synchronization method that can uniformly describe the characteristics of image, model, configuration and business data content, and complete incremental synchronization based on content differences, business dependencies and priorities. This would solve the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical business and insufficient synchronization consistency in existing methods when adding, restoring or expanding nodes in multi-node video analysis clusters or algorithm image clusters. Summary of the Invention

[0003] This application provides a node data synchronization method to address the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical services, and insufficient synchronization consistency in existing methods when adding, restoring, or expanding nodes in multi-node video analysis clusters or algorithm mirror clusters require synchronizing large amounts of data. By using a global content index of the data to be synchronized, the method performs difference identification processing on the local content feature set of the target node and the target content feature set to obtain the difference identification result. Based on the difference identification result, a hierarchical synchronization plan for the target node is generated. Using the hierarchical synchronization plan, incremental data synchronization processing is performed to synchronize the data to be synchronized to the target node. This solves the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical services, and insufficient synchronization consistency in existing methods when adding, restoring, or expanding nodes in multi-node video analysis clusters or algorithm mirror clusters require synchronizing large amounts of data.

[0004] In a first aspect, embodiments of the present invention provide a node data synchronization method, the method comprising the following steps:

[0005] When the target node is in a state of waiting to be synchronized, the global content index of the data to be synchronized is obtained, as well as the local content feature set and the target content feature set of the target node are obtained. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content that the target node needs to carry.

[0006] Based on the global content index, a difference recognition process is performed between the local content feature set and the target content feature set to obtain the difference recognition result.

[0007] Based on the difference identification results, a hierarchical synchronization plan for the target node is generated;

[0008] Based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing.

[0009] Optionally, obtaining the global content index of the data to be synchronized includes:

[0010] Get the data to be synchronized;

[0011] The data to be synchronized is divided into multiple preset data levels by hierarchical processing.

[0012] For each data object in each data level, generate corresponding content feature codes;

[0013] The content feature encoding is written into the global content index, which includes the mapping relationship between data objects, physical storage locations, business objects, node holding status, and version references.

[0014] Optionally, the content feature encoding includes a data object identifier, a block hash, a version identifier, and a dependency identifier. Writing the content feature encoding into the global content index includes:

[0015] If the data object identifier already exists in the global content index and the hash verification status is valid, then add a reference relationship to the global content index;

[0016] If the data object identifier does not exist in the global content index, but the block hash exists, then the missing data block is written into the global content index, and a block-level mapping relationship is established in the global content index;

[0017] If the data object identifier exists in the global content index and the version references are different, then a new business reference relationship is established based on the version identifier and the dependency relationship identifier;

[0018] If the data object identifier already exists in the global content index and the hash verification status is abnormal, it is marked as an object to be repaired and added to the difference synchronization list.

[0019] Optionally, the step of performing difference recognition processing on the local content feature set and the target content feature set based on the global content index to obtain the difference recognition result includes:

[0020] Based on the global content index, object-level difference recognition processing is performed on the local content feature set and the target content feature set to obtain object-level difference recognition results.

[0021] Data objects with a value greater than a preset threshold are selected from the object-level difference recognition results. Block-level difference recognition processing is then performed on the data objects with a value greater than the preset threshold to obtain block-level difference recognition results.

[0022] The local content feature set and the target content feature set are subjected to version difference identification processing to obtain version difference identification results;

[0023] The local content feature set and the target content feature set are subjected to hash verification to identify data objects that fail the verification, and the data objects that fail the verification are added back to the difference synchronization list.

[0024] Optionally, generating a hierarchical synchronization plan for the target node based on the difference identification result includes:

[0025] Based on the difference identification results, the corresponding hierarchical type, dependency relationship link, and business priority are determined;

[0026] Based on the hierarchical type, dependency chain, and business priority, a hierarchical synchronization plan for the target node is generated according to a preset priority order.

[0027] Optionally, the incremental data synchronization processing based on the hierarchical synchronization plan includes:

[0028] Based on the aforementioned hierarchical synchronization plan, the synchronization task is divided into multiple parallel execution phases;

[0029] Based on the multiple parallel execution stages, incremental data synchronization processing is performed in a pipelined parallel manner through task queues and state machines.

[0030] Optionally, after synchronizing the data to be synchronized to the target node, the method further includes:

[0031] After data synchronization is complete, all synchronized data is written to a temporary area, and a hash check is performed to obtain the hash check result.

[0032] If the hash verification result is successful, the business reference is updated, and the data to be synchronized is converted into an available state through atomic switching;

[0033] If the hash verification result is a failure, the formal reference will not be updated, and the data object that failed to synchronize will be recorded in the retry queue.

[0034] Secondly, embodiments of the present invention also provide a node data synchronization device, the node data synchronization device comprising:

[0035] The acquisition module is used to acquire the global content index of the data to be synchronized when the target node is in a state of waiting to be synchronized, as well as the local content feature set and the target content feature set of the target node. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content to be carried by the target node.

[0036] The difference recognition module is used to perform difference recognition processing on the local content feature set and the target content feature set based on the global content index, and obtain the difference recognition result;

[0037] A generation module is used to generate a hierarchical synchronization plan for the target node based on the difference identification results;

[0038] The data synchronization module is used to synchronize the data to be synchronized to the target node through incremental data synchronization processing based on the hierarchical synchronization plan.

[0039] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the node data synchronization method provided in embodiments of the present invention.

[0040] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the node data synchronization method provided in the embodiments of the present invention.

[0041] In this embodiment of the invention, when the target node is in a state of pending synchronization, the global content index of the data to be synchronized is obtained, as well as the local content feature set and the target content feature set of the target node are obtained. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content to be carried by the target node. Based on the global content index, the local content feature set and the target content feature set are subjected to difference identification processing to obtain difference identification results. Based on the difference identification results, a hierarchical synchronization plan for the target node is generated. Based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing. This invention uses a global content index of the data to be synchronized to perform difference identification processing on the local content feature set of the target node and the target content feature set, obtaining the difference identification result. Based on the difference identification result, a hierarchical synchronization plan for the target node is generated. Using the hierarchical synchronization plan, incremental data synchronization processing is used to synchronize the data to be synchronized to the target node. This solves the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical business and insufficient synchronization consistency in existing methods when adding, restoring or expanding nodes in multi-node video analysis clusters or algorithm mirror clusters that need to synchronize a large amount of data.

[0042] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a node data synchronization method provided in one embodiment of this application;

[0045] Figure 2 A flowchart illustrating a difference recognition method provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram of a node data synchronization device provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0049] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0050] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0051] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0052] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0054] like Figure 1 As shown, Figure 1 This is a flowchart of a node data synchronization method provided in an embodiment of the present invention. The node data synchronization method includes the following steps:

[0055] 101. When the target node is in a state of waiting to be synchronized, obtain the global content index of the data to be synchronized, as well as the local content feature set of the target node and the target content feature set of the target node.

[0056] In this embodiment of the invention, the above-described node data synchronization method can be applied to a data synchronization platform. The data synchronization platform can be built on a server-based or distributed system. The data synchronization platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to obtain the global content index of the data to be synchronized, and to obtain the local content feature set and the target content feature set of the target node when the target node is in a state awaiting synchronization. The knowledge database construction program can be used to construct the knowledge database, which is specifically used to provide additional association information for the identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0057] The target node mentioned above can be a node that receives synchronization data, or a node that needs to join the cluster or recover from a failure.

[0058] The aforementioned state of pending synchronization can be that the target node is in a state that needs to be synchronized.

[0059] The data to be synchronized mentioned above can be the data that needs to be transmitted to the target node during the data synchronization process.

[0060] The aforementioned global content index records the mapping relationship between the unique identifier, physical storage location, business object, node holding status, and version reference of all data objects in the cluster.

[0061] The global content index of the data to be synchronized mentioned above can be the content data index corresponding to the data to be synchronized in the global content index.

[0062] The aforementioned local content feature set is determined based on the existing data content of the target node. This existing data can be the target node's local mirror repository, object cache, model directory, configuration directory, and business data directory, etc. The aforementioned local content feature set can be generated by scanning the target node's local storage directory, object cache, model directory, configuration directory, and business data directory, etc. The aforementioned target content feature set is determined based on the content that the target node needs to carry. This required content can be the algorithm tasks, business library scope, and node roles that the target node needs to carry, etc. The aforementioned target content set can be a target content feature set generated based on the algorithm tasks, business library scope, and node roles carried by the target node.

[0063] 102. Based on the global content index, perform difference recognition processing on the local content feature set and the target content feature set to obtain the difference recognition result.

[0064] In this embodiment of the invention, the above-mentioned difference identification process may be a process of comparing the local content feature set with the target content feature set based on the global content index, and identifying existing data objects, missing data objects, corrupted data objects, and version conflicts.

[0065] The aforementioned difference identification results can be a structured difference synchronization list output after performing difference identification processing on the local content feature set and the target content feature set based on the global content index. These results mark existing data objects, missing data objects, partially missing objects, and corrupted data objects, and specifically list the data objects that need to be synchronized.

[0066] It should be noted that, based on the mapping relationship between the unique identifier, physical storage location, business object, node holding status, and version reference of the data object recorded in the global content index, the local content feature set and the target content feature set can be processed to identify differences such as existing data objects, missing data objects, and damaged data objects, and obtain the difference identification results. This can avoid duplicate data transmission and reduce the amount of data transmission.

[0067] 103. Based on the difference identification results, generate a hierarchical synchronization plan for the target node.

[0068] In this embodiment of the invention, the difference identification results can be sorted, grouped, and divided into stages according to a preset priority order to generate a hierarchical synchronization plan for the target node.

[0069] The aforementioned preset priority order can be a pre-set priority order, or it can be calculated according to rules. The rules can be that configuration, certificates, orchestration files, and algorithm registration metadata have the highest priority; the base image layer, runtime dependency layer, and algorithm code layer of the target algorithm have priority over business images and archived data; the model weights and inference engines that the current task depends on have priority over models that are not currently being retrieved; hot business libraries, hot indexes, and features required by the current retrieval task have priority over cold data; when there is network congestion or insufficient disk space, low-priority large objects are postponed; within the same dependency chain, upstream dependencies are synchronized first, and then downstream data is synchronized.

[0070] The aforementioned hierarchical synchronization plan can be an incremental data synchronization execution scheme for target nodes generated by sorting, grouping, and dividing the difference identification results according to a preset priority order.

[0071] It should be noted that by using a tiered synchronization plan, critical data can be synchronized first, shortening the time it takes for nodes to come online.

[0072] 104. Based on the hierarchical synchronization plan, incremental data synchronization processing is used to synchronize the data to be synchronized to the target node.

[0073] In this embodiment of the invention, the above-described incremental data synchronization process can be a process that only transmits the difference identification results, rather than repeatedly transmitting the entire dataset. The difference identification results mark the missing data objects and list the data objects that need to be synchronized.

[0074] Specifically, based on the hierarchical synchronization plan, incremental data synchronization processing can be used to synchronize missing data objects and data objects that need to be synchronized in the difference identification results to the target node.

[0075] In this embodiment of the invention, the present invention identifies the existing content of the target node through global content indexing, and only synchronizes missing or damaged data objects and data blocks. According to the hierarchical synchronization plan of the target node, incremental data synchronization processing is used to synchronize the data to be synchronized to the target node, so that the target node can reach a runnable state. This solves the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical business and insufficient synchronization consistency in existing methods when adding, restoring or expanding nodes in multi-node video analysis clusters or algorithm mirror clusters and needing to synchronize a large amount of data.

[0076] In this embodiment of the invention, when the target node is in a state of pending synchronization, the global content index of the data to be synchronized is obtained, as well as the local content feature set and the target content feature set of the target node are obtained. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content to be carried by the target node. Based on the global content index, difference identification processing is performed on the local content feature set and the target content feature set to obtain the difference identification result. Based on the difference identification result, a hierarchical synchronization plan for the target node is generated. Based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing. This invention solves the problems of large data transmission volume, long synchronization time, difficulty in identifying duplicate content, slow recovery of critical business, and insufficient synchronization consistency in existing methods when adding, restoring, or expanding nodes in multi-node video analysis clusters or algorithm mirror clusters require synchronizing a large amount of data.

[0077] It is understood that in the specific implementation of this application, data related to synchronization data, feature data, and difference data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0078] Optionally, in the step of obtaining the global content index of the data to be synchronized, the data to be synchronized can be obtained; the data to be synchronized can be divided into multiple preset data levels; for each data object in each data level, a corresponding content feature code can be generated; and the content feature code can be written into the global content index.

[0079] In this embodiment of the invention, the data to be synchronized may be data that needs to be transmitted to the target node during the data synchronization process.

[0080] The above-described hierarchical division process involves classifying the data to be synchronized according to its data type and business attributes. The data type can refer to the business role and content format category of the data to be synchronized within the cluster. The business attributes can refer to the role the data to be synchronized plays in the business process.

[0081] The aforementioned preset data layers include the operating system base layer, runtime dependency layer, algorithm code layer, model weight layer, configuration layer, business image layer, business feature layer, business index layer, and archive layer. The operating system base layer (OS_BASE) can be a privileged software layer running directly on the physical hardware within the operating system, responsible for shielding hardware differences and managing core resources. The runtime dependency layer (RUNTIME_LIB) can be runtime dependencies such as CUDA, inference frameworks, OpenCV, and Python packages. The algorithm code layer (ALGO_CODE) can be the main program, entry script, service files, etc., implementing the specific algorithm logic. The model weight layer (MODEL_WEIGHT) can be model weights, inference engine, and model configuration. The configuration layer (CONFIG) can be algorithm configuration, node configuration, service orchestration configuration, etc. The business image layer (BIZ_IMAGE) can be original images and thumbnails of people, vehicles, or images from an image library. The business feature layer (BIZ_FEATURE) can be feature files corresponding to people, vehicles, or images. The aforementioned business index layer (BIZ_INDEX) can be a retrieval index, vector index, inverted index, and business index, etc. The aforementioned archive layer (ARCHIVE) can be historical logs, backup data, and low-frequency access data, etc.

[0082] The content feature encoding described above is used to describe the content identity, hierarchical attributes, version attributes, and business dependency data of a data object or data block. This content feature encoding includes strong content hashing, weak hashing, a block hash list, hierarchical type, version identifier, dependency identifier, reuse scope, and priority. The strong content hash can be a fixed-length hash value calculated using a hash algorithm on the complete content of the data object; such hash algorithms could be SHA-256, SHA-512, etc. The weak hash can be a shorter hash value calculated using a lightweight hash algorithm on the complete content of the data object; such lightweight hash algorithms could be CRC32, Alder-32, etc. The block hash list can be a list formed by dividing the data object into multiple data blocks of a fixed size and calculating the hash value for each data block. The hierarchical type includes the operating system base layer, runtime dependency layer, etc. The version identifier is used to identify the content version number of the data object. The dependency identifier describes the set of upstream data objects that the data object depends on. The reuse scope controls whether the content of the data object can be reused across nodes, warehouses, or business libraries. The above priorities are used to identify the synchronization priority level of data objects.

[0083] It is understandable that the content feature encoding structure can be represented as:

[0084] ContentFeature { content_id: string, weak_hash: string, hash_algo:string, block_size: integer, block_hash_list: list, layer_type: enum, version_id: string, dependency_id: string, reuse_scope: enum, priority_level: enum, object_meta: map}

[0085] Among them, content_id is the data object identifier, generated by strong hash, used to uniquely identify the content; weak_hash is the weak hash check value, used to quickly identify candidate duplicate content; hash_algo is the hash algorithm identifier; block_size is the block size; block_hash_list is the block hash list, used for block-level difference calculation of large objects; layer_type is the synchronization schedule; version_id is the version identifier, used to distinguish algorithm, model, configuration or business library version; dependency_id is the dependency identifier, used to describe the minimum data set required for service startup; reuse_scope is the reuse scope identifier, used to control whether content can be reused across nodes, repositories or business libraries.

[0086] The aforementioned global content index includes the mapping relationship between data objects, physical storage locations, business objects, node holding status, and version references.

[0087] It should be noted that by performing hierarchical division on the data to be synchronized, multiple preset data levels are obtained. For each data object in each data level, a corresponding content feature code is generated and written into the global content index. Through the content feature code, completely identical data objects can be identified, and the synchronization order can be determined according to the level and dependency relationship.

[0088] Optionally, the content feature encoding includes data object identifier, block hash, version identifier, and dependency identifier. In the step of writing the content feature encoding into the global content index, if the data object identifier already exists in the global content index and the hash verification status is valid, then a reference relationship is added to the global content index; if the data object identifier does not exist in the global content index but the block hash exists, then the missing data block is written into the global content index, and a block-level mapping relationship is established in the global content index; if the data object identifier exists in the global content index and the version reference is different, then a new business reference relationship is established based on the version identifier and dependency identifier; if the data object identifier already exists in the global content index and the hash verification status is abnormal, then it is marked as an object to be repaired and added to the difference synchronization list.

[0089] In this embodiment of the invention, the aforementioned global content index table records the mapping relationship between the unique identifier, physical storage location, business object, node holding status, and version reference of all data objects in the cluster.

[0090] The aforementioned data object identifier can be generated by a strong hash and used to uniquely identify the content of the data object. The block hash can be a single block hash value from a list of block hashes, used to determine the existence of data blocks. The version identifier can be a version number that identifies the data object. The dependency identifier can be an identifier describing dependencies.

[0091] The above-mentioned difference synchronization list records all data objects and their status information that need to be synchronized to the target node, including completely missing data objects, partially missing data objects, and data objects with version conflicts.

[0092] It should be noted that in the global content index, each data object or data block uses a data object identifier or block hash as its content address. The actual data can be stored in a mirror repository, object storage, local content pool, or distributed file system, and the actual data can be the original file of the data object or data block itself. When content feature encoding is written to the global content index, if the data object identifier already exists in the global content index and the hash verification status of the data object is valid, no actual data is written; only a reference relationship is added to the global content index. If the data object identifier does not exist in the global content index, but the block hash exists, the missing data block is written to the global content index, and a block-level mapping relationship is established in the global content index. If the data object identifier exists in the global content index, but the version references are different, a new business reference relationship is established based on the version identifier and dependency relationship identifier. If the data object identifier already exists in the global content index, but the hash verification status is abnormal, it is marked as an object to be repaired and added to the difference synchronization list.

[0093] Optionally, in the step of performing difference identification processing on the local content feature set and the target content feature set based on the global content index to obtain the difference identification result, the following steps can be taken: First, perform object-level difference identification processing on the local content feature set and the target content feature set based on the global content index to obtain object-level difference identification results. Then, filter out data objects larger than a preset threshold from the object-level difference identification results, and perform block-level difference identification processing on these data objects to obtain block-level difference identification results. Next, perform version difference identification processing on the local content feature set and the target content feature set to obtain version difference identification results. Finally, perform hash verification processing on the local content feature set and the target content feature set to identify data objects that fail verification, and re-add these data objects to the difference synchronization list.

[0094] In this embodiment of the invention, the aforementioned local content feature set is determined based on the existing data content of the target node. The aforementioned target content set is determined based on the content that the target node needs to carry.

[0095] The object-level difference identification process described above can be a process of determining whether a complete data object already exists for each data object in the local content feature set and the target content feature set based on the data object identifier, according to the global content index. Object-level difference identification can also filter out identical data objects based on weak hash checksums, file size, and hierarchical type.

[0096] The above object-level difference identification results can be based on a global content index, and are output after performing object-level difference identification processing on the local content feature set and the target content feature set. Object-level difference identification includes existing complete data objects and completely missing data objects. For completely missing data objects, they are added to the difference synchronization list.

[0097] Furthermore, for existing complete data objects, data objects larger than a preset threshold are selected, and block-level difference identification processing is performed on these data objects to obtain block-level difference identification results. The preset threshold can be a pre-set critical value for the size of the data objects; data objects larger than the preset threshold require block-level difference identification.

[0098] The aforementioned block-level difference identification process can be used to determine missing data blocks for data objects larger than a preset threshold based on a block hash list. Specifically, for data objects larger than the preset threshold, they can be divided into multiple data blocks of a fixed size. By comparing the block hash lists of the local content feature set and the target content feature set, it can be determined block by block whether the data block exists, is missing, or has changed, thereby identifying the data blocks that need to be synchronized.

[0099] The aforementioned block-level difference identification result can be the matching status of each data block in a data object exceeding a preset threshold after performing block-level difference identification processing. The block-level difference identification result records the data blocks that need to be synchronized.

[0100] The version difference identification process described above can be a process of comparing whether the version identifiers of existing complete data objects in the local content feature set and the target content feature set are consistent. Specifically, it can determine whether existing complete data objects conform to the target business version based on version identifiers, dependency relationship identifiers, and business reference relationships.

[0101] The version difference identification result mentioned above can be the version difference identification result obtained after performing version difference identification processing on the local content feature set and the target content feature set. The difference identification result indicates whether the version identifiers of the existing complete data objects in the local content feature set and the target content feature set are consistent. If the version identifiers of the existing complete data objects are consistent, it is not necessary to add them to the difference synchronization list. If the version identifiers of the existing complete data objects are inconsistent, the corresponding data objects can be marked as version conflicts and added to the difference synchronization list.

[0102] The hash verification process described above can be a process of performing hash verification between the local content feature set and the target content feature set. Hash verification can also involve rereading the actual data locally and calculating a strong hash value for an existing complete and version-matched data object, then comparing it with the data object identifier recorded in the target content feature set to verify whether the data object is corrupted at the physical storage level.

[0103] The above-mentioned difference synchronization list records all data objects and their status information that need to be synchronized to the target node, including completely missing data objects, partially missing data objects, data objects with version conflicts, and data objects with failed verification.

[0104] like Figure 2 As shown, Figure 2This is a flowchart of a difference identification method provided by an embodiment of the present invention. Specifically, the difference identification method includes the following steps: 201. Input the target content feature set; 202. Read the local content feature set; 203. Perform object-level comparison according to the data object identifier; 204. Are the data object identifiers consistent? 205. Mark the existing data object; 206. Check the version identifier and dependency relationship identifier; 207. Do the version identifier and dependency relationship identifier match? 208. Only update the reference relationship or node holding status; 209. Filter candidate objects according to weak hash check value, file size, and synchronization schedule; 210. Is it a large object? 211. Read the block hash list; 212. Calculate missing blocks, abnormal blocks, and version conflict blocks; 213. Add to the difference synchronization list. It can be understood that for existing data objects with matching versions, only the reference relationship is updated; for large files or large indexes, block-level difference identification processing can be further performed to avoid retransmitting the complete data object.

[0105] In this embodiment of the invention, based on the global content index, object-level difference identification processing is performed on the local content feature set and the target content feature set to obtain object-level difference identification results. Data objects exceeding a preset threshold are filtered from these object-level difference identification results. Block-level difference identification processing is then performed on these data objects to obtain block-level difference identification results. Block-level difference identification processing allows large files to transmit only changed or missing data blocks, rather than the entire file, greatly reducing the amount of data transmitted. Version difference identification processing is performed on the local content feature set and the target content feature set to obtain version difference identification results. Version difference identification processing ensures that the target node receives the correct version of the data, avoiding business errors caused by version mismatch. Hash verification processing is performed on the local content feature set and the target content feature set to identify data objects that fail verification. These failed data objects are then re-added to the difference synchronization list. Hash verification processing proactively detects corrupted data, ensuring data accuracy and reliability.

[0106] Optionally, in the step of generating a hierarchical synchronization plan for the target node based on the difference identification results, the corresponding hierarchical type, dependency relationship link, and business priority can be determined based on the difference identification results; and the hierarchical synchronization plan for the target node can be generated according to the hierarchical type, dependency relationship link, and business priority in a preset priority order.

[0107] In this embodiment of the invention, the above-mentioned difference identification results mark existing data objects, missing data objects, partially missing objects, damaged data objects, etc., and specifically list the data objects that need to be synchronized.

[0108] Based on the data objects that need to be synchronized, the hierarchical type, dependency chain, and business priority of the data objects to be synchronized can be determined, and a hierarchical synchronization plan for the target nodes can be generated according to the hierarchical type, dependency chain, and business priority in a preset priority order.

[0109] The aforementioned hierarchical types include the operating system foundation layer, runtime dependency layer, and algorithm code layer. The aforementioned dependency chains can be identifiers of upstream and downstream dependencies between data objects. The aforementioned business priorities can be the priority of data objects in terms of business availability.

[0110] The aforementioned preset priority order can be a pre-set priority order, or it can be calculated according to rules. The rules can be that configuration, certificates, orchestration files, and algorithm registration metadata have the highest priority; the base image layer, runtime dependency layer, and algorithm code layer of the target algorithm have priority over business images and archived data; the model weights and inference engines that the current task depends on have priority over models that are not currently being retrieved; hot business libraries, hot indexes, and features required by the current retrieval task have priority over cold data; when there is network congestion or insufficient disk space, low-priority large objects are postponed; within the same dependency chain, upstream dependencies are synchronized first, and then downstream data is synchronized.

[0111] The aforementioned hierarchical synchronization plan can be an incremental data synchronization execution scheme that generates target nodes based on hierarchical type, dependency relationship links, and business priorities, through a preset priority order.

[0112] It should be noted that, based on the difference identification results, the corresponding hierarchical type, dependency chain, and business priority can be determined. Based on the hierarchical type, dependency chain, and business priority, a hierarchical synchronization plan for the target node is generated according to the preset priority order. Through the hierarchical synchronization plan, the target node can first obtain the minimum data set required for the algorithm to start, enter the runnable state first, and then complete the data in the background.

[0113] Optionally, in the step of incremental data synchronization processing based on a hierarchical synchronization plan, the synchronization task can be divided into multiple parallel execution stages based on the hierarchical synchronization plan; and incremental data synchronization processing can be executed in a pipelined parallel manner by connecting task queues and state machines according to the multiple parallel execution stages.

[0114] In this embodiment of the invention, the aforementioned parallel execution phase can be a process of decomposing the synchronization task into multiple parallel and independently runnable phases according to the hierarchical synchronization plan. The parallel execution phase includes a scanning phase, a comparison phase, a reading phase, a packaging phase, a transmission phase, a verification phase, and an activation phase. The scanning phase can involve reading data from the source end and generating content features for both the source and target ends. The comparison phase can involve querying the global content index and calculating differences, i.e., verifying whether the data to be synchronized is consistent with the hierarchical synchronization plan. The reading phase can involve reading missing data objects or missing data blocks from a mirror repository, object storage, or source node. The packaging phase can involve merging and packaging small files and compressing large files into blocks. The transmission phase can involve multi-channel transmission based on priority and support for resuming interrupted transmissions. The verification phase can involve the target node calculating hashes and confirming integrity. The activation phase can involve updating indexes, reference relationships, symbolic links, version pointers, and mirror manifests.

[0115] The task queue described above is responsible for data transfer and buffering between stages. The state machine is responsible for tracking and controlling the state of each data object throughout the entire process. By connecting the task queue and the state machine, it is possible to perform differential calculations and compression preparations simultaneously while the previous batch of data is being transmitted.

[0116] The above-mentioned pipelined parallel approach can be a method of breaking down synchronous tasks into multiple parallel execution stages using a pipeline. At the same time, different batches of data objects are in different stages and proceed in parallel. While the previous batch is being transmitted over the network, the next batch can simultaneously read and compress and package the data. The data that has arrived can be verified and made effective simultaneously.

[0117] The aforementioned incremental data synchronization process can be a process that transmits only the data objects or data blocks marked as missing in the difference identification results, or the data objects to be synchronized, rather than repeatedly transmitting the entire dataset.

[0118] It should be noted that multiple parallel stages are connected through task queues and state machines, allowing the next batch of data to undergo difference calculation and compression preparation simultaneously while the previous batch is being transmitted. For data that has already arrived, verification and disk write-in and index updates can be performed synchronously. This pipelined parallel transmission method ensures that the scanning, comparison, compression, transmission, verification, and activation stages execute concurrently, eliminating the need for sequential waiting. While the current batch of data is being transmitted over the network, the next batch can simultaneously undergo difference calculation and compression preparation, reducing CPU, network, and disk idle time and improving resource utilization.

[0119] Optionally, after synchronizing the data to be synchronized to the target node, all synchronized data can be written to a temporary area and a hash check can be performed to obtain the hash check result. If the hash check result is successful, the business reference is updated and the data to be synchronized is converted to an available state through an atomic switch. If the hash check result is unsuccessful, the formal reference is not updated and the data object that failed to synchronize is recorded in the retry queue.

[0120] In this embodiment of the invention, all of the above-mentioned synchronized data can be all data objects or data blocks synchronized to the target node.

[0121] The aforementioned temporary area can be an isolated storage area used to temporarily store all synchronized data. The temporary area can be a temporary directory, a temporary namespace, a temporary content address, etc.

[0122] The hash verification mentioned above can be a process of calculating the hash value of all synchronized data based on the data object identifier or block hash, and comparing it with the pre-stored hash value to verify whether the synchronized data has been damaged or tampered with during transmission or storage.

[0123] The hash verification results above include verification passed and verification failed.

[0124] The retry queue mentioned above is used to record data objects that failed to synchronize and data objects that need to be resynchronized. The core of the retry queue is to temporarily store and resubmit data objects that failed to synchronize, so as to avoid the loss of data objects that failed to synchronize due to network fluctuations or momentary service failures.

[0125] Furthermore, when the hash verification result is successful, the data to be synchronized can be converted to a usable state through an atomic switch by updating the business reference. When the hash verification result is unsuccessful, the official reference is not updated, and the data object that failed to synchronize is recorded in the retry queue. The above atomic switch can be performed by instantly switching the data reference from the old version to the new version after the hash verification is successful, so that the new data that is not visible to the business in the temporary area becomes a fully visible and usable state in the official area in an instant. Atomic switches can be performed by renaming the target, switching soft links, switching version pointers, updating the image manifest, etc., to convert the data to be synchronized into a usable state.

[0126] Understandably, if the hash verification passes, the business reference can be updated, causing the production path, mirror tag, or business index to point to the new version. If the hash verification fails, the previous stable version can be retained, the production reference not updated, and the failed data object recorded in the retry queue. After a hash verification failure, the corresponding data object or data block can be fetched again, and after successful verification, the business reference can be updated, and the data to be synchronized can be converted to a usable state through an atomic switch.

[0127] It should be noted that after data synchronization is complete, writing all synchronized data to a temporary area and performing hash verification can prevent data objects that fail verification from being read by business services.

[0128] like Figure 3 As shown, an embodiment of the present invention provides a node data synchronization device, which includes:

[0129] The acquisition module 301 is used to acquire the global content index of the data to be synchronized when the target node is in a state of waiting to be synchronized, and to acquire the local content feature set and the target content feature set of the target node. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content to be carried by the target node.

[0130] The difference recognition module 302 is used to perform difference recognition processing on the local content feature set and the target content feature set based on the global content index to obtain the difference recognition result;

[0131] The generation module 303 is used to generate a hierarchical synchronization plan for the target node based on the difference identification results;

[0132] The data synchronization module 304 is used to synchronize the data to be synchronized to the target node through incremental data synchronization processing based on the hierarchical synchronization plan.

[0133] Optionally, the acquisition module 301 is further configured to acquire data to be synchronized; perform hierarchical division processing on the data to be synchronized to obtain multiple preset data levels; generate corresponding content feature codes for each data object in each data level; and write the content feature codes into a global content index, wherein the global content index includes the mapping relationship between data objects, physical storage locations, business objects, node holding status, and version references.

[0134] Optionally, the acquisition module 301 is further configured to: if the data object identifier already exists in the global content index and the hash verification status is valid, add a reference relationship in the global content index; if the data object identifier does not exist in the global content index but the block hash exists, write the missing data block into the global content index and establish a block-level mapping relationship in the global content index; if the data object identifier exists in the global content index but the version reference is different, establish a new business reference relationship based on the version identifier and the dependency relationship identifier; if the data object identifier already exists in the global content index but the hash verification status is abnormal, mark it as an object to be repaired and add it to the difference synchronization list.

[0135] Optionally, the difference identification module 302 is further configured to perform object-level difference identification processing on the local content feature set and the target content feature set based on the global content index to obtain object-level difference identification results; filter out data objects with values ​​greater than a preset threshold from the object-level difference identification results, perform block-level difference identification processing on the data objects with values ​​greater than the preset threshold to obtain block-level difference identification results; perform version difference identification processing on the local content feature set and the target content feature set to obtain version difference identification results; perform hash verification processing on the local content feature set and the target content feature set to determine data objects that fail verification, and re-add the data objects that fail verification to the difference synchronization list.

[0136] Optionally, the generation module 303 is further configured to determine the corresponding hierarchical type, dependency relationship link, and business priority based on the difference identification result; and generate a hierarchical synchronization plan for the target node according to the hierarchical type, dependency relationship link, and business priority in a preset priority order.

[0137] Optionally, the data synchronization module 304 is further configured to split the synchronization task into multiple parallel execution stages based on the hierarchical synchronization plan; and to perform incremental data synchronization processing in a pipelined parallel manner by connecting the multiple parallel execution stages through task queues and state machines.

[0138] Optionally, the device is further configured to, after data synchronization is completed, write all synchronized data into a temporary area and perform hash verification to obtain a hash verification result; if the hash verification result is a pass, update the business reference and convert the data to be synchronized into an available state through atomic switching; if the hash verification result is a fail, do not update the formal reference and record the data object that failed to synchronize into the retry queue.

[0139] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described node data synchronization methods.

[0140] Specifically, it includes a processor 401 and a memory 402, as well as a computer program stored in the memory 402 and capable of running on the processor 401, which executes a node data synchronization method, wherein:

[0141] The processor 401 executes the calculator program for the node data synchronization method stored in the memory 402, performing the following steps:

[0142] When the target node is in a state of waiting to be synchronized, the global content index of the data to be synchronized is obtained, as well as the local content feature set and the target content feature set of the target node are obtained. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content that the target node needs to carry.

[0143] Based on the global content index, a difference recognition process is performed between the local content feature set and the target content feature set to obtain the difference recognition result.

[0144] Based on the difference identification results, a hierarchical synchronization plan for the target node is generated;

[0145] Based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing.

[0146] Optionally, the global content index for obtaining the data to be synchronized, executed by processor 401, includes:

[0147] Get the data to be synchronized;

[0148] The data to be synchronized is divided into multiple preset data levels by hierarchical processing.

[0149] For each data object in each data level, generate corresponding content feature codes;

[0150] The content feature encoding is written into the global content index, which includes the mapping relationship between data objects, physical storage locations, business objects, node holding status, and version references.

[0151] Optionally, the content feature encoding includes a data object identifier, a block hash, a version identifier, and a dependency identifier. The step of writing the content feature encoding into the global content index, executed by processor 401, includes:

[0152] If the data object identifier already exists in the global content index and the hash verification status is valid, then add a reference relationship to the global content index;

[0153] If the data object identifier does not exist in the global content index, but the block hash exists, then the missing data block is written into the global content index, and a block-level mapping relationship is established in the global content index;

[0154] If the data object identifier exists in the global content index and the version references are different, then a new business reference relationship is established based on the version identifier and the dependency relationship identifier;

[0155] If the data object identifier already exists in the global content index and the hash verification status is abnormal, it is marked as an object to be repaired and added to the difference synchronization list.

[0156] Optionally, the processor 401 performs a difference recognition process based on the global content index to identify the difference between the local content feature set and the target content feature set, obtaining a difference recognition result, including:

[0157] Based on the global content index, object-level difference recognition processing is performed on the local content feature set and the target content feature set to obtain object-level difference recognition results.

[0158] Data objects with a value greater than a preset threshold are selected from the object-level difference recognition results. Block-level difference recognition processing is then performed on the data objects with a value greater than the preset threshold to obtain block-level difference recognition results.

[0159] The local content feature set and the target content feature set are subjected to version difference identification processing to obtain version difference identification results;

[0160] The local content feature set and the target content feature set are subjected to hash verification to identify data objects that fail the verification, and the data objects that fail the verification are added back to the difference synchronization list.

[0161] Optionally, the process executed by processor 401 to generate a hierarchical synchronization plan for the target node based on the difference identification result includes:

[0162] Based on the difference identification results, the corresponding hierarchical type, dependency relationship link, and business priority are determined;

[0163] Based on the hierarchical type, dependency chain, and business priority, a hierarchical synchronization plan for the target node is generated according to a preset priority order.

[0164] Optionally, the incremental data synchronization processing based on the hierarchical synchronization plan executed by processor 401 includes:

[0165] Based on the aforementioned hierarchical synchronization plan, the synchronization task is divided into multiple parallel execution phases;

[0166] Based on the multiple parallel execution stages, incremental data synchronization processing is performed in a pipelined parallel manner through task queues and state machines.

[0167] Optionally, after synchronizing the data to be synchronized to the target node, the method executed by the processor 401 further includes:

[0168] After data synchronization is complete, all synchronized data is written to a temporary area, and a hash check is performed to obtain the hash check result.

[0169] If the hash verification result is successful, the business reference is updated, and the data to be synchronized is converted into an available state through atomic switching;

[0170] If the hash verification result is a failure, the formal reference will not be updated, and the data object that failed to synchronize will be recorded in the retry queue.

[0171] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the node data synchronization method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0172] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A node data synchronization method, characterized in that, The method includes the following steps: When the target node is in a state of waiting to be synchronized, the global content index of the data to be synchronized is obtained, as well as the local content feature set and the target content feature set of the target node are obtained. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content that the target node needs to carry. Based on the global content index, a difference recognition process is performed between the local content feature set and the target content feature set to obtain the difference recognition result. Based on the difference identification results, a hierarchical synchronization plan for the target node is generated; Based on the hierarchical synchronization plan, the data to be synchronized is synchronized to the target node through incremental data synchronization processing.

2. The node data synchronization method as described in claim 1, characterized in that, The process of obtaining the global content index of the data to be synchronized includes: Get the data to be synchronized; The data to be synchronized is divided into multiple preset data levels by hierarchical processing. For each data object in each data level, generate corresponding content feature codes; The content feature encoding is written into the global content index, which includes the mapping relationship between data objects, physical storage locations, business objects, node holding status, and version references.

3. The node data synchronization method as described in claim 2, characterized in that, The content feature encoding includes a data object identifier, a block hash, a version identifier, and a dependency identifier. Writing the content feature encoding into the global content index includes: If the data object identifier already exists in the global content index and the hash verification status is valid, then add a reference relationship to the global content index; If the data object identifier does not exist in the global content index, but the block hash exists, then the missing data block is written into the global content index, and a block-level mapping relationship is established in the global content index; If the data object identifier exists in the global content index and the version references are different, then a new business reference relationship is established based on the version identifier and the dependency relationship identifier; If the data object identifier already exists in the global content index and the hash verification status is abnormal, it is marked as an object to be repaired and added to the difference synchronization list.

4. The node data synchronization method as described in any one of claims 1 to 3, characterized in that, The step of performing difference recognition processing on the local content feature set and the target content feature set based on the global content index to obtain the difference recognition result includes: Based on the global content index, object-level difference recognition processing is performed on the local content feature set and the target content feature set to obtain object-level difference recognition results. Data objects with a value greater than a preset threshold are selected from the object-level difference recognition results. Block-level difference recognition processing is then performed on the data objects with a value greater than the preset threshold to obtain block-level difference recognition results. The local content feature set and the target content feature set are subjected to version difference identification processing to obtain version difference identification results; Perform hash verification on the local content feature set and the target content feature set to identify data objects that fail the verification, and then add the data objects that fail the verification back to the difference synchronization list.

5. The node data synchronization method as described in any one of claims 1 to 3, characterized in that, The step of generating a hierarchical synchronization plan for the target node based on the difference identification results includes: Based on the difference identification results, the corresponding hierarchical type, dependency relationship link, and business priority are determined; Based on the hierarchical type, dependency chain, and business priority, a hierarchical synchronization plan for the target node is generated according to a preset priority order.

6. The node data synchronization method as described in any one of claims 1 to 3, characterized in that, The incremental data synchronization process based on the hierarchical synchronization plan includes: Based on the aforementioned hierarchical synchronization plan, the synchronization task is divided into multiple parallel execution phases; Based on the multiple parallel execution stages, incremental data synchronization processing is performed in a pipelined parallel manner through task queues and state machines.

7. The node data synchronization method as described in any one of claims 1 to 3, characterized in that, After synchronizing the data to be synchronized to the target node, the method further includes: After data synchronization is complete, all synchronized data is written to a temporary area, and a hash check is performed to obtain the hash check result. If the hash verification result is successful, the business reference is updated, and the data to be synchronized is converted into an available state through atomic switching; If the hash verification result is a failure, the formal reference will not be updated, and the data object that failed to synchronize will be recorded in the retry queue.

8. A node data synchronization device, characterized in that, The node data synchronization device includes: The acquisition module is used to acquire the global content index of the data to be synchronized when the target node is in a state of waiting to be synchronized, as well as the local content feature set and the target content feature set of the target node. The local content feature set is determined based on the existing data content of the target node, and the target content feature set is determined based on the content to be carried by the target node. The difference recognition module is used to perform difference recognition processing on the local content feature set and the target content feature set based on the global content index, and obtain the difference recognition result; A generation module is used to generate a hierarchical synchronization plan for the target node based on the difference identification results; The data synchronization module is used to synchronize the data to be synchronized to the target node through incremental data synchronization processing based on the hierarchical synchronization plan.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the node data synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the node data synchronization method as described in any one of claims 1 to 7.

Images