A hierarchical distribution model warehouse synchronization system and method

Abstract

The present invention provides a hierarchical distribution model warehouse synchronization system and method, which ensures the rapid synchronization of critical and frequently accessed data through the design of multiple synchronization strategies and the implementation of dynamic scheduling; reduces the amount of data transmission through business classification strategies and on-demand synchronization strategies, reduces network bandwidth occupancy and storage resource requirements; ensures efficient synchronization tasks through real-time adjustment of distribution paths and load balancing, and avoids single-point synchronization overload; the adaptive fault-tolerant mechanism can automatically identify and handle abnormal situations during the synchronization process, thereby improving the reliability of data synchronization. In addition, the present invention proposes an end-to-end data encryption and integrity verification mechanism to protect the security of data during transmission; the role-based access control mechanism ensures that only authorized users and systems can access and synchronize data, thereby enhancing data security management. The present invention has good scalability and can flexibly adapt to synchronization needs of different scales and support new synchronization strategies in the future.

Description

Technical Field

[0001] The present invention belongs to the field of model synchronization, and in particular relates to a hierarchical distribution model warehouse synchronization system and method. Background Art

[0002] With the rapid development of machine learning and deep learning technologies, enterprises and research institutions need to manage and synchronize a large number of model files. These model files are often large in size and require frequent synchronization between development, testing, and production environments, even involving cross-enterprise collaboration. Therefore, an efficient and secure model repository synchronization mechanism is crucial to ensure model version consistency, improve development efficiency, and optimize resource utilization.

[0003] Currently, mainstream model repository synchronization technologies rely on version control systems (such as Git) and their large file support (such as Git LFS), or object storage services (such as Amazon S3 and Alibaba Cloud OSS). These technologies provide basic data synchronization and version management capabilities, enabling model files to be shared across different environments. However, with the continuous expansion of model scale and the increasing complexity of application scenarios, traditional synchronization methods face numerous challenges in practical applications.

[0004] First, in terms of synchronization efficiency, existing methods often do not distinguish between factors such as file size and update frequency, but instead use full synchronization or simple incremental updates for all model data. This method will occupy a large amount of bandwidth when synchronizing model data on a large scale, resulting in a slow synchronization process and may even cause network congestion. Secondly, the problem of resource consumption is also very prominent. Full or simple incremental synchronization requires a large amount of storage space and network bandwidth, especially when synchronizing large files. Resource consumption is particularly serious, increasing the operating costs of the enterprise. In addition, in the case of an unstable network environment or high server load, the existing synchronization mechanism may cause synchronization failure, and cannot guarantee the consistency and integrity of the model data, thereby affecting the normal development and deployment of the model.

[0005] Security and permission control are also crucial issues. Most current synchronization solutions primarily focus on data transmission and storage, but offer insufficient support for access control and permission management, making them unable to meet the security requirements of multi-tenant environments. Furthermore, existing solutions are limited in flexibility and scalability, making it difficult to adapt to rapidly changing business needs. This is especially true in emerging scenarios like cloud computing and edge computing. Traditional synchronization mechanisms lack dynamic adjustment capabilities and cannot efficiently support complex distributed environments.

[0006] Therefore, to address the above issues, there is an urgent need for an efficient, reliable, scalable model warehouse synchronization solution with complete security controls to improve synchronization efficiency, reduce resource consumption, enhance synchronization stability, and meet the company's strict requirements for data security. Summary of the Invention

[0007] The present invention provides a hierarchical distribution model warehouse synchronization system and method to solve the problems of low efficiency, large resource consumption and difficulty in ensuring data security in existing synchronization methods.

[0008] In order to solve the above technical problems, the embodiments of the present invention disclose the following technical solutions:

[0009] One aspect of the present invention provides a hierarchical distributed model repository synchronization system for synchronizing a model in a source model repository to a target model repository, comprising:

[0010] a model priority calculation module configured to calculate the priority of each model in the source model repository based on model characteristics, wherein the model characteristics include at least update frequency, access frequency, importance, and type;

[0011] A topology feature acquisition module is configured to obtain a priority of each target node and a transmission quality parameter between connected target nodes, wherein the target node is a server for synchronizing the model;

[0012] The synchronization module is configured to generate a synchronization strategy based on the model priority, the target node priority and the transmission quality parameters between the connected nodes, and synchronize the model to the target end model warehouse.

[0013] Optionally, the model priority calculation module includes:

[0014] An update frequency submodule is configured to determine the update frequency parameter value of all models based on the historical update data of each model in the source model warehouse;

[0015] The access frequency submodule is configured to determine the access frequency parameter values ​​of all models based on the historical access data of each model in the source model warehouse;

[0016] The importance quantification submodule is configured to quantify the importance parameters of each model in the source model warehouse based on user input data;

[0017] The type quantification submodule is configured to quantify the type parameters of each model in the source model warehouse according to the type of the model.

[0018] Optionally, the model priority calculation module further includes a priority submodule configured to calculate the priority of the model according to the following formula:

[0019]

[0020] in, is the update frequency parameter value; is the access frequency parameter value; is the importance parameter value; is the type parameter value; is the weight of the update frequency parameter, which is obtained by looking up the preset update frequency reference table, where the update frequency parameter value is proportional to the weight; is the weight of the access frequency parameter, which is obtained by looking up the preset access frequency reference table, where the access frequency parameter value is proportional to the weight; is the weight of the importance parameter, which is obtained by looking up the preset importance reference table; is the weight of the type parameter, which is found from the preset type reference table.

[0021] Optionally, the topological feature acquisition module includes:

[0022] A node priority submodule is configured to update the priority of each target node according to the real-time status parameters of the target node or the input data of the user, wherein the priority includes core, common and edge;

[0023] a link weight submodule configured to determine a weight of a link between any two connected target nodes based on transmission quality parameters, wherein the transmission quality parameters include at least real-time bandwidth, latency, and packet loss rate;

[0024] The dynamic bandwidth allocation submodule is configured to update the bandwidth of each link according to the real-time traffic and weight of the link.

[0025] Optionally, the synchronization module includes:

[0026] A model grading submodule is configured to rank each model in the source model repository according to its priority, wherein a model having a priority greater than a first threshold is set to a first level, a model having a priority less than the first threshold and greater than a second threshold is set to a second level, and a model having a priority less than the second threshold is set to a third level;

[0027] a distribution submodule configured to transmit the first-level model via a link having a bandwidth exceeding a preset bandwidth threshold and a delay below a preset delay threshold, or to fragment the first-level model and transmit the fragments through multiple target nodes;

[0028] The distribution submodule is further configured to simultaneously transmit a plurality of second-level models within a preset time period;

[0029] The distribution submodule is further configured to transmit the third level model, or suspend transmission of the third level model, when the network occupancy rate is lower than a set occupancy rate threshold.

[0030] Optionally, the synchronization module compares the same model in the source-end model warehouse and the target-end model warehouse, obtains the changed part in the model data, and compresses only the changed part and transmits it to the target end.

[0031] Optionally, the system further includes a cleaning module configured to delete models in edge nodes whose access frequency parameter values ​​are less than a preset cleaning threshold after each preset time period.

[0032] Optionally, the system further includes:

[0033] A network environment perception module is configured to detect network bandwidth and packet loss rate, and to predict network environment data at future moments using a machine learning algorithm;

[0034] The server status awareness module is configured to detect the storage space, CPU load, and I / O performance of each target node.

[0035] Optionally, the synchronization module is in communication with the network environment perception module and the server status perception module, and is configured to set the transmission speed according to the network bandwidth, and determine whether to retransmit according to the packet loss rate;

[0036] The synchronization module is further configured to dynamically adjust the synchronization strategy according to the network environment data and the server status data.

[0037] Optionally, the system further includes:

[0038] A data security module is configured to encrypt model data during transmission and perform integrity verification on synchronized data;

[0039] The permission control module is configured to match the corresponding synchronization function according to the user's identity permission.

[0040] Optionally, the system further comprises an adaptive fault-tolerance module configured to automatically detect errors during the synchronization process and execute a preset recovery strategy when a synchronization error, network instability or server failure occurs;

[0041] The adaptive fault-tolerant module is further configured to record synchronization history logs and error logs.

[0042] Optionally, the system further includes a data receiving and merging module, which is configured to decompress and decrypt the received synchronization data at the target end and merge it into the target end model warehouse.

[0043] Another aspect of the present invention provides a hierarchical distribution model repository synchronization method, which is applied to a hierarchical distribution model repository synchronization system and includes:

[0044] Calculating the priority of each model in the model repository based on model characteristics, wherein the model characteristics include at least update frequency, access frequency, importance, and type;

[0045] Obtaining a priority of each target node and a transmission quality parameter between connected target nodes, wherein the target node is a server for synchronizing a model;

[0046] Generate a synchronization strategy based on the model priority, target node priority, and transmission quality parameters between connected nodes, and synchronize the model to the target end model warehouse.

[0047] The present invention provides a hierarchical distribution model warehouse synchronization system and method. By designing multiple synchronization strategies and implementing dynamic scheduling, this system ensures rapid synchronization of critical and frequently accessed data, avoiding inefficiencies caused by indiscriminate synchronization. By reducing data transmission volume through business tiering and on-demand synchronization, the system effectively reduces network bandwidth usage and storage resource requirements, thereby reducing resource consumption and operating costs. Real-time adjustment of distribution paths and load balancing ensure efficient synchronization tasks, avoiding overloads caused by single-point synchronization. Furthermore, an adaptive fault-tolerance mechanism automatically identifies and handles anomalies during the synchronization process (such as network instability and server failures), improving the reliability of data synchronization.

[0048] In addition, the present invention proposes an end-to-end data encryption and integrity verification mechanism to protect data security during transmission, preventing the risk of data being illegally stolen or tampered with during synchronization. A role-based access control mechanism ensures that only authorized users and systems can access and synchronize data, enhancing data security management.

[0049] This invention has good scalability and can flexibly adapt to synchronization needs of different scales and support new synchronization strategies in the future, meeting the needs of large-scale expansion of synchronization tasks. While ensuring the efficiency and reliability of model data synchronization, it reduces resource consumption and strengthens data security and permission management.

[0050] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0052] Figure 1A schematic diagram of the structure of a hierarchical distribution model warehouse synchronization system provided by an embodiment of the present invention;

[0053] Figure 2 An embodiment of the present invention provides a Figure 1 Schematic diagram of the structure of the model priority calculation module;

[0054] Figure 3 An embodiment of the present invention provides a Figure 1 Schematic diagram of the structure of the topological feature acquisition module;

[0055] Figure 4 An embodiment of the present invention provides a Figure 1 Schematic diagram of the structure of the synchronization module;

[0056] Figure 5 A flowchart of a hierarchical distribution model warehouse synchronization method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0059] Figure 1 This is a structural diagram of a hierarchical distribution model warehouse synchronization system provided by an embodiment of the present invention, which is used to synchronize the model of the source model warehouse to the target model warehouse. Figure 1 As shown, the system includes the following modules:

[0060] (1) Model priority calculation module

[0061] The model priority calculation module 1 is used to calculate the priority of each model in the source model warehouse according to the model characteristics, wherein the model characteristics include at least update frequency, access frequency, importance and type.

[0062] In one embodiment disclosed in the present invention, Figure 2 As shown, the model priority calculation module 1 includes an update frequency submodule 11, an access frequency submodule 12, an importance quantization submodule 13 and a type quantization submodule 14, which are respectively used to obtain the quantized values ​​of different model features, wherein,

[0063] (1) The update frequency submodule 11 is used to determine the update frequency parameter values ​​of all models based on the historical update data of each model in the source model warehouse.

[0064] After obtaining the historical update data for each model, the update cycle is determined according to the update frequency, and the update cycles are graded. For example, a model that is updated daily is set as level 1, and the update frequency parameter value of the level 1 model is set to 5; a model that is updated weekly is set as level 2, and the update frequency parameter value of the level 2 model is set to 4; a model that is updated monthly is set as level 3, and the update frequency parameter value of the level 3 model is set to 3; a model that is updated quarterly is set as level 4, and the update frequency parameter value of the level 4 model is set to 2; a model with a longer update cycle is set as level 5, and the update frequency parameter value of the level 5 model is set to 1.

[0065] (2) The access frequency submodule 12 is used to determine the access frequency parameter values ​​of all models based on the historical access data of each model in the source model warehouse.

[0066] Normalize the number of visits to the model and assign different weights to different visit periods. For example, visits in the last 7 days have a weight of 1.5, visits in the last 30 days have a weight of 1.2, visits in the last 90 days have a weight of 1, and visits before 90 days have a weight of 0.8.

[0067] The access frequency parameter of each model in a preset time period (for example, half a year) is calculated as follows:

[0068] Access frequency parameter = (number of visits in the past 7 days × 1.5) + (number of visits in the past 30 days × 1.2) + (number of visits in the past 90 days × 1.0) + (number of visits more than 90 days ago × 0.8).

[0069] (3) The importance quantification submodule 13 is used to quantify the importance parameters of each model in the source model warehouse based on the user's input data.

[0070] Based on business needs, the impact of the model on the business is evaluated and divided into core models, general models and backup models. Among them, the core model affects the core functions of the business, is updated frequently, and has a large number of visits; the general model has a smaller impact on the business, has a medium update frequency, and has a medium number of visits; the backup model serves as a backup for the core model, has a low update frequency, and has a low number of visits.

[0071] The user sets the importance parameter for each model according to the above division rules. For example, the importance parameter of the core model is 10, the importance parameter of the general model is 5, and the importance parameter of the backup model is 2.

[0072] (4) The type quantification submodule 14 is used to quantify the type parameters of each model in the source model warehouse according to the type of the model.

[0073] Pre-label different types of model data, such as training models, inference models, configuration files, etc., and set different quantization values ​​for different types.

[0074] In one embodiment disclosed in the present invention, the model priority calculation module 1 further includes a priority submodule, which can calculate the priority of the model according to the following formula:

[0075]

[0076] in, is the update frequency parameter value; is the access frequency parameter value; is the importance parameter value; is the type parameter value; is the weight of the update frequency parameter, which is obtained by looking up the preset update frequency reference table, where the update frequency parameter value is proportional to the weight; is the weight of the access frequency parameter, which is obtained by looking up the preset access frequency reference table, where the access frequency parameter value is proportional to the weight; is the weight of the importance parameter, which is obtained by looking up the preset importance reference table; is the weight of the type parameter, which is found from the preset type reference table.

[0077] - It can be adjusted according to actual business needs. For example, the weight coefficient of the target node can be set according to the business needs of the target node. The target node is the server used to synchronize the model. It can be divided into core nodes, ordinary nodes and edge nodes according to priority. Edge nodes will pay more attention to access frequency and importance. and Compared with other types of nodes, core nodes are more concerned with update frequency and type. and Higher compared to other types of nodes.

[0078] (2) Topological feature acquisition module

[0079] The topology feature acquisition module 2 is used to obtain the priority of each target node and the transmission quality parameters between connected target nodes.

[0080] In one embodiment disclosed in the present invention, Figure 3 As shown, the topology feature acquisition module 2 includes a node priority submodule 21 , a link weight submodule 22 and a dynamic bandwidth allocation submodule 23 .

[0081] (1) The node priority submodule 21 is used to update the priority of each target node according to the real-time status parameters of the target node or the user's input data.

[0082] Priorities include core, ordinary, and edge. Core nodes carry critical services and have priority in obtaining bandwidth resources. Ordinary nodes allocate remaining resources to meet daily synchronization needs. Edge nodes have limited resources and only obtain critical data.

[0083] In a specific embodiment disclosed in the present invention, the node priority can be calculated using the following formula:

[0084] Priority = (bandwidth usage × 0.4) + (packet loss rate × -0.3) + (CPU usage × 0.2) + (task queue length × 0.1) + (user policy weight).

[0085] The priority classification rules are as follows: core nodes, priority score ≥ 0.8; ordinary nodes, 0.4 ≤ priority score < 0.8; edge nodes, priority score < 0.4.

[0086] The system recalculates priorities and adjusts resource allocation strategies periodically or under specific trigger conditions (such as bandwidth surges, task backlogs, etc.).

[0087] (2) The link weight submodule 22 is used to determine the weight of the link between any two connected target nodes based on the transmission quality parameter.

[0088] A network topology is constructed based on all target nodes, and the weight of the link between each two connected nodes is calculated. The weight is determined by transmission quality parameters, which include at least real-time bandwidth, delay, and packet loss rate.

[0089] The weight of a link represents the transmission capacity of the link. The larger the weight, the higher the link quality. This submodule calculates the weight of each link by integrating multiple transmission quality parameters.

[0090] To adapt to changes in network status, link weights need to be dynamically adjusted. For example, bandwidth, latency, and packet loss rate are remeasured and weights are updated every N seconds (e.g., 30 seconds). Alternatively, when a link experiences a sudden decrease in bandwidth, increase in latency, or increase in packet loss rate, its weight is immediately adjusted and the transmission path is reselected.

[0091] (3) Dynamic bandwidth allocation submodule 23, used to update the bandwidth of each link according to the real-time traffic and weight of each link.

[0092] A higher weight indicates that the link carries more important data traffic, while a lower weight indicates that the link may only be a backup link or a secondary data transmission path. This submodule monitors link traffic in real time and adjusts bandwidth allocation strategies, automatically increasing bandwidth for highly loaded links to ensure service stability and reducing bandwidth for less loaded links to avoid resource waste. Furthermore, when link weights change, bandwidth is automatically reallocated to ensure that links with higher weights receive more resources.

[0093] (3) Synchronization module

[0094] The synchronization module 3 is used to synchronize the model to the target end model warehouse according to the model priority, the target node priority and the transmission quality parameters between the connected nodes.

[0095] In one embodiment disclosed in the present invention, Figure 4 As shown, the synchronization module 3 includes the following submodules:

[0096] (1) Model grading submodule 31, for grading each model in the source model repository according to its priority. After obtaining the priority of each model, models with a priority greater than a first threshold are set to the first level, models with a priority less than the first threshold and greater than a second threshold are set to the second level, and models with a priority less than the second threshold are set to the third level.

[0097] (2) The distribution submodule 32 is used to adopt the first-level model of transmission of links with bandwidth exceeding a preset bandwidth threshold and delay lower than a preset delay threshold, that is, to adopt the first-level model of transmission of links with high bandwidth and low delay first.

[0098] Alternatively, the first-level model can be fragmented and transmitted across multiple destination nodes to improve transmission efficiency. This module splits the model file into several data chunks (e.g., 100MB / chunk), with each node responsible for transmitting a portion of the chunks, ultimately piecing together the complete model. Each chunk has a unique index to ensure correct splicing. A hash checksum (e.g., MD5 / SHA-256) can also be used to prevent data corruption.

[0099] The distribution submodule 32 is also used to simultaneously transmit multiple second-level models within a preset time period to achieve timed batch synchronization and reduce network pressure.

[0100] The distribution submodule 32 is further configured to transmit the third level model when the network occupancy is lower than a set occupancy threshold, or suspend transmission of the third level model, that is, transmit the third level model when the network is idle, or suspend synchronization.

[0101] In one embodiment disclosed in the present invention, the synchronization module 3 compares the same model in the source model warehouse and the target model warehouse, obtains the changed part in the model data, and compresses only the changed part and transmits it to the target end.

[0102] This module can employ version management methods to synchronize only the changed data, reducing transmission volume and improving efficiency. For example, using version control tools (such as Git) to manage model versions and only transmitting incremental data (such as file changes) can be achieved through file-level and block-level differentials. Furthermore, using efficient compression algorithms to compress incremental data further reduces transmission volume.

[0103] In one embodiment disclosed in the present invention, the synchronization module 3 can also transmit only specific branches or files of the model according to user needs.

[0104] In one embodiment disclosed herein, the system also includes a cleaning module. Given the limited storage resources of edge nodes, the module deletes models whose access frequency parameter value is less than a preset cleaning threshold from edge nodes after a preset period of time (e.g., 30 days). This module regularly cleans up low-activity model data to free up storage space. When cleaned model data is requested again, the model can be retrieved.

[0105] In one embodiment disclosed in the present invention, the system further includes the following modules:

[0106] (1) Network environment perception module, which is used to detect network bandwidth, dynamically adjust synchronization speed, and detect packet loss rate, and take measures such as retransmission when the packet loss rate is high. This module can also use machine learning algorithms to predict network environment data at future times to adjust synchronization strategies in advance.

[0107] This module regularly samples network bandwidth, identifies the currently available bandwidth, and dynamically adjusts the synchronization speed to ensure efficient transmission without causing network congestion. It also continuously tracks packet transmission status and calculates the packet loss rate. When the packet loss rate exceeds a threshold, it automatically triggers retransmission and attempts to change the transmission path.

[0108] This module can also predict future network status based on historical network data using time series analysis or deep learning (such as LSTM, GRU) models, so as to reduce the synchronization rate or adjust the synchronization period in advance when network congestion is expected.

[0109] (2) Server status perception module, which is used to detect the storage space, CPU load and I / O performance of each target node to avoid insufficient storage space or resource exhaustion, thereby ensuring synchronization efficiency.

[0110] The server status awareness module monitors the resource status of the target node to ensure that data synchronization does not affect the normal operation of the server. This module regularly checks storage space to prevent synchronization failures due to insufficient space. When storage space is nearing capacity, it clears infrequently accessed data or temporary files. Furthermore, it monitors CPU load to avoid resource contention caused by synchronization tasks. During periods of high load, synchronization priority is lowered and synchronization resumes after resources have recovered. This module also analyzes disk read and write performance in real time to prevent I / O bottlenecks. It uses asynchronous block writes for large file transfers to reduce I / O wait times.

[0111] In one embodiment disclosed in the present invention, the synchronization module 3 is in communication with the network environment perception module and the server status perception module, and is configured to set the transmission speed according to the network bandwidth, and determine whether to retransmit according to the packet loss rate.

[0112] Synchronization Module 3 is also configured to dynamically adjust synchronization strategies based on network environment data and server status data. This module dynamically adjusts transmission speeds based on real-time network bandwidth, ensuring faster synchronization when bandwidth is sufficient and slower synchronization when bandwidth is limited, minimizing the impact on other services. Furthermore, the module monitors packet loss rates and, when high, initiates actions such as retransmission, error correction, or link switching to ensure data integrity.

[0113] Regarding server resource management, the Server Status Awareness Module monitors the target node's storage space, CPU load, and I / O resources in real time to prevent synchronization failures caused by resource exhaustion. Synchronization Module 3 adjusts data synchronization strategies accordingly, for example prioritizing critical data when storage space is insufficient or lowering the priority of synchronization tasks when CPU load is excessive, to ensure stable system operation.

[0114] Furthermore, by incorporating machine learning algorithms, it predicts future network conditions and server resource status, optimizing synchronization strategies in advance. For example, if bandwidth is predicted to decrease, the system can complete large file transfers in advance; if server load is expected to increase, non-urgent synchronization tasks can be postponed. Through this intelligent regulation, Synchronization Module 3 improves the efficiency, stability, and reliability of data transmission.

[0115] In one embodiment disclosed in the present invention, the system further includes the following modules:

[0116] (1) Data security module, which is used to encrypt model data during transmission and perform integrity verification on synchronized data. At the same time, a fine-grained permission control system is designed to ensure the legality and security of data access and synchronization operations.

[0117] The Data Security Module implements end-to-end encryption to ensure the security of the synchronization process and prevent data from being illegally intercepted or tampered with during transmission. For example, by using symmetric encryption (such as AES) or asymmetric encryption (such as RSA), only the authorized recipient can decrypt the data, thereby enhancing data security.

[0118] To further ensure data integrity, the data security module also integrates hash checksums (such as SHA-256) and digital signature technologies to perform integrity checks after data synchronization is complete, ensuring that the data has not been tampered with. If data anomalies are detected during the verification process, the module can automatically trigger a retransmission mechanism or notify the administrator for manual intervention to prevent erroneous data from affecting system operations.

[0119] (2) The permission control module is configured to match the corresponding synchronization function according to the user's identity permissions. This module uses role-based access control (RBAC) to ensure that only authorized users can perform specific synchronization tasks.

[0120] This module incorporates a fine-grained permission control system to strictly restrict data access and synchronization operations. Through a role-based access control (RBAC) mechanism, this ensures that different users can only perform synchronization operations that fall within their scope of authority, thereby improving system security and manageability. This module strictly restricts access to data and system resources based on user identity, role level, and permission configuration, preventing unauthorized operations or malicious data tampering.

[0121] Under the RBAC mechanism, users are assigned to different roles, each with a specific set of permissions. For example, a system administrator has the highest permissions, able to manage all synchronization tasks, adjust synchronization policies, and assign permissions. Regular users can only access and synchronize model data for which they have been granted permissions, while guest users may only have limited read-only permissions. Furthermore, the module supports a permission inheritance mechanism based on the organizational structure. For example, a department head can manage the synchronization permissions of their team members, while regular employees can only access the model data for which they are responsible.

[0122] To further enhance security, the permission control module incorporates multi-factor authentication (MFA) and behavioral analysis technologies. MFA requires users to provide additional authentication (e.g., SMS verification code, dynamic password) when performing sensitive operations (e.g., modifying synchronization rules, adjusting node priorities). Behavioral analysis can be used to detect abnormal operations, such as frequent large-scale synchronization within a short period of time or attempts to access unauthorized data. Once an anomaly is detected, the module can automatically restrict access or trigger a security alert.

[0123] In addition, the module provides detailed permission logging and auditing capabilities, recording all synchronization requests and permission changes, facilitating security audits and issue tracking for administrators. Combined with RBAC mechanisms, multi-factor authentication, and log auditing, this permission control module effectively safeguards the security and compliance of the synchronization system, ensuring efficient and secure data flow within a controlled scope.

[0124] In one embodiment disclosed in the present invention, the system further includes an adaptive fault-tolerant module configured to automatically detect errors during the synchronization process and execute a preset recovery strategy when a synchronization error, network instability, or server failure occurs.

[0125] This module has an adaptive error recovery mechanism, including automatic retry and failure fallback strategies, to cope with abnormal situations such as network instability and server failure. For example, it automatically detects errors during synchronization and adopts appropriate recovery strategies such as retry and switching to an alternative channel.

[0126] This module can monitor in real time various types of failures that may occur during the synchronization process, such as data transmission interruption, network fluctuations, server downtime or storage anomalies, and dynamically select the optimal fault-tolerant strategy based on the type and severity of the failure.

[0127] When a synchronization error is detected, the module first performs fault diagnosis and analyzes the source of the error. For example, a high network packet loss rate may indicate an unstable link, while insufficient storage space on the target node may cause data write failures. The module uses a layered recovery mechanism for different types of failures. For example, when a brief network jitter causes packet loss, an automatic retransmission mechanism can be used for rapid recovery. If the synchronization process is interrupted for a long time, a switch to a backup link or a renegotiation of the synchronization strategy can be used to reduce the scope of the impact. For server failures, the module supports a master-slave switchover mechanism. When a target node is detected to be unavailable, the synchronization task is automatically migrated to an available backup node to ensure that the task is not interrupted.

[0128] The adaptive fault-tolerant module is also configured to record synchronization history logs and error logs, and provide fault diagnosis and early warning functions to reduce manual intervention.

[0129] During data synchronization, the module records detailed historical information in real time, including parameters such as synchronization time, data volume, target node, and network status, and archives it. This historical data can be used for subsequent analysis and optimization, providing a reliable reference for the system. Furthermore, for all synchronization errors, the module generates detailed error logs, documenting the time of occurrence, scope of impact, specific cause, and recovery measures. These logs are not only useful for troubleshooting by technicians, but also serve as training data for machine learning models, helping the system continuously optimize fault prediction and recovery strategies.

[0130] To reduce manual intervention, the module also integrates intelligent fault diagnosis. When an anomaly occurs during synchronization, the system automatically analyzes the error log and, combined with historical data, determines the root cause. For example, if the disk space on a target node consistently approaches its upper limit, the module issues an early warning, prompting the administrator to clean up storage or expand capacity. If the packet loss rate on a link increases abnormally, it infers possible network congestion or hardware failure and automatically adjusts the synchronization path.

[0131] Furthermore, the module supports a real-time early warning mechanism. Based on preset alarm thresholds, such as high packet loss rates, long outstanding synchronization tasks, and excessive server load, the system automatically sends notifications to operations and maintenance personnel, alerting them to potential risks and providing possible solutions. For example, when server CPU load approaches its limit, the module can recommend reducing the synchronization rate or adjusting the synchronization task allocation to avoid system crashes.

[0132] In one embodiment disclosed in the present invention, the system further includes a data receiving and merging module, which is configured to decompress and decrypt the received synchronization data at the target end and merge the data into the target end model warehouse.

[0133] First, the module receives synchronized data from the source on the target end and decompresses it. During data transmission, the system may compress the data to improve bandwidth utilization and transmission efficiency. Therefore, the data reception and merging module needs to decompress the received compressed data to restore the original data structure and ensure the accuracy of subsequent processing.

[0134] Next, this module decrypts the data. During data transmission, the system employs end-to-end encryption to ensure data security and prevent unauthorized access or tampering. Therefore, the target end requires a secure key to decrypt the data, restoring the readability and integrity of the synchronized data. This process is typically combined with access control mechanisms to ensure that only authorized nodes or users can successfully decrypt the data.

[0135] After decompression and decryption, this module is responsible for data merging, integrating the received model data with the existing model repository on the target end. This process involves data version management, duplicate data checking, and consistency verification to ensure data accuracy. For example, the system can implement version control mechanisms to determine whether to overwrite, append, or create a new version based on model version information. Furthermore, methods such as hash checks can be used to check data integrity to prevent corruption or tampering during transmission.

[0136] The module also supports parallel merging and distributed processing, improving data integration efficiency, particularly in large-scale model synchronization scenarios. For fragmented data transmission, the module reassembles multiple data fragments into a complete model according to a pre-set merging strategy to ensure data accuracy and availability.

[0137] Through the processing of the data reception and merging module, the target end can seamlessly connect and synchronize data, ensuring the continuous updating of the model warehouse while maintaining data consistency and security, thereby improving the stability and reliability of the system.

[0138] In one embodiment disclosed in the present invention, the system may be structured according to the following aspects:

[0139] (1) User interface layer

[0140] Users initiate model synchronization requests and manage synchronization tasks through this interface.

[0141] (2) Application service layer

[0142] Permission Management Service: Handles user permission verification to ensure that only authorized users can initiate synchronization tasks.

[0143] Synchronization scheduling service: responsible for receiving synchronization requests and scheduling synchronization tasks according to the tiering strategy and network conditions.

[0144] Synchronization policy service: implements data synchronization strategies, dynamically calculates synchronization priorities and server resource allocation.

[0145] (3) Data processing layer

[0146] Data compression and encryption service: compress and encrypt differential data in preparation for transmission.

[0147] Data decryption and decompression service: After receiving the data, it is decrypted and decompressed back to its original state.

[0148] Model data merging service: merges the received data into the target warehouse.

[0149] (4) Network transport layer

[0150] Transport service: Responsible for the secure transmission of data, including the implementation of retry and failure fallback mechanisms.

[0151] Monitoring service: realize server and network environment status monitoring

[0152] Log Service: stores logs generated during the synchronization process, including task status, error records, etc.

[0153] In one embodiment disclosed in the present invention, the synchronization system can complete the task according to the following process:

[0154] (1) User-initiated synchronization request: The user initiates a model data synchronization request through the user interface layer.

[0155] (2) Permission verification: The permission management service in the application service layer verifies the user's requested permissions.

[0156] (3) Request scheduling: The scheduling service determines the priority of the synchronization task and the allocated network bandwidth based on the classification strategy and the current network conditions.

[0157] (4) Confirm the synchronized data: Compare the source warehouse with the target warehouse to determine the incremental data that needs to be synchronized.

[0158] (5) Data processing: Data compression and encryption services process differential data and perform compression and encryption.

[0159] (6) Secure transmission: Encrypted data is securely transmitted to the target warehouse through the network transport layer.

[0160] (7) Data reception and merging: The data decryption and decompression service of the synchronization target layer processes the received data, and the model data merging service merges it into the target model warehouse.

[0161] (8) Status feedback: After synchronization is completed, the system will feedback the synchronization results to the user.

[0162] Figure 5 A flowchart of a hierarchical distribution model warehouse synchronization method provided by the present invention is shown as follows: Figure 5 As shown, the method includes the following steps:

[0163] Step S100: Calculate the priority of each model in the model repository according to model characteristics, where the model characteristics include at least update frequency, access frequency, importance, and type.

[0164] Step S200: obtaining the priority of each target node and the transmission quality parameters between connected target nodes, where the target node is a server for synchronizing the model.

[0165] Step S300: Generate a synchronization strategy based on the model priority, the target node priority and the transmission quality parameters between the connected nodes, and synchronize the model to the target end model warehouse.

[0166] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A hierarchical distribution model warehouse synchronization system for synchronizing models from a source model warehouse to a target model warehouse, characterized in that: include: a model priority calculation module configured to calculate the priority of each model in the source model repository based on model characteristics, wherein the model characteristics include at least update frequency, access frequency, importance, and type; The model priority calculation module includes: An update frequency submodule is configured to determine the update frequency parameter value of all models based on the historical update data of each model in the source model warehouse; The access frequency submodule is configured to determine the access frequency parameter values ​​of all models based on the historical access data of each model in the source model warehouse; The importance quantification submodule is configured to quantify the importance parameters of each model in the source model warehouse based on user input data; The type quantification submodule is configured to quantify the type parameters of each model in the source model warehouse based on the model type; The model priority calculation module further includes a priority submodule configured to calculate the priority of the model according to the following formula: Priority = w1*p1+w2*p2+w3*p3+w4*p4 Among them, p1 is the update frequency parameter value; p2 is the access frequency parameter value; p3 is the importance parameter value; p4 ​​is the type parameter value; w1 is the weight of the update frequency parameter, which is obtained by looking up the preset update frequency reference table, where the update frequency parameter value is proportional to the weight; w2 is the weight of the access frequency parameter, which is obtained by looking up the preset access frequency reference table, where the access frequency parameter value is proportional to the weight; w3 is the weight of the importance parameter, which is obtained by looking up the preset importance reference table; w4 is the weight of the type parameter, which is obtained by looking up the preset type reference table; A topology feature acquisition module is configured to obtain a priority of each target node and a transmission quality parameter between connected target nodes, wherein the target node is a server for synchronizing a model; The topological feature acquisition module includes: A node priority submodule is configured to update the priority of each target node according to the real-time status parameters of the target node or the input data of the user, wherein the priority includes core, common and edge; a link weight submodule configured to determine a weight of a link between any two connected target nodes based on transmission quality parameters, wherein the transmission quality parameters include at least real-time bandwidth, latency, and packet loss rate; A dynamic bandwidth allocation submodule is configured to update the bandwidth of each link according to the real-time traffic and weight of the link; a synchronization module configured to generate a synchronization strategy based on the model priority, the target node priority, and the transmission quality parameters between the connected nodes, and synchronize the model to the target end model repository; The synchronization module includes: A model grading submodule is configured to rank each model in the source model repository according to its priority, wherein a model having a priority greater than a first threshold is set to a first level, a model having a priority less than the first threshold and greater than a second threshold is set to a second level, and a model having a priority less than the second threshold is set to a third level; a distribution submodule configured to transmit the first-level model via a link having a bandwidth exceeding a preset bandwidth threshold and a delay below a preset delay threshold, or to fragment the first-level model and transmit the fragments through multiple target nodes; The distribution submodule is further configured to simultaneously transmit a plurality of second-level models within a preset time period; The distribution submodule is further configured to transmit the third level model, or suspend transmission of the third level model, when the network occupancy rate is lower than a set occupancy rate threshold.

2. The synchronization system according to claim 1, characterized in that The synchronization module compares the same model in the source model warehouse and the target model warehouse, obtains the changed part in the model data, and compresses only the changed part and transmits it to the target end.

3. The synchronization system according to claim 1, characterized in that The system further includes a cleaning module configured to delete models in edge nodes whose access frequency parameter values ​​are less than a preset cleaning threshold after each preset time period.

4. The synchronization system according to claim 1, characterized in that The system further comprises: A network environment perception module is configured to detect network bandwidth and packet loss rate, and to predict network environment data at future moments using a machine learning algorithm; The server status awareness module is configured to detect the storage space, CPU load, and I / O performance of each target node.

5. The synchronization system according to claim 4, characterized in that The synchronization module is in communication with the network environment perception module and the server status perception module, and is configured to set the transmission speed according to the network bandwidth and determine whether to retransmit according to the packet loss rate; The synchronization module is further configured to dynamically adjust the synchronization strategy according to the network environment data and the server status data.

6. The synchronization system according to claim 1, characterized in that The system further comprises: A data security module is configured to encrypt model data during transmission and perform integrity verification on synchronized data; The permission control module is configured to match the corresponding synchronization function according to the user's identity permission.

7. The synchronization system according to claim 1, characterized in that The system also includes an adaptive fault-tolerance module configured to automatically detect errors during the synchronization process and execute a preset recovery strategy when a synchronization error, network instability, or server failure occurs; The adaptive fault-tolerant module is further configured to record synchronization history logs and error logs.

8. The synchronization system according to claim 1, characterized in that The system also includes a data receiving and merging module, which is configured to decompress and decrypt the received synchronization data at the target end and merge it into the target end model warehouse.

9. A hierarchical distribution model warehouse synchronization method, characterized in that: The method uses the system according to any one of claims 1 to 8, including: Calculating the priority of each model in the model warehouse based on model characteristics, wherein the model characteristics include at least update frequency, access frequency, importance, and type, including: determining update frequency parameter values ​​of all models based on historical update data of each model in the source model warehouse; determining access frequency parameter values ​​of all models based on historical access data of each model in the source model warehouse; quantifying the importance parameter of each model in the source model warehouse based on user input data; and quantifying the type parameter of each model in the source model warehouse based on the type of the model; The priority of a model is calculated according to the following formula: Priority = w1*p1+w2*p2+w3*p3+w4*p4 Among them, p1 is the update frequency parameter value; p2 is the access frequency parameter value; p3 is the importance parameter value; p4 ​​is the type parameter value; w1 is the weight of the update frequency parameter, which is obtained by looking up the preset update frequency reference table, where the update frequency parameter value is proportional to the weight; w2 is the weight of the access frequency parameter, which is obtained by looking up the preset access frequency reference table, where the access frequency parameter value is proportional to the weight; w3 is the weight of the importance parameter, which is obtained by looking up the preset importance reference table; w4 is the weight of the type parameter, which is obtained by looking up the preset type reference table; Obtain the priority of each target node and the transmission quality parameters between connected target nodes, including: Update the priority of each target node according to the real-time status parameters of the target node or the user's input data, wherein the priority includes core, common and edge; the target node is a server for synchronizing the model; Determining a weight of a link between any two connected target nodes based on transmission quality parameters, wherein the transmission quality parameters include at least real-time bandwidth, delay, and packet loss rate; Update the bandwidth of each link according to the real-time traffic and weight of the link; Generate a synchronization strategy based on model priority, target node priority, and transmission quality parameters between connected nodes to synchronize the model to the target model repository, including: Classify each model in the source model warehouse according to its priority, wherein models with a priority greater than a first threshold are set to the first level, models with a priority less than the first threshold and greater than a second threshold are set to the second level, and models with a priority less than the second threshold are set to the third level; Transmitting the first-level model via a link with a bandwidth exceeding a preset bandwidth threshold and a delay below a preset delay threshold, or fragmenting the first-level model and transmitting the fragments through multiple target nodes; Simultaneously transmitting multiple second-level models within a preset time period; When the network occupancy rate is lower than the set occupancy rate threshold, the third level model is transmitted, or the transmission of the third level model is suspended.

Images