Distributed collaborative management system and method based on federal learning architecture

Abstract

The invention provides a distributed collaborative management system and method based on a federated learning architecture, the system comprises a load balancer, an intelligent predictor and a distributed router, the load balancer is used for receiving a user request and performing load balancing scheduling of an intelligent routing node; the intelligent predictor is used for monitoring the load of each heterogeneous cluster in real time and performing load prediction through an LSTM prediction model; and the distributed router is used for determining an intelligent routing node according to a load balancing scheduling result, completing grammar analysis and semantic verification on a user request, and realizing balance of data element loads in combination with a Q-learning algorithm according to a load prediction result. According to the method, distributed storage and collaborative management of different clusters in the same mode are realized through virtualization of the virtual data bin, and data storage and data access paths are decoupled. Moreover, global data element traffic optimization is realized based on LSTM and Q-learning algorithms in combination with polymorphic features of the data element, and a routing strategy is flexibly adjusted according to a real-time load condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data flow, in particular to a distributed collaborative management system and method based on a federated learning architecture. BACKGROUND

[0002] As the smallest unit of data element flow and transaction, data elements are divided into two different forms of configuration and mode, and the data elements are managed by element dictionary table, and are stored in different multiple heterogeneous database clusters or homogeneous multiple clusters such as hive, doris, hdfs and miluvs in the form of structured, unstructured and vectorization. In the multi-cluster environment, the cross-modal retrieval of data elements faces the following challenges: Interface adaptation problem: heterogeneous clusters use heterogeneous API interface specifications, resulting in protocol incompatibility and parameter standard inconsistency in cross-cluster service calls, significantly increasing the development and maintenance cost of cross-cluster collaboration.

[0003] Lack of management architecture: in the multi-cluster environment, a global unified namespace has not been built, and the metadata management is scattered, making it difficult to realize centralized control and efficient use of data.

[0004] Retrieval mechanism defects: when retrieving data elements across heterogeneous clusters, the data source needs to be manually specified, and the data location cannot be automatically located, making it difficult to realize transparent retrieval and logical view unification between clusters.

[0005] The scheduling strategy cannot be coordinated: due to the multi-modal distribution characteristics of data elements and the dynamic changes of load, the existing scheduling strategy lacks dynamic load sensing capability, and cannot optimize the path according to the real-time resource state. SUMMARY

[0006] The technical problem to be solved by the present application is how to solve the efficient management of data elements in the cross-subject, multi-modal and multi-cluster environment. The present application provides a distributed collaborative management system and method based on a federated learning architecture.

[0007] The distributed collaborative management system based on the federated learning architecture according to the embodiment of the present application comprises: A load balancer for receiving user requests and performing intelligent load balancing scheduling of routing nodes; An intelligent predictor for monitoring the load of each heterogeneous cluster in real time and predicting the load through an LSTM prediction model; A distributed router for determining intelligent routing nodes according to the load balancing scheduling result, completing syntax analysis and semantic verification of user requests, and balancing the load of data elements according to the load prediction result combined with the Q-learning algorithm to determine the virtual data warehouse and the corresponding heterogeneous cluster.

[0008] The distributed collaborative management system based on the federated learning architecture according to the embodiment of the present application realizes the distributed storage and collaborative management of different clusters of the same mode through the virtual implementation of the virtual data warehouse, decouples the data storage and data access path, and realizes the unified management between different modal data. Moreover, in combination with the polymorphic characteristics of the data element, the global data element flow optimization is realized based on the LSTM and Q-learning algorithms, the routing strategy is flexibly adjusted according to the real-time load condition, and the performance bottleneck or insufficient space of a certain heterogeneous cluster node due to excessive load is avoided.

[0009] According to some embodiments of the present application, the system further comprises a virtual collaborative manager for realizing the mapping between the virtual data warehouse and the heterogeneous clusters, realizing the distributed storage and collaborative management of different heterogeneous clusters of the same mode.

[0010] In some embodiments of the present application, the virtual collaborative manager comprises: a plurality of virtual data warehouses, each of which manages different clusters of the same mode, encapsulates different resources by using different virtual instances; a configuration manager for realizing the mapping of different modes of the virtual data warehouse and the heterogeneous clusters; a heterogeneous cluster mode orchestrator for real-time monitoring of the survival state of the heterogeneous clusters, unified orchestration and dynamic scheduling of the resources of the heterogeneous clusters, and construction of a standardized metadata warehouse.

[0011] According to some embodiments of the present application, the distributed router comprises: a sql parser for completing the syntax analysis and semantic verification of the user request; an element mode parser for extracting the mode attribute in the user request and obtaining the mapping relationship between the user request and the virtual data warehouse; an intelligent routing algorithm module for realizing the balance of the data element load based on the load prediction result of the LSTM prediction model in combination with the Q-learning algorithm, and determining the virtual data warehouse and the corresponding heterogeneous group.

[0012] In some embodiments of the present application, the load balancer adopts at least one of the polling algorithm, the random algorithm and the adaptive load algorithm for the load balancing scheduling of the routing node.

[0013] The distributed collaborative management method based on the federated learning architecture according to the embodiment of the present application comprises: A10, receiving a user request and performing the load balancing scheduling of the intelligent routing node; A20, and real-time monitoring the load of each heterogeneous cluster and performing the load prediction through the LSTM prediction model; A30, determining the intelligent routing node according to the load balancing scheduling result, completing syntax analysis and semantic verification on the user request, and balancing the data element load according to the load prediction result, combining the Q-learning algorithm, determining the virtual data warehouse and the corresponding heterogeneous cluster.

[0014] According to the distributed collaborative management method based on the federated learning architecture, the distributed storage and collaborative management of the same modal different clusters are realized through the virtual implementation of the virtual data warehouse, the data storage and the data access path are decoupled, and the unified management between different modal data is realized. Moreover, the global data element flow optimization is realized based on the LSTM and the Q-learning algorithm in combination with the polymorphic characteristics of the data element, the routing strategy is flexibly adjusted according to the real-time load condition, and the performance bottleneck or the insufficient space of a certain heterogeneous cluster node due to the excessively high load is avoided.

[0015] According to the distributed collaborative management method based on the federated learning architecture, the distributed storage and collaborative management of the same modal different clusters are realized through the virtual implementation of the virtual data warehouse, the data storage and the data access path are decoupled, and the unified management between different modal data is realized. Moreover, the global data element flow optimization is realized based on the LSTM and the Q-learning algorithm in combination with the polymorphic characteristics of the data element, the routing strategy is flexibly adjusted according to the real-time load condition, and the performance bottleneck or the insufficient space of a certain heterogeneous cluster node due to the excessively high load is avoided.

[0016] According to some embodiments of the present application, before receiving the user request, the method further comprises: performing mapping between the virtual data warehouse and the heterogeneous cluster.

[0017] In some embodiments of the present application, before step A20, the method further comprises training the LSTM prediction model, specifically comprising: A21, obtaining historical load data of the virtual data warehouse, including: cpu utilization, memory utilization and disk space utilization, and performing normalization processing to obtain training data; A22, training the LSTM prediction model through the training data, and outputting the load data at the future time.

[0018] According to some embodiments of the present application, in step A30, the space state vector of the Q-learning algorithm is: ; Wherein, is the current load state, is the predicted load state, is the task queue state, and TYPE is the task type.

[0019] In some embodiments of the present application, in step A10, at least one of a polling algorithm, a random algorithm and an adaptive load algorithm is used for load balancing scheduling of the routing nodes. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A schematic diagram of a multi-cluster federation collaborative manager according to an embodiment of the present application; Figure 2 A schematic diagram of an LSTM-Q-learning load balancing architecture device composition structure according to an embodiment of the present application; Figure 3 A schematic diagram of an element construction process according to an embodiment of the present application; Figure 4 A flowchart of an element load balancing process realized by combining a load prediction model LSTM and a Q-Learning decision model according to an embodiment of the present application; Figure 5 A distributed virtual collaborative management architecture according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purposes, the present application is described in detail below in combination with the drawings and preferred embodiments.

[0022] The description of the method process in the specification of the present application and the steps of the flowchart in the drawings of the present application do not necessarily strictly follow the step numbers, and the method steps can change the execution order. Moreover, certain steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be divided into multiple steps for execution.

[0023] The present application proposes a distributed collaborative management system based on a federated learning architecture. The system integrates Consensus Protocol, Namespace Isolation, Reinforcement Learning Algorithm and Virtualization Technology, and constructs a multi-level technical architecture, as shown in Figure 1 .

[0024] As shown in Figure 1 , the distributed collaborative management system based on the federated learning architecture according to an embodiment of the present application includes a load balancer, an intelligent predictor and a distributed router.

[0025] The load balancer is configured to receive a user request and perform intelligent routing node load balancing scheduling.

[0026] The distributed router is configured to determine the intelligent routing node according to the load balancing scheduling result, complete syntax analysis and semantic verification of the user request, and balance data element load by combining the Q-learning algorithm according to the load prediction result, to determine the virtual data warehouse and the corresponding heterogeneous cluster.

[0027] According to some embodiments of the present application, as shown in Figure 1 The system further includes a virtual cooperative manager configured to implement mapping between the virtual data warehouse and the heterogeneous cluster, and implement distributed storage and cooperative management of different heterogeneous clusters of the same mode. Figure 5 As shown in

[0028] Each virtual data warehouse manages different clusters of the same mode, encapsulates different resources by using different virtual instances, and the configuration manager is configured to implement mapping between the virtual data warehouse and different modes of the heterogeneous cluster. The heterogeneous cluster mode orchestrator is configured to monitor the survival state of the heterogeneous cluster in real time, uniformly orchestrate and dynamically schedule resources of the heterogeneous cluster, and construct a standardized metadata warehouse.

[0029] According to some embodiments of the present application, as shown in Figure 2 The distributed router includes a sql parser, an element mode parser, and an intelligent routing algorithm module.

[0030] The sql parser is configured to complete syntax analysis and semantic verification of the user request, the element mode parser is configured to extract the mode attribute in the user request, and obtain the mapping relationship between the user request and the virtual data warehouse. The intelligent routing algorithm module is configured to balance data element load based on the load prediction result of the LSTM prediction model, and determine the virtual data warehouse and the corresponding heterogeneous cluster by combining the Q-learning algorithm.

[0031] In some embodiments of the present application, as shown in Figure 1 The load balancer is configured to perform load balancing scheduling of the routing node by using at least one of a polling algorithm, a random algorithm, and an adaptive load algorithm.

[0032] The distributed cooperative management method based on the federated learning architecture according to the embodiments of the present application includes: A10, receiving the user request and performing load balancing scheduling of the intelligent routing node; in some embodiments of the present application, at least one of a polling algorithm, a random algorithm and an adaptive load algorithm is used for load balancing scheduling of the routing node in step A10. It should be noted that before receiving the user request, the method further comprises: performing mapping between the virtual data warehouse and the heterogeneous clusters.

[0033] A20, monitoring the load of each heterogeneous cluster in real time, and performing load prediction through an LSTM prediction model; In some embodiments of the present application, before step A20, the method further comprises training the LSTM prediction model, specifically comprising: A21, obtaining historical load data of the virtual data warehouse, including: cpu utilization, memory utilization and disk space utilization, and performing normalization processing to obtain training data; A22, training the LSTM prediction model through the training data, and outputting load data at a future time.

[0034] According to some embodiments of the present application, in step A30, the space state vector of the Q-learning algorithm is: ; Wherein, is the current load state, is the predicted load state, is the task queue state, and TYPE is the task type.

[0035] A30, determining the intelligent routing node according to the load balancing scheduling result, completing syntax analysis and semantic verification on the user request, and balancing the data element load according to the load prediction result and combining the Q-learning algorithm, to determine the virtual data warehouse and the corresponding heterogeneous cluster.

[0036] The present application has the following beneficial effects: The distributed collaborative management system based on the federated learning architecture according to the embodiments of the present application realizes distributed storage and collaborative management of the same modal different clusters through virtual implementation of the virtual data warehouse, decouples the data storage and data access path, and realizes unified management between different modal data. Moreover, combined with the polymorphic characteristics of the data element, global data element flow optimization is realized based on the LSTM and Q-learning algorithm, the routing strategy is flexibly adjusted according to the real-time load condition, and performance bottleneck or insufficient space of a certain heterogeneous cluster node due to excessive load is avoided.

[0037] The present application will be described below with reference to the accompanying drawings and one specific embodiment, and it should be understood that the following description is only exemplary and should not be construed as a specific limitation of the present application.

[0038] The distributed collaborative management system based on the federated learning architecture is composed of distributed routing, collaborative manager, intelligent load predictor, configuration manager, load balancer and metadata manager, and the core system core components and processes are as follows: 1. Load balancing and routing resolution: The user request is first routed to the load balancing node by the load balancer (Load Balancer), and then the syntax analysis and semantic verification are completed by the distributed router (Distributed Router) to ensure the compliance of the request.

[0039] 2. Multi-modal feature analysis and virtual mapping: The modal features are extracted by the multi-modal parser of the data element, and are mapped to the corresponding virtual data warehouse (Virtual Data Warehouse) to realize the logical abstraction of the heterogeneous storage system.

[0040] 3. Dynamic routing strategy generation: The router selects the modal parser or Q-learning algorithm (Q-Learning Algorithm) according to the operation type (write / retrieve) of the data element, generates a dynamic routing strategy based on real-time load prediction, and realizes intelligent distribution and load balancing of the data element.

[0041] 4. Federated collaborative management and unified retrieval: Through the virtual data warehouse of the virtual coordinator (Virtual Coordinator), the heterogeneous cluster orchestrator (Heterogeneous Cluster Orchestrator) and the configuration manager (Configuration Manager), a unified management view across modalities and clusters is constructed to realize semantic-level collaborative unified retrieval and life cycle management of heterogeneous resources.

[0042] I. Data element load prediction and dynamic routing; The data element has multi-modal and dynamic characteristics, and traditional load balancing algorithms such as round robin, random algorithm and threshold-based scheduling algorithm cannot adapt to complex dynamic load characteristics. Therefore, the present application combines the multi-modal characteristics of the data element, realizes global data element flow optimization based on LSTM and Q-learning algorithm, and flexibly adjusts the routing strategy according to the real-time cpu, memory, disk size and network status. The module is composed of load distribution layer, load perception layer and prediction decision layer, wherein the load perception, load predictor and router are the core components of the distribution layer, load perception layer and prediction decision layer, and the architecture is as shown in Figure 2 .

[0043] (a) Component storage and retrieval process; In the component construction phase, an architecture based on a reinforcement learning-based intelligent routing algorithm accepts user requests. These requests are balanced by the allocation layer. The routing layer receives outputs from both the allocation and load-aware layers. An SQL parser verifies the data components, and a component modality parser analyzes the modal attributes to determine the virtual data warehouse. The intelligent routing component utilizes the LSTM prediction results from the load-aware layer, combined with a Q-learning algorithm, to balance the component load. The main flowchart of component construction is as follows: Figure 3 As shown.

[0044] (ii) Mathematical model of intelligent routing; Intelligent routing is a core component of data element load balancing. The intelligent routing algorithm dynamically writes data to appropriate cluster nodes based on the load and storage capacity of each cluster, thereby achieving load balancing. This algorithm, through the virtual data warehouse's resource manager, can monitor the resource usage of each cluster in real time, such as CPU utilization, memory usage, and storage space. Based on this information, it dynamically adjusts the data writing strategy to prevent performance bottlenecks or insufficient storage space on any cluster node due to excessive load. This component consists of a load prediction model LSTM and a reinforcement learning Q-Learning decision model. The execution flow of the algorithm is as follows: Figure 4 As shown.

[0045] 1. Data acquisition and normalization processing; Collect historical load data (CPU, memory, disk capacity), and normalize the CPU, memory, and disk space data to eliminate the influence of units: ; Where μ and σ are the mean and standard deviation of the data, respectively.

[0046] 2. LSTM prediction model; (1) Input Definition ; Where τ is the length of the time window, and each row corresponds to the three-dimensional load data of a time step, namely CPU utilization, memory utilization and disk space utilization.

[0047] (2) Output prediction The model predicts CPU, memory, and disk space utilization over the next T time steps: ; 3. LSTM network structure; (3.1) Mathematical model of LSTM unit; The LSTM unit handles long-term dependencies in sequences through a gating mechanism, whose core formula is as follows: Input Gate: controls the writing of new information ; Forget Gate: controls the preservation of old information ; Output Gate: controls the output of the memory cell ; Candidate Cell: calculates the new candidate memory ; Memory Cell Update: combines the results of the Forget and Input Gates ; Hidden State Update: generates a new hidden state based on the Output Gate and Memory Cell ; where σ is the Sigmoid function, ⊙ is the element-wise multiplication, W, U, b are the weight and bias matrices, Ht is the current hidden state, and Ct is the memory cell state.

[0048] (3.2) Multi-layer LSTM structure; To enhance the model's ability to capture complex patterns, multiple layers can be stacked ; where l represents the number of layers, and the hidden state of the upper layer is used as the input of the lower layer.

[0049] (3.3) Output layer; The hidden state of the last layer of LSTM is mapped to the predicted value through a fully connected layer: where W o and b o are the weights and biases of the output layer.

[0050] 4、Q-learning decision model; (4.1) State space design The state vector includes the current load and the predicted load, and the space state vector is: ; Current load state: real-time CPU and memory utilization ; where N is the number of nodes.

[0051] Predicted load state: future load prediction value (take future 1 time step) output by LSTM: ; Task queue state: number of pending tasks and task type TYPE (e.g. CPU-intensive or memory-intensive).

[0052] (4.2) Action space definition; where a i represents assigning a task to node i.

[0053] (4.3) Reward function design; The reward function of the system is: ; Task completion time reward: the shorter the task completion time on node i , the higher the reward: ; where T min and T max are the experienced extreme values of task completion time, and a is the weight; Resource utilization reward: reward the CPU and memory utilization of node i close to target values CPU target andMemtarget: ; where b is the weight.

[0054] Load balancing reward: reward the overall load balancing of the system, the smaller the standard deviation the better ; is the comprehensive utilization rate of node load ∈[0,1] is the weight, and satisfies the following constraints ; Overload penalty term: ; is the overload threshold, and is the indicator function.

[0055] (4.3) Q value update mechanism; Q-learning updates the state-action value Q(s t ,a t ) through the Bellman equation: ; wherein: η is the learning rate, controlling the update step size; γ is the discount factor, balancing short-term and long-term rewards; Q is the optimal action value for the next state.

[0056] II. Distributed federated collaborative management; The distributed virtual collaborative manager decouples data storage and data access paths by using the modal arranger and the configuration manager, realizes unified management between different modal data, and realizes distributed storage and collaborative management of different clusters of the same mode through virtual implementation of the virtual data warehouse. The architecture is shown in Figure 5 .

[0057] Figure 5 The distributed virtual collaborative manager shown in the figure is mainly composed of a virtual data warehouse, a heterogeneous cluster modal arranger and a configuration manager core component, and its main functions are as follows: Virtual data warehouse: The virtual data warehouse accepts the input of the modal manager and manages different clusters of the same mode. It uses different virtual instances to encapsulate different cpu, memory and other resources, and reports the resources to the resource manager. At the same time, through the element data converter, different virtual instances are persisted to the metadata manager. The resource manager reports the resources of different clusters to the intelligent router for predicting load.

[0058] Configuration manager: The configuration manager is mainly used for mapping of the virtual data warehouse and different modal of the heterogeneous cluster.

[0059] Heterogeneous cluster modal arranger: Real-time monitoring of the survival state of multiple cluster nodes (heartbeat mechanism), unified arrangement and dynamic scheduling of computing, storage and other resources of heterogeneous clusters, and construction of standardized metadata warehouse to maintain the description information of cross-cluster data assets (such as cluster topology, resource attributes, data lineage, etc.). The system adopts modular architecture design, supports multiple independent instances to take over different types (such as computing cluster, storage cluster, container cluster, etc.) of special data management tasks, so as to realize cross-cluster resource coordination and metadata collaborative governance in distributed environment.

[0060] Through the description of the specific embodiments, the technical means and effects adopted by the present application to achieve the predetermined purpose can be more deeply and specifically understood. However, the accompanying drawings are provided for reference and explanation only, and are not used to limit the present application.

Claims

1. A distributed collaborative management system based on a federated learning architecture, characterized in that, include: A load balancer is used to receive user requests and perform load balancing scheduling of intelligent routing nodes. The intelligent predictor is used to monitor the load of each heterogeneous cluster in real time and perform load prediction using an LSTM prediction model. The distributed router is used to determine intelligent routing nodes based on load balancing scheduling results, perform syntax parsing and semantic verification of user requests, and balance the load of data elements based on load prediction results and the Q-learning algorithm, thereby determining the virtual data warehouse and the corresponding heterogeneous cluster.

2. The distributed collaborative management system based on federated learning architecture according to claim 1, characterized in that, The system also includes a virtual collaboration manager, which is used to realize the mapping between the virtual data warehouse and the heterogeneous cluster, and to realize the distributed storage and collaborative management of different heterogeneous clusters with the same modality.

3. The distributed collaborative management system based on federated learning architecture according to claim 2, characterized in that, The virtual collaboration manager includes: Multiple virtual data warehouses, each managing different clusters of the same modality, utilizing different virtual instances to encapsulate different resources; The configuration manager is used to map different modalities between the virtual data warehouse and the heterogeneous cluster; The heterogeneous cluster modal orchestrator is used to monitor the liveness status of heterogeneous clusters in real time, perform unified orchestration and dynamic scheduling of resources in heterogeneous clusters, and build a standardized metadata repository.

4. The distributed collaborative management system based on federated learning architecture according to claim 1, characterized in that, The distributed router includes: The SQL parser is used to perform syntax parsing and semantic validation of user requests. The component modal parser is used to extract modal attributes from user requests and obtain the mapping relationship between user requests and the virtual data warehouse. The intelligent routing algorithm module is used to balance the load of data elements based on the load prediction results of the LSTM prediction model and the Q-learning algorithm, and to determine the virtual data warehouse and the corresponding heterogeneous group.

5. The distributed collaborative management system based on a federated learning architecture according to any one of claims 1-4, characterized in that, The load balancer uses at least one of the round-robin algorithm, random algorithm, and adaptive load algorithm to perform load balancing scheduling of routing nodes.

6. A distributed collaborative management method based on a federated learning architecture, characterized in that, include: A10 receives user requests and performs load balancing scheduling of intelligent routing nodes. A20, and monitors the load of each heterogeneous cluster in real time, and performs load prediction using an LSTM prediction model; The A30 determines intelligent routing nodes based on load balancing scheduling results, performs syntax parsing and semantic verification on user requests, and balances the load of data elements based on load prediction results and the Q-learning algorithm, thereby determining the virtual data warehouse and the corresponding heterogeneous cluster.

7. The distributed collaborative management method based on a federated learning architecture according to claim 6, characterized in that, Before receiving a user request, the method further includes: performing a mapping between the virtual data warehouse and the heterogeneous cluster.

8. The distributed collaborative management method based on a federated learning architecture according to claim 7, characterized in that, Before step A20, the method further includes training the LSTM prediction model, specifically including: A21: Obtain historical load data of the virtual data warehouse, including CPU utilization, memory utilization, and disk space utilization, and perform normalization processing to obtain training data. A22, the LSTM prediction model is trained using the training data to output load data for future time periods.

9. The distributed collaborative management method based on a federated learning architecture according to claim 8, characterized in that, In step A30, the spatial state vector of the Q-learning algorithm is: ； in, This represents the current load status. To predict load status, This indicates the task queue status, and TYPE represents the task type.

10. The distributed collaborative management method based on a federated learning architecture according to any one of claims 6-9, characterized in that, In step A10, at least one of the round-robin algorithm, random algorithm, and adaptive load algorithm is used to perform load balancing scheduling of routing nodes.

Images