Distributed real-time database intelligent management method and system based on big data

Abstract

This invention provides a distributed real-time database intelligent management method and system based on big data, belonging to the field of database management technology. The method includes predicting resource requirements using a neural network model, selecting target data nodes, generating a data sharding mapping table using a ring data structure, and constructing a sharding routing table to realize the writing and migration of data shards. After receiving data processing tasks, the method determines the data distribution according to the sharding routing table, splits subtasks and allocates them to target data nodes based on priority, monitors resource occupancy and reallocates tasks when necessary, and finally generates data processing results. This method improves database management efficiency and resource utilization.

Description

Technical Field

[0001] The present invention relates to database management technology, and in particular to a distributed real-time database intelligent management method and system based on big data. Background Art

[0002] Traditional database management methods lack intelligence and adaptability. They usually adopt fixed resource allocation strategies and cannot dynamically adjust resource allocation according to the characteristics of data processing requests and system load, resulting in low resource utilization and failure to meet the needs of large-scale real-time data processing.

[0003] Existing data sharding and routing technologies have limitations. Most methods use simple hash algorithms for data sharding, which is difficult to deal with uneven data distribution and dynamic changes in nodes. At the same time, the lack of an effective load balancing mechanism can easily cause some nodes to be overloaded while other nodes are idle.

[0004] Current task scheduling and execution strategies are relatively rigid. They often use static task allocation methods and cannot be flexibly scheduled according to task priorities and dependencies. In addition, there is a lack of real-time monitoring and dynamic adjustment mechanisms for task execution, making it difficult to ensure system stability and performance. Summary of the invention

[0005] The embodiments of the present invention provide a distributed real-time database intelligent management method and system based on big data, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provided is a distributed real-time database intelligent management method based on big data, comprising:

[0008] Receive a data processing request, extract a data type identifier, a data processing rule, data scale information, and a data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain a computing resource demand prediction value and a storage resource demand prediction value for data processing, call a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and select a data node whose node load value is lower than a first preset threshold as a target data node;

[0009] Divide the data into shards on the target data node, generate a data shard mapping table using a ring data structure, construct a virtual node by polynomial calculation, assign a unique identifier to each data shard, store the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table, write the data shard into the corresponding target data node according to the shard routing table, collect the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, select the data node with the lowest node load value as the target migration node, migrate the data shard to the target migration node, and update the shard routing table;

[0010] Receive a data processing task, determine the distribution location of the data shard according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtasks to the corresponding target data nodes based on the priority, monitor the resource occupancy rate of the subtasks, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result.

[0011] In an optional embodiment,

[0012] Calling a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and selecting a data node whose node load value is lower than a first preset threshold as a target data node includes:

[0013] Collect computing resource usage data and storage resource usage data of data nodes, extract features of computing resource usage data and storage resource usage data respectively, and fuse them to generate data node load features;

[0014] The data node load characteristics are segmented using an adaptive time window, the window size of the adaptive time window is determined according to the fluctuation period of the data node load characteristics, the segmented data node load characteristics are input into a timing analysis network and multi-scale decomposition is performed to obtain a load evolution law;

[0015] The load evolution law is input into a data node evaluation model, the data node evaluation model uses a deep temporal network to extract abnormal features, and calculates the influence of the abnormal features based on an attention mechanism to obtain the health of the data node;

[0016] A multi-level adaptive threshold system is constructed based on the load evolution law and the health of the data node. The multi-level adaptive threshold system includes a main threshold model and an auxiliary threshold model. The main threshold model determines a reference threshold according to the probability distribution of the data node load characteristics. The auxiliary threshold model realizes dynamic adjustment of the threshold based on a reinforcement learning method. The reference threshold and the dynamically adjusted threshold are combined to obtain a first preset threshold.

[0017] The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into a data node load calculation module, and the data node load calculation module uses a multi-layer computing framework to calculate the node load value;

[0018] The node load value is compared with the first preset threshold value, and based on the comparison result and the health of the data node, a node sorting algorithm is used to select a data node with a node load value lower than the first preset threshold value and the highest health as the target data node.

[0019] In an optional embodiment,

[0020] The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into the data node load calculation module, and the data node load calculation module uses a multi-layer calculation framework to calculate the node load value including:

[0021] Establish a multi-dimensional load feature extraction module, which performs time series decomposition on the computing resource demand prediction value, the storage resource demand prediction value and the load history data to obtain trend components, periodic components and random components, and inputs the trend components, periodic components and random components into an adaptive standardization module, which dynamically selects a standardization strategy according to data distribution characteristics, and performs adaptive standardization processing on the input data to obtain a standardized feature vector;

[0022] Constructing a multi-head attention network to extract the feature correlation of the standardized feature vector, inputting the feature correlation into a feature fusion layer to generate a feature weight matrix, and optimizing the feature weight matrix using an adaptive optimization algorithm to obtain an optimized feature weight matrix;

[0023] The standardized feature vector and the optimized feature weight matrix are input into the first computing layer, the first computing layer uses a bidirectional long short-term memory network to extract time series features, and performs weighted combination of the standardized feature vectors through an attention mechanism to obtain an initial load value;

[0024] The initial load value is input into a second computing layer, and the second computing layer uses a residual connection structure and a dilated convolution to enhance the features of the initial load value, and highlights key load features through a self-attention mechanism to obtain enhanced load features;

[0025] The enhanced load feature and the historical load time series are input into a third computing layer, the third computing layer uses a gated cycle unit to extract a timing pattern, and adaptively fuses the timing pattern and the enhanced load feature through a dynamic weight coefficient to obtain a fused load value;

[0026] The fused load value is input into a fourth computing layer, and the fourth computing layer performs a nonlinear transformation on the fused load value through a multi-layer perceptron, and obtains an initial node load value in combination with a batch normalization operation;

[0027] The multidimensional error index between the initial node load value and the actual load value is calculated. When the multidimensional error index exceeds a preset threshold, the network parameters of the multilayer perceptron are smoothly optimized using the exponential moving average method to obtain the optimized network parameters. The optimized network parameters are applied to the fourth calculation layer, and the fused load value is recalculated to obtain the final node load value.

[0028] In an optional embodiment,

[0029] Storing the correspondence between the unique identifier of the data shard and the target data node in a shard routing table, writing the data shard to the corresponding target data node according to the shard routing table, collecting the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, selecting the data node with the lowest node load value as the target migration node includes:

[0030] Perform consistent hashing on the unique identifier of the data shard to obtain a hash value, map the hash value clockwise to a ring data structure, and use a binary search algorithm to locate the nearest target data node in the skip list of the ring data structure;

[0031] Extract the node identifier of the target data node, construct a mapping relationship between the unique identifier of the data shard and the node identifier of the target data node, encapsulate the mapping relationship into a routing record, and organize the routing record in a tree structure to generate a shard routing table;

[0032] Constructing a data transmission buffer pool, storing the data slices in the data transmission buffer pool, determining a transmission target based on the slice routing table, creating an asynchronous transmission thread, and transmitting the data slices from the data transmission buffer pool to a target data node corresponding to the slice routing table;

[0033] A sliding time window is used to monitor the resource usage of the target data node, and the processor usage rate, memory usage rate, disk read / write rate and network bandwidth usage rate of the target data node are collected as resource indicators. The resource indicators are subjected to principal component analysis to obtain resource indicator characteristics, and the node load value of the target data node is calculated based on the resource indicator characteristics.

[0034] When it is detected that the node load value exceeds a second preset threshold value within a specified number of consecutive time windows, a confidence interval of the load anomaly is calculated based on the load anomaly mark, and the confidence interval is input into a pre-trained trend prediction model to obtain a predicted load value within a future time window;

[0035] Construct a load balancing scoring model, take the node load value, fluctuation variance, and network delay as scoring indicators, calculate the weight vector of the scoring indicator based on the hierarchical analysis method, and take the product of the scoring indicator and the weight vector as the node score value;

[0036] An optimization algorithm is used to optimize the node score values ​​of all data nodes to obtain an optimal load balancing solution, and the data node with the highest score value is selected from the optimal load balancing solution as the target migration node.

[0037] In an optional embodiment,

[0038] The node score values ​​of all data nodes are optimized by using an optimization algorithm to obtain an optimal load balancing solution, and the data node with the highest score value is selected as the target migration node from the optimal load balancing solution, including:

[0039] Constructing a multi-layer neural network to extract node features, wherein the input layer of the multi-layer neural network receives a processor utilization time series, a memory occupancy time series, and an input and output load time series, and a skip connection structure and a batch normalization layer are set in the hidden layer to generate a node feature vector through an activation function;

[0040] Optimizing the hidden state of the multi-layer neural network, calculating the attention weight matrix and the feature increment matrix, and weighted combining the feature increment matrix with the original hidden state matrix to obtain an optimized node hidden state;

[0041] The node load sequence is obtained based on a dynamic sliding window mechanism. The size of the sliding window is adaptively adjusted according to the load fluctuation degree. The query matrix, key matrix and value matrix of the multi-head attention module are constructed, and the attention weight is calculated to obtain the node time series characteristics.

[0042] Input the node time series features into the multi-head attention layer to generate multiple attention head features, fuse the multiple attention head features using nonlinear transformation, and output the node fusion feature vector;

[0043] Construct a similarity calculation module based on graph neural network, represent data nodes as nodes in the graph structure, calculate the similarity matrix between nodes based on the node fusion feature vector, and generate the initial node score based on the topological relationship between the nodes;

[0044] Design a scoring optimization model with a dual neural network architecture, including a scoring strategy network and a value assessment network, which takes the node initial score, system state vector and historical score sequence as input and outputs the node score adjustment strategy;

[0045] The entropy weight method is used to calculate the weight coefficient of the node load index, and the weight coefficient is combined with the score adjustment strategy to perform multi-objective optimization on the initial node score to obtain the optimized node score value. At the same time, the federated learning framework is used to design a distributed score optimization scheme. Each node trains a local score optimization model locally, and a secure aggregation algorithm is used to fuse the local score optimization model parameters to construct a global score optimization strategy.

[0046] According to the global scoring optimization strategy, combined with the historical load level, resource utilization and network performance indicators of the nodes, the scoring values ​​of all data nodes are comprehensively optimized to generate an optimal load balancing solution. Based on the optimal load balancing solution, the data node with the highest scoring value and meeting the migration constraints is selected as the target migration node.

[0047] In an optional embodiment,

[0048] Receiving a data processing task, determining the distribution location of the data shard according to the shard routing table, splitting the data processing task into multiple subtasks, constructing a dependency graph of the subtasks, calculating the priority of each subtask according to the dependency graph, and allocating the subtasks to corresponding target data nodes based on the priority includes:

[0049] Receive a data processing task, parse the input data set, operation operator sequence and result output specification in the data processing task, construct a task feature vector, input the task feature vector into a preset resource demand prediction model, obtain the computing resource demand value and storage resource demand value of the data processing task, find the location information of each data shard in the input data set based on the shard routing table, and collect the current resource status of each data node;

[0050] Constructing a data affinity matrix, the data affinity matrix records the position overlap and access association between different data shards, performing data shard clustering based on the data affinity matrix, dividing the data shards with affinities higher than a preset affinity threshold into the same data group, and combining the calculation characteristics of the operation operator sequence and the distribution characteristics of the data group to split the data processing task into multiple subtasks;

[0051] Construct a two-layer dependency graph, in which the first-layer nodes of the two-layer dependency graph represent subtasks, and the second-layer nodes represent data shards. The two-layer dependency graph describes the execution dependency relationship and data access relationship between subtasks. Based on the two-layer dependency graph, a critical path algorithm is used to calculate the execution priority of the subtask. The execution priority is determined based on the computational overhead and data access overhead of the subtask. A task priority queue is established, and subtasks that meet the execution conditions are added to the task priority queue in the order of the execution priority.

[0052] Calculate the position matching degree between the subtask and the data node in the task priority queue, where the position matching degree is determined based on the data access scope of the subtask and the data distribution of the data node, and assign the subtask whose position matching degree is higher than a preset matching degree threshold and meets the computing resource requirement value and the storage resource requirement value to the corresponding target data node.

[0053] In an optional embodiment,

[0054] Allocating subtasks whose location matching degree is higher than a preset matching degree threshold and meet the computing resource requirement value and the storage resource requirement value to the corresponding target data node includes:

[0055] Acquire data access information of the task to be assigned and storage status information of the data node, wherein the data access information includes a data shard identifier that the task to be assigned needs to access and a corresponding access operation type, and the storage status information includes a data shard identifier currently stored by the data node;

[0056] Based on the data access information and the storage status information, the number of intersections of the data shard set accessed by the to-be-assigned task and the data shard set stored by the data node is calculated to obtain the data shard overlap;

[0057] Obtain the weight coefficients of the read operation and the write operation in the task to be assigned, calculate the weighted access matching degree based on the weight coefficient and the data shard overlap, and combine the data shard overlap and the weighted access matching degree according to a preset mixing coefficient to obtain the position matching degree;

[0058] Obtain computing resource requirement information of the task to be assigned and available resource information of the data node, calculate resource satisfaction based on the computing resource requirement information and the available resource information, filter out candidate data nodes that meet the resource requirements according to the resource satisfaction, and select the node with the highest position matching degree among the candidate data nodes as the target data node.

[0059] According to a second aspect of the embodiments of the present invention,

[0060] Provided is a distributed real-time database intelligent management system based on big data, including:

[0061] The first unit is used to receive a data processing request, extract a data type identifier, a data processing rule, data scale information and a data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain a computing resource demand prediction value and a storage resource demand prediction value for data processing, call a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and select a data node whose node load value is lower than a first preset threshold as a target data node;

[0062] The second unit is used to divide the data shards on the target data node, generate a data shard mapping table using a ring data structure, construct a virtual node through polynomial calculation, assign a unique identifier to each data shard, store the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table, write the data shards into the corresponding target data nodes according to the shard routing table, collect the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, select the data node with the lowest node load value as the target migration node, migrate the data shards to the target migration node, and update the shard routing table;

[0063] The third unit is used to receive a data processing task, determine the distribution location of the data shard according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtask to the corresponding target data node based on the priority, monitor the resource occupancy rate of the subtask, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result.

[0064] According to a third aspect of the embodiments of the present invention,

[0065] An electronic device is provided, comprising:

[0066] processor;

[0067] a memory for storing processor-executable instructions;

[0068] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0069] A fourth aspect of the embodiments of the present invention is:

[0070] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0071] In this embodiment, the resource requirements of data processing are predicted by a neural network model, and the appropriate target data node is selected based on the node load value, thereby realizing intelligent resource allocation. The data sharding is managed by a ring data structure and a sharding routing table, and combined with a dynamic load monitoring and data migration mechanism, the load balancing of the system is effectively realized, and the overall performance and resource utilization are improved. The data processing task is split into subtasks, a dependency graph is constructed, and tasks are assigned based on priority, thereby realizing flexible and efficient task scheduling. By monitoring the resource occupancy rate of subtasks in real time and redistributing tasks when necessary, the stability and efficiency of task execution are guaranteed, and the problem of single node overload is effectively avoided. The scalability of the system is improved by adopting a distributed architecture and data sharding technology. The fault tolerance and reliability of the system are enhanced by dynamic data migration and task redistribution mechanisms. This design enables the system to better cope with large-scale data processing needs, and can quickly adjust when a node fails or the load is too high, thereby ensuring the continuity of the service and the stability of performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of a flow chart of a distributed real-time database intelligent management method based on big data according to an embodiment of the present invention;

[0073] Figure 2 It is a structural diagram of a distributed real-time database intelligent management system based on big data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0076] Figure 1 FIG. 1 is a flow chart of a distributed real-time database intelligent management method based on big data according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0077] S101. Receive a data processing request, extract the data type identifier, data processing rules, data scale information and data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain the computing resource demand prediction value and the storage resource demand prediction value for data processing, call the data node load calculation module to calculate the node load value based on the computing resource demand prediction value and the storage resource demand prediction value, and select the data node whose node load value is lower than the first preset threshold as the target data node.

[0078] First, a data processing request is received, and the data type identifier, data processing rules, data scale information, and data priority are extracted from the request. A data feature vector is constructed based on the data type identifier, and the feature vector is input into a preset neural network model to obtain the computing resources and storage resource demand forecast values ​​required for data processing.

[0079] Specifically, the data type identifier may include information such as data structure type (such as structured, semi-structured, unstructured), data format (such as text, image, audio, video), etc. Data processing rules may include operations such as data cleaning, conversion, and aggregation. Data scale information includes data volume, growth rate, etc. Data priority is used to identify the urgency of data processing.

[0080] Taking text data as an example, features such as word frequency, part-of-speech distribution, and text length can be extracted to construct feature vectors. For image data, color distribution, texture features, edge features, etc. can be extracted. The neural network model uses a multi-layer perceptron structure, with the input layer corresponding to the feature vector dimension, the hidden layer using the ReLU activation function, and the output layer corresponding to two predicted values ​​of computing resources and storage resources.

[0081] Next, the data node load calculation module is called to calculate the load value of each node. The load value calculation takes into account factors such as CPU utilization, memory usage, and disk I / O. Nodes with load values ​​lower than a preset threshold (such as 60%) are selected as target data nodes.

[0082] In an optional embodiment,

[0083] The computing resource demand prediction value and the storage resource demand prediction value, calling a data node load calculation module to calculate a node load value, and selecting a data node whose node load value is lower than a first preset threshold as a target data node comprises:

[0084] Collect computing resource usage data and storage resource usage data of data nodes, extract features of computing resource usage data and storage resource usage data respectively, and fuse them to generate data node load features;

[0085] The data node load characteristics are segmented using an adaptive time window, the window size of the adaptive time window is determined according to the fluctuation period of the data node load characteristics, the segmented data node load characteristics are input into a timing analysis network and multi-scale decomposition is performed to obtain a load evolution law;

[0086] The load evolution law is input into a data node evaluation model, the data node evaluation model uses a deep temporal network to extract abnormal features, and calculates the influence of the abnormal features based on an attention mechanism to obtain the health of the data node;

[0087] A multi-level adaptive threshold system is constructed based on the load evolution law and the health of the data node. The multi-level adaptive threshold system includes a main threshold model and an auxiliary threshold model. The main threshold model determines a reference threshold according to the probability distribution of the data node load characteristics. The auxiliary threshold model realizes dynamic adjustment of the threshold based on a reinforcement learning method. The reference threshold and the dynamically adjusted threshold are combined to obtain a first preset threshold.

[0088] The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into a data node load calculation module, and the data node load calculation module uses a multi-layer computing framework to calculate the node load value;

[0089] The node load value is compared with the first preset threshold value, and based on the comparison result and the health of the data node, a node sorting algorithm is used to select a data node with a node load value lower than the first preset threshold value and the highest health as the target data node.

[0090] Exemplarily, first collect the computing resource and storage resource usage data of the data node, extract features and fuse them to generate data node load features. Then use the adaptive time window to segment the load features, input the timing analysis network for multi-scale decomposition to obtain the load evolution law. Then input the load evolution law into the data node evaluation model, and calculate the health of the data node. Based on the load evolution law and health, a multi-level adaptive threshold system is constructed, including a main threshold model and an auxiliary threshold model, to obtain a first preset threshold. Finally, the resource demand prediction value and the load characteristics are input into the load calculation module to obtain the node load value, which is compared with the first preset threshold and combined with the health to select the optimal target data node.

[0091] First, collect the computing resource usage data and storage resource usage data of the data node. The computing resource usage data includes CPU utilization, memory utilization, etc.; the storage resource usage data includes disk utilization, IO read and write speed, etc. The collected raw data is preprocessed, including denoising, normalization and other operations. Then, the time series features, frequency domain features and statistical features of the computing resource and storage resource usage data are extracted respectively, and feature fusion is performed to generate a load feature vector that describes the overall load of the data node.

[0092] Next, the load characteristics of the data nodes are segmented using an adaptive time window. First, the autocorrelation function of the load characteristics is calculated, and the fluctuation period T of the load characteristics is determined based on the periodicity of the autocorrelation function. The time window size is set to kT, where k is a positive integer, and the optimal k value is selected through experiments. The load characteristics are segmented using the sliding window method, with a window size of kT and a step size of T / 2. The segmented load characteristic sequence is input into a timing analysis network constructed by a long short-term memory network (LSTM) for multi-scale wavelet decomposition to obtain the load change trend at different time scales, that is, the load evolution law.

[0093] Then, the obtained load evolution law is input into the data node evaluation model. The model adopts a deep temporal network structure, which includes a multi-layer convolutional neural network and an LSTM network. First, the local features in the load evolution law are extracted using the convolution layer, and then input into the LSTM network to capture the long-term dependencies. The attention mechanism is added to the output layer of the network to calculate the importance weights of features at different time steps. The abnormal features are weighted and summed according to the weights to obtain the health index that reflects the health status of the data node. The higher the health, the more stable the node operation.

[0094] Next, a multi-level adaptive threshold system is constructed based on the load evolution law and the health of data nodes. The main threshold model determines the benchmark threshold according to the probability distribution of the load characteristics. First, the kernel density of the load characteristics is estimated to obtain its probability density function. Then the cumulative distribution function is calculated, and the load value corresponding to the cumulative probability of 95% is selected as the benchmark threshold. The auxiliary threshold model uses the Deep Q-Network (DQN) reinforcement learning algorithm to achieve dynamic adjustment of the threshold. Load characteristics, health, etc. are used as state inputs, and the threshold adjustment range is used as action output. A reward function is constructed to guide the model to learn the optimal adjustment strategy. Finally, the benchmark threshold is combined with the dynamically adjusted threshold to obtain the first preset threshold.

[0095] Subsequently, the predicted computing resource demand value, storage resource demand predicted value and data node load characteristics are input into the data node load calculation module. This module adopts a multi-layer computing framework, including a resource demand layer, a load characteristic layer and a comprehensive evaluation layer. In the resource demand layer, the predicted computing and storage resource requirements are matched with the currently available resources of the node. In the load characteristic layer, the statistics of the load characteristics are extracted and normalized. In the comprehensive evaluation layer, the calculation results of the first two layers are input into the fully connected neural network to obtain the load value reflecting the overall load of the node.

[0096] Finally, the calculated node load value is compared with the first preset threshold. The improved Top-K algorithm is used to sort the nodes. First, a subset of nodes with load values ​​below the threshold is screened out, and then the nodes in the subset are sorted in descending order according to the node health, and the top-ranked node is selected as the target data node. If there is no node with a load value below the threshold, the node with the lowest load value and the highest health is selected.

[0097] In this embodiment, the load of data nodes is fully characterized through multi-dimensional feature extraction and fusion, which improves the accuracy of load calculation. The introduction of adaptive time window and multi-scale decomposition captures the time-varying characteristics of the load and effectively extracts the load evolution law. A multi-level adaptive threshold system is constructed to achieve dynamic adjustment of the threshold and improve the flexibility and robustness of node selection.

[0098] In an optional embodiment,

[0099] The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into the data node load calculation module, and the data node load calculation module uses a multi-layer calculation framework to calculate the node load value including:

[0100] Establish a multi-dimensional load feature extraction module, which performs time series decomposition on the computing resource demand prediction value, the storage resource demand prediction value and the load history data to obtain trend components, periodic components and random components, and inputs the trend components, periodic components and random components into an adaptive standardization module, which dynamically selects a standardization strategy according to data distribution characteristics, and performs adaptive standardization processing on the input data to obtain a standardized feature vector;

[0101] Constructing a multi-head attention network to extract the feature correlation of the standardized feature vector, inputting the feature correlation into a feature fusion layer to generate a feature weight matrix, and optimizing the feature weight matrix using an adaptive optimization algorithm to obtain an optimized feature weight matrix;

[0102] The standardized feature vector and the optimized feature weight matrix are input into the first computing layer, the first computing layer uses a bidirectional long short-term memory network to extract time series features, and performs weighted combination of the standardized feature vectors through an attention mechanism to obtain an initial load value;

[0103] The initial load value is input into a second computing layer, and the second computing layer uses a residual connection structure and a dilated convolution to enhance the features of the initial load value, and highlights key load features through a self-attention mechanism to obtain enhanced load features;

[0104] The enhanced load feature and the historical load time series are input into a third computing layer, the third computing layer uses a gated cycle unit to extract a timing pattern, and adaptively fuses the timing pattern and the enhanced load feature through a dynamic weight coefficient to obtain a fused load value;

[0105] The fused load value is input into a fourth computing layer, and the fourth computing layer performs a nonlinear transformation on the fused load value through a multi-layer perceptron, and obtains an initial node load value in combination with a batch normalization operation;

[0106] The multidimensional error index between the initial node load value and the actual load value is calculated. When the multidimensional error index exceeds a preset threshold, the network parameters of the multilayer perceptron are smoothly optimized using the exponential moving average method to obtain the optimized network parameters. The optimized network parameters are applied to the fourth calculation layer, and the fused load value is recalculated to obtain the final node load value.

[0107] Exemplarily, first, a multi-dimensional load feature extraction module is established. This module performs time series decomposition on the computing resource demand forecast value, storage resource demand forecast value and load history data to obtain trend components, periodic components and random components. Time series decomposition uses seasonal decomposition method to decompose the original time series into three parts: trend, seasonality and randomness. Taking the computing resource demand forecast value as an example, assuming that the original sequence is [100, 120, 110, 130, 115, 135, 125, 145], after time series decomposition, we get:

[0108] Trend component: [105, 110, 115, 120, 125, 130, 135, 140];

[0109] Periodic component: [-5, 10, -5, 10, -10, 5, -10, 5];

[0110] Random component: [0, 0, 0, 0, 0, 0, 0, 0];

[0111] Then the trend component, periodic component and random component are input into the adaptive standardization module. This module dynamically selects the standardization strategy according to the data distribution characteristics, and performs adaptive standardization on the input data to obtain the standardized feature vector. Specifically, the mean and standard deviation of each component are first calculated, and then the appropriate standardization method is selected according to the data distribution. For data with approximately normal distribution, Z-Score standardization is used; for skewed distribution data, logarithmic transformation is used before standardization; for data with outliers, robust standardization is used. Taking the trend component as an example, the calculated mean is 122.5 and the standard deviation is 12.5. The standardized trend component obtained by Z-Score standardization is: [-1.4, -1.0, -0.6, -0.2, 0.2, 0.6, 1.0, 1.4].

[0112] Next, a multi-head attention network is constructed to extract the feature correlation of the standardized feature vector. The multi-head attention network contains multiple attention heads, and each attention head independently learns the feature correlation of different subspaces. Taking 8 heads as an example, each head outputs an 8-dimensional correlation vector, and finally a 64-dimensional feature correlation vector is obtained. The feature correlation is input into the feature fusion layer to generate a feature weight matrix, and the Adam optimization algorithm is used to optimize the feature weight matrix to obtain the optimized feature weight matrix.

[0113] The standardized feature vector and the optimized feature weight matrix are input into the first computing layer. The first computing layer uses a bidirectional long short-term memory network (BiLSTM) to extract time series features. BiLSTM contains two LSTMs, forward and backward, which can capture bidirectional time series dependencies. Taking 128 hidden units as an example, BiLSTM outputs a 256-dimensional feature vector. Then, the standardized feature vector is weighted and combined through the attention mechanism to obtain the initial load value. The attention weight is calculated by the softmax function, and the corresponding feature vector is weighted and summed to obtain the initial load value.

[0114] The initial load value is input into the second computing layer. The second computing layer uses a residual connection structure and a dilated convolution to enhance the features of the initial load value. The residual connection structure contains a shortcut connection, which helps alleviate the gradient vanishing problem. The dilated convolution uses different dilation rates to expand the receptive field to obtain multi-scale features. Taking three convolutional layers as an example, the dilation rates are 1, 2, and 4, the convolution kernel size is 3x3, and the number of output channels is 64. Then, the self-attention mechanism is used to highlight the key load features to obtain enhanced load features. The self-attention mechanism calculates the similarity between features and gives higher weights to important features.

[0115] The enhanced load features and the historical load time series are input into the third computing layer. The third computing layer uses a gated recurrent unit (GRU) to extract the timing pattern. GRU contains an update gate and a reset gate, which can effectively model long-term dependencies. Taking 64 hidden units as an example, GRU outputs a 64-dimensional timing feature vector. Then, the timing pattern and the enhanced load features are adaptively fused through the dynamic weight coefficient to obtain the fused load value. The dynamic weight coefficient is calculated through the fully connected layer, and the timing pattern and the enhanced load features are weighted and summed.

[0116] The fused load value is input into the fourth computing layer. The fourth computing layer performs nonlinear transformation on the fused load value through a multilayer perceptron. The multilayer perceptron contains three hidden layers, with 128, 64, and 32 neurons in each layer. The ReLU activation function is used to introduce nonlinearity, and the initial node load value is obtained by combining the batch normalization operation. Batch normalization helps alleviate the problem of internal covariate shift and accelerate network convergence.

[0117] Calculate the multidimensional error index between the initial node load value and the actual load value. The multidimensional error index includes the mean absolute error (MAE), the root mean square error (RMSE), and the mean absolute percentage error (MAPE). When the multidimensional error index exceeds the preset threshold, the exponential moving average method is used to smoothly optimize the network parameters of the multilayer perceptron. Specifically, the exponential moving average of the parameters is calculated with an update frequency of 0.999. The optimized network parameters are applied to the fourth calculation layer, and the fused load value is recalculated to obtain the final node load value.

[0118] In this embodiment, multi-dimensional load feature extraction and adaptive normalization processing are adopted to effectively extract the multi-dimensional features of the load data, and the appropriate normalization strategy is dynamically selected according to the data distribution characteristics, thereby improving the feature expression ability and model robustness. A multi-head attention network and a multi-layer computing framework are constructed, which fully utilize the temporal correlation and feature relevance of the load data. The feature extraction ability of the model is enhanced through structures such as residual connection and void convolution, thereby improving the accuracy of load prediction. Dynamic weight fusion and adaptive optimization mechanisms are introduced, which can adaptively fuse according to the importance of different features, and dynamically optimize the model parameters through error feedback, thereby improving the generalization ability and prediction stability of the model.

[0119] S102. Divide the data into shards on the target data node, and generate a data shard mapping table using a ring data structure. The ring data structure constructs virtual nodes through polynomial calculations, assigns a unique identifier to each data shard, and stores the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table. According to the shard routing table, the data shard is written into the corresponding target data node, and the node load value of the target data node is collected based on the shard routing table. When it is detected that the node load value of the target data node exceeds a second preset threshold, the data node with the lowest node load value is selected as the target migration node, the data shard is migrated to the target migration node, and the shard routing table is updated.

[0120] Divide the data into shards on the target data node, and use the consistent hashing algorithm to build a ring data structure. Specifically, the integer space from 0 to 2^32-1 is regarded as a ring connected end to end, and the IP address or unique identifier of each data node is hashed, and the result is mapped to a point on the ring, which is the location of the node. In order to improve the uniformity of data distribution, the concept of virtual nodes is introduced, that is, each physical node is mapped to multiple virtual nodes through polynomial calculation.

[0121] Assign a unique identifier to each data shard, which can be generated using the UUID algorithm. Store the unique identifier of the data shard and the corresponding target data node in the shard routing table. Write the data shard to the corresponding target data node according to the shard routing table. Collect the load value of the target data node regularly. When it is detected that the load value of a node exceeds the preset threshold (such as 80%), select the node with the lowest current load value as the migration target, migrate part of the data shard to this node, and update the shard routing table.

[0122] In an optional embodiment,

[0123] Storing the correspondence between the unique identifier of the data shard and the target data node in a shard routing table, writing the data shard to the corresponding target data node according to the shard routing table, collecting the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, selecting the data node with the lowest node load value as the target migration node includes:

[0124] Perform consistent hashing on the unique identifier of the data shard to obtain a hash value, map the hash value clockwise to a ring data structure, and use a binary search algorithm to locate the nearest target data node in the skip list of the ring data structure;

[0125] Extract the node identifier of the target data node, construct a mapping relationship between the unique identifier of the data shard and the node identifier of the target data node, encapsulate the mapping relationship into a routing record, and organize the routing record in a tree structure to generate a shard routing table;

[0126] Constructing a data transmission buffer pool, storing the data slices in the data transmission buffer pool, determining a transmission target based on the slice routing table, creating an asynchronous transmission thread, and transmitting the data slices from the data transmission buffer pool to a target data node corresponding to the slice routing table;

[0127] A sliding time window is used to monitor the resource usage of the target data node, and the processor usage rate, memory usage rate, disk read / write rate and network bandwidth usage rate of the target data node are collected as resource indicators. The resource indicators are subjected to principal component analysis to obtain resource indicator characteristics, and the node load value of the target data node is calculated based on the resource indicator characteristics.

[0128] When it is detected that the node load value exceeds a second preset threshold value within a specified number of consecutive time windows, a confidence interval of the load anomaly is calculated based on the load anomaly mark, and the confidence interval is input into a pre-trained trend prediction model to obtain a predicted load value within a future time window;

[0129] Construct a load balancing scoring model, take the node load value, fluctuation variance, and network delay as scoring indicators, calculate the weight vector of the scoring indicator based on the hierarchical analysis method, and take the product of the scoring indicator and the weight vector as the node score value;

[0130] An optimization algorithm is used to optimize the node score values ​​of all data nodes to obtain an optimal load balancing solution, and the data node with the highest score value is selected from the optimal load balancing solution as the target migration node.

[0131] Exemplarily, first, a consistent hash calculation is performed on the unique identifier of the data shard to obtain a hash value. Specifically, a hash algorithm such as MD5 or SHA-1 can be used to perform a hash calculation on the unique identifier of the data shard (such as the shard ID) to obtain a 32-bit or 40-bit hash value. For example, an MD5 hash calculation is performed on the shard ID "shard001" to obtain a hash value "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6".

[0132] Then, the obtained hash value is mapped clockwise to the ring data structure. An integer ring from 0 to 2^32-1 can be constructed to map the hash value to a point on the ring. For example, the above hash value is mapped to point 3689348814 on the ring.

[0133] Next, a binary search algorithm is used to locate the nearest target data node in the skip list of the ring data structure. The skip list stores the mapping relationship between the hash value of the data node and the node information. Through binary search, the first node greater than or equal to the hash value can be quickly found, which is the target data node. For example, the node "node1" with a hash value of 3689348815 is found.

[0134] Then, the node identifier of the target data node is extracted, and a mapping relationship between the unique identifier of the data shard and the node identifier of the target data node is constructed. For example, a mapping relationship between "shard001" and "node1" is constructed. The mapping relationship is encapsulated as a routing record, and a tree structure (such as a B+ tree) is used to organize the routing record to generate a shard routing table.

[0135] Next, a data transmission buffer pool is built and the data shards are stored in it. The transmission target is determined based on the shard routing table, and an asynchronous transmission thread is created to transmit the data shards from the buffer pool to the target data node. The producer-consumer model can be adopted, where the producer writes the data into the buffer pool and the consumer asynchronously reads and transmits the data.

[0136] Then, a sliding time window is used to monitor the resource usage of the target data node. A 5-minute sliding window is set, and resource indicators are collected every 1 minute, including processor usage, memory usage, disk read and write rates, and network bandwidth usage. Principal component analysis is performed on the collected resource indicators to extract the main features, and the node load value is calculated based on the features.

[0137] When the node load value is detected to exceed 80% in three consecutive time windows, the confidence interval of the load anomaly is calculated based on the load anomaly mark. The bootstrap method is used to construct the confidence interval. For example, the 95% confidence interval is [82%, 88%]. The confidence interval is input into the pre-trained LSTM model to predict the load value of the next three time windows.

[0138] Next, a load balancing scoring model is constructed. The node load value, load fluctuation variance, and network delay are used as scoring indicators. The indicator weight vector is calculated based on the hierarchical analysis method. For example, the weight vector [0.5, 0.3, 0.2] is obtained. The product of the scoring indicator and the weight vector is used as the node score value.

[0139] Finally, the genetic algorithm is used to optimize the score values ​​of all data nodes to obtain the optimal load balancing solution. The data node with the highest score value is selected from the optimal solution as the target migration node. For example, "node5" with a score value of 0.92 is selected as the target migration node.

[0140] In this embodiment, the consistent hashing and skip table structures are used to achieve efficient data shard routing and improve data access efficiency. Based on the sliding time window and principal component analysis, the node load is monitored and evaluated in real time and accurately, and load anomalies are discovered in a timely manner. The load balancing scoring model and optimization algorithm are used to achieve global optimal load balancing and improve the overall performance and availability of the system.

[0141] In an optional embodiment,

[0142] The node score values ​​of all data nodes are optimized by using an optimization algorithm to obtain an optimal load balancing solution, and the data node with the highest score value is selected as the target migration node from the optimal load balancing solution, including:

[0143] Constructing a multi-layer neural network to extract node features, wherein the input layer of the multi-layer neural network receives a processor utilization time series, a memory occupancy time series, and an input and output load time series, and a skip connection structure and a batch normalization layer are set in the hidden layer to generate a node feature vector through an activation function;

[0144] Optimizing the hidden state of the multi-layer neural network, calculating the attention weight matrix and the feature increment matrix, and weighted combining the feature increment matrix with the original hidden state matrix to obtain an optimized node hidden state;

[0145] The node load sequence is obtained based on a dynamic sliding window mechanism. The size of the sliding window is adaptively adjusted according to the load fluctuation degree. The query matrix, key matrix and value matrix of the multi-head attention module are constructed, and the attention weight is calculated to obtain the node time series characteristics.

[0146] Input the node time series features into the multi-head attention layer to generate multiple attention head features, fuse the multiple attention head features using nonlinear transformation, and output the node fusion feature vector;

[0147] Construct a similarity calculation module based on graph neural network, represent data nodes as nodes in the graph structure, calculate the similarity matrix between nodes based on the node fusion feature vector, and generate the initial node score based on the topological relationship between the nodes;

[0148] Design a scoring optimization model with a dual neural network architecture, including a scoring strategy network and a value assessment network, which takes the node initial score, system state vector and historical score sequence as input and outputs the node score adjustment strategy;

[0149] The entropy weight method is used to calculate the weight coefficient of the node load index, and the weight coefficient is combined with the score adjustment strategy to perform multi-objective optimization on the initial node score to obtain the optimized node score value. At the same time, the federated learning framework is used to design a distributed score optimization scheme. Each node trains a local score optimization model locally, and a secure aggregation algorithm is used to fuse the local score optimization model parameters to construct a global score optimization strategy.

[0150] According to the global scoring optimization strategy, combined with the historical load level, resource utilization and network performance indicators of the nodes, the scoring values ​​of all data nodes are comprehensively optimized to generate an optimal load balancing solution. Based on the optimal load balancing solution, the data node with the highest scoring value and meeting the migration constraints is selected as the target migration node.

[0151] Exemplarily, a multi-layer neural network is first constructed to extract node features. The input layer of the network receives time series data of processor utilization, memory occupancy, and I / O load. For example, for a certain data node, its CPU utilization sequence in the last 24 hours is [30%, 35%, 40%, ..., 25%], the memory occupancy sequence is [60%, 65%, 70%, ..., 55%], and the I / O load sequence is [500, 600, 700, ..., 450] IOPS. A skip connection structure is set in the hidden layer to directly transfer shallow features to the deep layer to alleviate the gradient vanishing problem. At the same time, a batch normalization layer is introduced to standardize each batch of data to accelerate network convergence. Finally, through activation functions such as ReLU, a feature vector representing the node state is generated with a dimension of 128.

[0152] Next, the hidden state of the multi-layer network is optimized. The attention weight matrix is ​​calculated to measure the importance of features at different time steps. At the same time, a feature increment matrix is ​​generated to capture the changing trend of time series data. The feature increment matrix is ​​weighted combined with the original hidden state matrix to obtain the optimized node hidden state matrix with a dimension of 24×128.

[0153] Then, the node load sequence is obtained based on the dynamic sliding window mechanism. The sliding window size is adaptively adjusted according to the degree of load fluctuation. When the fluctuation is severe, the window is reduced, and when the fluctuation is gentle, the window is expanded. For example, when the standard deviation of CPU utilization is greater than 10%, the window is set to 6 hours; when it is less than 5%, the window is set to 12 hours. On this basis, the query matrix, key matrix and value matrix of the multi-head attention module are constructed, and the attention weight is calculated to obtain the time series feature vector of the node with a dimension of 256.

[0154] The above time series features are input into the multi-head attention layer to generate 8 attention head features. The dimension of each head is 32, capturing time series patterns of different scales. Nonlinear transformation is used to fuse multiple attention head features. Specifically, a two-layer fully connected network is used, with a ReLU activation function added in the middle, and finally a node fusion feature vector with a dimension of 128 is output.

[0155] Construct a similarity calculation module based on graph neural network. Represent data nodes as nodes in the graph structure, and the edges between nodes represent physical connection relationships. Calculate the cosine similarity matrix between nodes based on the node fusion feature vector. Combine the topological distance between nodes to generate the initial node score. The score range is 0-100, and the higher the score, the more suitable the node is as a migration target.

[0156] Design a scoring optimization model with a dual neural network architecture. The scoring strategy network uses a three-layer fully connected structure. The input is the initial node score, the current system load state vector, and the historical score sequence. The output is the node score adjustment strategy, that is, the increase or decrease in the score. The value assessment network also uses a three-layer fully connected structure to evaluate the long-term benefits of the current scoring strategy. The two networks are trained alternately to continuously optimize the scoring strategy.

[0157] The entropy weight method is used to calculate the weight coefficient of the node load index. The information entropy is calculated for indicators such as CPU utilization, memory occupancy, and I / O load. The larger the entropy value, the smaller the weight. The weight coefficient is combined with the score adjustment strategy to perform multi-objective optimization on the initial score of the node. At the same time, the federated learning framework is used to design a distributed score optimization scheme. Each node trains a local score optimization model locally, uses a secure aggregation algorithm to fuse local model parameters, and constructs a global score optimization strategy.

[0158] Finally, according to the global scoring optimization strategy, combined with the historical load level, resource utilization and network performance indicators of the nodes, the scoring values ​​of all data nodes are comprehensively optimized to generate the optimal load balancing solution. The data nodes with the highest scoring values ​​and meeting the migration constraints (such as sufficient available resources and sufficient network bandwidth) are selected as the target migration nodes.

[0159] In this embodiment, through multi-layer neural networks and attention mechanisms, multi-dimensional time series features of nodes are effectively extracted, node status is fully characterized, and the accuracy of the load balancing solution is improved. The graph neural network and dual neural network architecture are adopted to fully consider the relationship between nodes and long-term benefits, making the load balancing strategy more intelligent and efficient. Federated learning and secure aggregation algorithms are introduced to achieve distributed optimization while protecting data privacy, improving the scalability and robustness of the system.

[0160] S103. Receive a data processing task, determine the distribution location of the data shards according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtasks to the corresponding target data nodes based on the priority, monitor the resource occupancy rate of the subtasks, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result.

[0161] When receiving a specific data processing task, the distribution location of the relevant data shards is first determined according to the shard routing table. The data processing task is split into multiple subtasks, and a dependency graph between the subtasks is constructed. The dependency graph is represented by a directed acyclic graph (DAG) structure, where nodes represent subtasks and edges represent dependencies.

[0162] Calculate the priority of each subtask based on the dependency graph. Priority calculation takes into account factors such as the urgency of the task, dependencies, and estimated execution time. Normalize the calculated priority value to a range of 0-100. Assign the subtask to the corresponding target data node for execution based on the priority.

[0163] During the task execution, the resource usage of each subtask is monitored in real time. When the resource usage of a subtask (such as CPU usage) exceeds the preset threshold (such as 90%), the execution of the subtask is suspended. The node with a current load value lower than the preset threshold (such as 50%) is selected as the task redistribution node, and the subtask is migrated to the node for continued execution. The intermediate state of the task is saved during the migration process to achieve seamless migration.

[0164] Finally, the execution results of all subtasks are summarized to generate the final data processing results. The result summary process adopts the MapReduce model. The Map stage processes the results of each subtask in parallel, and the Reduce stage merges the intermediate results to obtain the final output.

[0165] In an optional embodiment,

[0166] Receiving a data processing task, determining the distribution location of the data shard according to the shard routing table, splitting the data processing task into multiple subtasks, constructing a dependency graph of the subtasks, calculating the priority of each subtask according to the dependency graph, and allocating the subtasks to corresponding target data nodes based on the priority includes:

[0167] Receive a data processing task, parse the input data set, operation operator sequence and result output specification in the data processing task, construct a task feature vector, input the task feature vector into a preset resource demand prediction model, obtain the computing resource demand value and storage resource demand value of the data processing task, find the location information of each data shard in the input data set based on the shard routing table, and collect the current resource status of each data node;

[0168] Constructing a data affinity matrix, the data affinity matrix records the position overlap and access association between different data shards, performing data shard clustering based on the data affinity matrix, dividing the data shards with affinities higher than a preset affinity threshold into the same data group, and combining the calculation characteristics of the operation operator sequence and the distribution characteristics of the data group to split the data processing task into multiple subtasks;

[0169] Construct a two-layer dependency graph, in which the first-layer nodes of the two-layer dependency graph represent subtasks, and the second-layer nodes represent data shards. The two-layer dependency graph describes the execution dependency relationship and data access relationship between subtasks. Based on the two-layer dependency graph, a critical path algorithm is used to calculate the execution priority of the subtask. The execution priority is determined based on the computational overhead and data access overhead of the subtask. A task priority queue is established, and subtasks that meet the execution conditions are added to the task priority queue in the order of the execution priority.

[0170] Calculate the position matching degree between the subtask and the data node in the task priority queue, where the position matching degree is determined based on the data access scope of the subtask and the data distribution of the data node, and assign the subtask whose position matching degree is higher than a preset matching degree threshold and meets the computing resource requirement value and the storage resource requirement value to the corresponding target data node.

[0171] Exemplarily, first, a data processing task is received, and the input data set, operation operator sequence, and result output specification in the task are parsed. For example, the input data set may contain multiple distributed tables, the operation operator sequence may contain operations such as join, filter, aggregation, and the result output specification specifies the output format and storage location. Based on the parsing results, a task feature vector is constructed, which contains features such as data scale, operator complexity, and data dependencies. The task feature vector is input into a pre-trained resource demand prediction model, which can be a machine learning model trained based on historical task data. The model outputs an estimate of the computing resources (such as the number of CPU cores, memory size) and storage resources (such as disk space) required for the data processing task.

[0172] Next, query the shard routing table maintained in the system to obtain the location information of each data shard in the input data set. The shard routing table records the mapping relationship between data shards and physical nodes. At the same time, collect the current resource usage status of each data node, including CPU utilization, memory usage, disk usage and other indicators.

[0173] Then construct a data affinity matrix, the rows and columns of which represent data shards, and the matrix element aij represents the affinity between shard i and shard j. Affinity is calculated based on two indicators: position overlap and access association. Position overlap indicates the degree to which two shards are located on the same node, and access association indicates the frequency with which two shards are accessed at the same time. For example, for shards of two tables that are frequently joined, their affinity is higher. Cluster analysis is performed based on the affinity matrix, and algorithms such as k-means can be used to divide data shards with an affinity higher than a preset threshold (such as 0.8) into the same data group.

[0174] Combine the computational characteristics of the operator sequence (such as the operator's parallelism, data dependency, etc.) and the distribution characteristics of the data grouping to split the original data processing task into multiple subtasks. For example, for a task that includes join and aggregation, the join operation can be split into multiple parallel subtasks, each of which processes part of the data, and then the aggregation is treated as a separate subtask.

[0175] Next, a two-layer dependency graph is constructed, where the first-layer nodes represent subtasks and the second-layer nodes represent data shards. In the first layer, the edge connecting two subtasks indicates that there is an execution dependency between them. In the second layer, the edge connecting a subtask and a data shard indicates that the subtask needs to access the data shard. For example, the subtask of the join operation will be connected to the corresponding shards of its two input tables. Based on this two-layer dependency graph, the critical path algorithm is used to calculate the execution priority of each subtask. The priority calculation takes into account the computational overhead of the subtask (estimated based on the complexity of the operation) and the data access overhead (estimated based on the location and size of the data). The calculated priority is used as the key value to establish a priority queue, and the subtasks that meet the execution conditions (that is, all the dependent tasks have been completed) are added to the queue in priority order.

[0176] Finally, for the subtasks in the priority queue, calculate the location matching degree between them and each data node. The location matching degree is calculated based on the distribution of the data shards that the subtask needs to access on the node. The higher the access data ratio, the higher the matching degree. Select the data node with a location matching degree higher than the preset threshold (such as 0.7) and that meets the computing resources and storage resources required by the subtask as the target node. If there are multiple nodes that meet the conditions, you can select the node with the lighter current load. Assign the subtask to the selected target data node for execution.

[0177] In this embodiment, by constructing a data affinity matrix and performing data sharding clustering, data locality is improved, cross-node data transmission is reduced, thereby reducing network overhead and improving task execution efficiency. A two-layer dependency graph and a critical path algorithm are used to calculate task priorities, and the order of task execution is reasonably arranged, resource conflicts and task waiting are avoided, and the overall resource utilization of the system is improved. Taking into account multiple factors such as data location, task characteristics, and node resource status, a more intelligent and flexible task scheduling is achieved, which adapts to complex distributed computing environments and improves the execution efficiency and scalability of large-scale data processing tasks.

[0178] In an optional embodiment,

[0179] Allocating subtasks whose location matching degree is higher than a preset matching degree threshold and meet the computing resource requirement value and the storage resource requirement value to the corresponding target data node includes:

[0180] Acquire data access information of the task to be assigned and storage status information of the data node, wherein the data access information includes a data shard identifier that the task to be assigned needs to access and a corresponding access operation type, and the storage status information includes a data shard identifier currently stored by the data node;

[0181] Based on the data access information and the storage status information, the number of intersections of the data shard set accessed by the to-be-assigned task and the data shard set stored by the data node is calculated to obtain the data shard overlap;

[0182] Obtain the weight coefficients of the read operation and the write operation in the task to be assigned, calculate the weighted access matching degree based on the weight coefficient and the data shard overlap, and combine the data shard overlap and the weighted access matching degree according to a preset mixing coefficient to obtain the position matching degree;

[0183] Obtain computing resource requirement information of the task to be assigned and available resource information of the data node, calculate resource satisfaction based on the computing resource requirement information and the available resource information, filter out candidate data nodes that meet the resource requirements according to the resource satisfaction, and select the node with the highest position matching degree among the candidate data nodes as the target data node.

[0184] Exemplarily, first obtain the data access information of the task to be assigned and the storage status information of the data node. The data access information includes the data shard identifier that the task to be assigned needs to access and the corresponding access operation type, and the storage status information includes the data shard identifier currently stored by the data node. For example, the data access information of the task to be assigned is: {shard 1: read, shard 2: write, shard 3: read}, the storage status information of data node A is: {shard 1, shard 2, shard 4}, and the storage status information of data node B is: {shard 2, shard 3, shard 5}.

[0185] Then, based on the data access information and storage status information, the number of intersections between the data shard set accessed by the task to be assigned and the data shard set stored by the data node is calculated to obtain the data shard overlap. For the above example, the data shard overlap between the task to be assigned and data node A is 2, and the data shard overlap between the task to be assigned and data node B is 2.

[0186] Then, the weight coefficients of the read and write operations in the task to be assigned are obtained, and the weighted access matching degree is calculated based on the weight coefficients and the data shard overlap. For example, if the read operation weight is 0.3 and the write operation weight is 0.7, then for data node A, the weighted access matching degree is 0.3*1+0.7*1=1; for data node B, the weighted access matching degree is 0.3*1+0.7*1=1.

[0187] The data shard overlap and weighted access matching are combined according to the preset mixing coefficient to obtain the position matching. Assuming the mixing coefficient is 0.5, the position matching of data node A is 0.5*2+0.5*1=1.5, and the position matching of data node B is 0.5*2+0.5*1=1.5.

[0188] Then, the computing resource requirement information of the task to be assigned and the available resource information of the data node are obtained, and the resource satisfaction is calculated based on the computing resource requirement information and the available resource information. For example, the task to be assigned requires a 2-core CPU and 4GB of memory, data node A has a 4-core CPU and 8GB of available memory, and data node B has a 2-core CPU and 4GB of available memory. The resource satisfaction of data node A is calculated to be 200%, and the resource satisfaction of data node B is 100%.

[0189] According to the resource satisfaction, the candidate data nodes that meet the resource requirements are screened out, and the node with the highest position matching degree is selected as the target data node among the candidate data nodes. For the above example, data nodes A and B both meet the resource requirements and have the same position matching degree, and one of them can be randomly selected as the target data node.

[0190] Finally, the subtasks whose location matching degree is higher than the preset matching degree threshold and meet the computing resource requirement value and storage resource requirement value are allocated to the corresponding target data node.

[0191] In this embodiment, by calculating the data shard overlap and weighted access matching, the location correlation between tasks and data nodes can be more accurately measured, thereby improving data locality. Combining computing resource requirements with available resources of data nodes for matching can avoid resource overload and improve overall resource utilization. Using hybrid matching and resource satisfaction for comprehensive evaluation can achieve a better balance between data locality and load balancing, thereby improving overall system performance.

[0192] Figure 2 FIG. 1 is a schematic diagram of the structure of a distributed real-time database intelligent management system based on big data according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0193] The first unit is used to receive a data processing request, extract a data type identifier, a data processing rule, data scale information and a data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain a computing resource demand prediction value and a storage resource demand prediction value for data processing, call a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and select a data node whose node load value is lower than a first preset threshold as a target data node;

[0194] The second unit is used to divide the data shards on the target data node, generate a data shard mapping table using a ring data structure, construct a virtual node through polynomial calculation, assign a unique identifier to each data shard, store the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table, write the data shards into the corresponding target data nodes according to the shard routing table, collect the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, select the data node with the lowest node load value as the target migration node, migrate the data shards to the target migration node, and update the shard routing table;

[0195] The third unit is used to receive a data processing task, determine the distribution location of the data shard according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtask to the corresponding target data node based on the priority, monitor the resource occupancy rate of the subtask, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result.

[0196] According to a third aspect of the embodiments of the present invention,

[0197] An electronic device is provided, comprising:

[0198] processor;

[0199] a memory for storing processor-executable instructions;

[0200] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0201] A fourth aspect of the embodiments of the present invention is:

[0202] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0203] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed real-time database intelligent management method based on big data, characterized in that: include: Receive a data processing request, extract a data type identifier, a data processing rule, data scale information, and a data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain a computing resource demand prediction value and a storage resource demand prediction value for data processing, call a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and select a data node whose node load value is lower than a first preset threshold as a target data node; Divide the data into shards on the target data node, generate a data shard mapping table using a ring data structure, construct a virtual node by polynomial calculation, assign a unique identifier to each data shard, store the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table, write the data shard into the corresponding target data node according to the shard routing table, collect the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, select the data node with the lowest node load value as the target migration node, migrate the data shard to the target migration node, and update the shard routing table; Receive a data processing task, determine the distribution location of the data shards according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtasks to corresponding target data nodes based on the priority, monitor the resource occupancy rate of the subtasks, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result; Calling a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and selecting a data node whose node load value is lower than a first preset threshold as a target data node includes: Collect computing resource usage data and storage resource usage data of data nodes, extract features of computing resource usage data and storage resource usage data respectively, and fuse them to generate data node load features; The data node load characteristics are segmented using an adaptive time window, the window size of the adaptive time window is determined according to the fluctuation period of the data node load characteristics, the segmented data node load characteristics are input into a timing analysis network and multi-scale decomposition is performed to obtain a load evolution law; The load evolution law is input into a data node evaluation model, the data node evaluation model uses a deep temporal network to extract abnormal features, and calculates the influence of the abnormal features based on an attention mechanism to obtain the health of the data node; A multi-level adaptive threshold system is constructed based on the load evolution law and the health of the data node. The multi-level adaptive threshold system includes a main threshold model and an auxiliary threshold model. The main threshold model determines a reference threshold according to the probability distribution of the data node load characteristics. The auxiliary threshold model realizes dynamic adjustment of the threshold based on a reinforcement learning method. The reference threshold and the dynamically adjusted threshold are combined to obtain a first preset threshold. The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into a data node load calculation module, and the data node load calculation module uses a multi-layer computing framework to calculate the node load value; The node load value is compared with the first preset threshold value, and based on the comparison result and the health of the data node, a node sorting algorithm is used to select a data node with a node load value lower than the first preset threshold value and the highest health as the target data node.

2. The method according to claim 1, characterized in that The computing resource demand prediction value, the storage resource demand prediction value and the data node load characteristics are input into the data node load calculation module, and the data node load calculation module uses a multi-layer calculation framework to calculate the node load value including: Establish a multi-dimensional load feature extraction module, which performs time series decomposition on the computing resource demand prediction value, the storage resource demand prediction value and the load history data to obtain trend components, periodic components and random components, and inputs the trend components, periodic components and random components into an adaptive standardization module, which dynamically selects a standardization strategy according to data distribution characteristics, and performs adaptive standardization processing on the input data to obtain a standardized feature vector; Constructing a multi-head attention network to extract the feature correlation of the standardized feature vector, inputting the feature correlation into a feature fusion layer to generate a feature weight matrix, and optimizing the feature weight matrix using an adaptive optimization algorithm to obtain an optimized feature weight matrix; The standardized feature vector and the optimized feature weight matrix are input into the first computing layer, the first computing layer uses a bidirectional long short-term memory network to extract time series features, and performs weighted combination of the standardized feature vectors through an attention mechanism to obtain an initial load value; The initial load value is input into a second computing layer, and the second computing layer uses a residual connection structure and a dilated convolution to enhance the features of the initial load value, and highlights key load features through a self-attention mechanism to obtain enhanced load features; The enhanced load feature and the historical load time series are input into a third computing layer, the third computing layer uses a gated cycle unit to extract a timing pattern, and adaptively fuses the timing pattern and the enhanced load feature through a dynamic weight coefficient to obtain a fused load value; The fused load value is input into a fourth computing layer, and the fourth computing layer performs a nonlinear transformation on the fused load value through a multi-layer perceptron, and obtains an initial node load value in combination with a batch normalization operation; The multidimensional error index between the initial node load value and the actual load value is calculated. When the multidimensional error index exceeds a preset threshold, the network parameters of the multilayer perceptron are smoothly optimized using the exponential moving average method to obtain the optimized network parameters. The optimized network parameters are applied to the fourth calculation layer, and the fused load value is recalculated to obtain the final node load value.

3. The method according to claim 1, characterized in that Storing the correspondence between the unique identifier of the data shard and the target data node in a shard routing table, writing the data shard to the corresponding target data node according to the shard routing table, collecting the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, selecting the data node with the lowest node load value as the target migration node includes: Perform consistent hashing on the unique identifier of the data shard to obtain a hash value, map the hash value clockwise to a ring data structure, and use a binary search algorithm to locate the nearest target data node in the skip list of the ring data structure; Extract the node identifier of the target data node, construct a mapping relationship between the unique identifier of the data shard and the node identifier of the target data node, encapsulate the mapping relationship into a routing record, and organize the routing record in a tree structure to generate a shard routing table; Constructing a data transmission buffer pool, storing the data slices in the data transmission buffer pool, determining a transmission target based on the slice routing table, creating an asynchronous transmission thread, and transmitting the data slices from the data transmission buffer pool to a target data node corresponding to the slice routing table; A sliding time window is used to monitor the resource usage of the target data node, and the processor usage rate, memory usage rate, disk read / write rate and network bandwidth usage rate of the target data node are collected as resource indicators. The resource indicators are subjected to principal component analysis to obtain resource indicator characteristics, and the node load value of the target data node is calculated based on the resource indicator characteristics. When it is detected that the node load value exceeds a second preset threshold value within a specified number of consecutive time windows, a confidence interval of the load anomaly is calculated based on the load anomaly mark, and the confidence interval is input into a pre-trained trend prediction model to obtain a predicted load value within a future time window; Construct a load balancing scoring model, take the node load value, fluctuation variance, and network delay as scoring indicators, calculate the weight vector of the scoring indicator based on the hierarchical analysis method, and take the product of the scoring indicator and the weight vector as the node score value; An optimization algorithm is used to optimize the node score values ​​of all data nodes to obtain an optimal load balancing solution, and the data node with the highest score value is selected from the optimal load balancing solution as the target migration node.

4. The method according to claim 3, characterized in that The node score values ​​of all data nodes are optimized by using an optimization algorithm to obtain an optimal load balancing solution, and the data node with the highest score value is selected as the target migration node from the optimal load balancing solution, including: Constructing a multi-layer neural network to extract node features, wherein the input layer of the multi-layer neural network receives a processor utilization time series, a memory occupancy time series, and an input and output load time series, and a skip connection structure and a batch normalization layer are set in the hidden layer to generate a node feature vector through an activation function; Optimizing the hidden state of the multi-layer neural network, calculating the attention weight matrix and the feature increment matrix, and weighted combining the feature increment matrix with the original hidden state matrix to obtain an optimized node hidden state; The node load sequence is obtained based on a dynamic sliding window mechanism. The size of the sliding window is adaptively adjusted according to the load fluctuation degree. The query matrix, key matrix and value matrix of the multi-head attention module are constructed, and the attention weight is calculated to obtain the node time series characteristics. Input the node time series features into the multi-head attention layer to generate multiple attention head features, fuse the multiple attention head features using nonlinear transformation, and output the node fusion feature vector; Construct a similarity calculation module based on graph neural network, represent data nodes as nodes in the graph structure, calculate the similarity matrix between nodes based on the node fusion feature vector, and generate the initial node score based on the topological relationship between the nodes; Design a scoring optimization model with a dual neural network architecture, including a scoring strategy network and a value assessment network, which takes the node initial score, system state vector and historical score sequence as input and outputs the node score adjustment strategy; The entropy weight method is used to calculate the weight coefficient of the node load index, and the weight coefficient is combined with the score adjustment strategy to perform multi-objective optimization on the initial node score to obtain the optimized node score value. At the same time, the federated learning framework is used to design a distributed score optimization scheme. Each node trains a local score optimization model locally, and a secure aggregation algorithm is used to fuse the local score optimization model parameters to construct a global score optimization strategy. According to the global scoring optimization strategy, combined with the historical load level, resource utilization and network performance indicators of the nodes, the scoring values ​​of all data nodes are comprehensively optimized to generate an optimal load balancing solution. Based on the optimal load balancing solution, the data node with the highest scoring value and meeting the migration constraints is selected as the target migration node.

5. The method according to claim 1, characterized in that Receiving a data processing task, determining the distribution location of the data shard according to the shard routing table, splitting the data processing task into multiple subtasks, constructing a dependency graph of the subtasks, calculating the priority of each subtask according to the dependency graph, and allocating the subtasks to corresponding target data nodes based on the priority includes: Receive a data processing task, parse the input data set, operation operator sequence and result output specification in the data processing task, construct a task feature vector, input the task feature vector into a preset resource demand prediction model, obtain the computing resource demand value and storage resource demand value of the data processing task, find the location information of each data shard in the input data set based on the shard routing table, and collect the current resource status of each data node; Constructing a data affinity matrix, the data affinity matrix records the position overlap and access association between different data shards, performing data shard clustering based on the data affinity matrix, dividing the data shards with affinities higher than a preset affinity threshold into the same data group, and combining the calculation characteristics of the operation operator sequence and the distribution characteristics of the data group to split the data processing task into multiple subtasks; Construct a two-layer dependency graph, in which the first-layer nodes of the two-layer dependency graph represent subtasks, and the second-layer nodes represent data shards. The two-layer dependency graph describes the execution dependency relationship and data access relationship between subtasks. Based on the two-layer dependency graph, a critical path algorithm is used to calculate the execution priority of the subtask. The execution priority is determined based on the computational overhead and data access overhead of the subtask. A task priority queue is established, and subtasks that meet the execution conditions are added to the task priority queue in the order of the execution priority. Calculate the position matching degree between the subtask and the data node in the task priority queue, where the position matching degree is determined based on the data access scope of the subtask and the data distribution of the data node, and assign the subtask whose position matching degree is higher than a preset matching degree threshold and meets the computing resource requirement value and the storage resource requirement value to the corresponding target data node.

6. The method according to claim 5, characterized in that Allocating subtasks whose location matching degree is higher than a preset matching degree threshold and meet the computing resource requirement value and the storage resource requirement value to the corresponding target data node includes: Acquire data access information of the task to be assigned and storage status information of the data node, wherein the data access information includes a data shard identifier that the task to be assigned needs to access and a corresponding access operation type, and the storage status information includes a data shard identifier currently stored by the data node; Based on the data access information and the storage status information, the number of intersections of the data shard set accessed by the to-be-assigned task and the data shard set stored by the data node is calculated to obtain the data shard overlap; Obtain the weight coefficients of the read operation and the write operation in the task to be assigned, calculate the weighted access matching degree based on the weight coefficient and the data shard overlap, and combine the data shard overlap and the weighted access matching degree according to a preset mixing coefficient to obtain the position matching degree; Obtain computing resource requirement information of the task to be assigned and available resource information of the data node, calculate resource satisfaction based on the computing resource requirement information and the available resource information, filter out candidate data nodes that meet the resource requirements according to the resource satisfaction, and select the node with the highest position matching degree among the candidate data nodes as the target data node.

7. A distributed real-time database intelligent management system based on big data, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to receive a data processing request, extract a data type identifier, a data processing rule, data scale information and a data priority in the data processing request, construct a data feature vector based on the data type identifier, input the data feature vector into a preset neural network model, obtain a computing resource demand prediction value and a storage resource demand prediction value for data processing, call a data node load calculation module to calculate a node load value according to the computing resource demand prediction value and the storage resource demand prediction value, and select a data node whose node load value is lower than a first preset threshold as a target data node; The second unit is used to divide the data shards on the target data node, generate a data shard mapping table using a ring data structure, construct a virtual node through polynomial calculation, assign a unique identifier to each data shard, store the corresponding relationship between the unique identifier of the data shard and the target data node in a shard routing table, write the data shards into the corresponding target data nodes according to the shard routing table, collect the node load value of the target data node based on the shard routing table, and when it is detected that the node load value of the target data node exceeds a second preset threshold, select the data node with the lowest node load value as the target migration node, migrate the data shards to the target migration node, and update the shard routing table; The third unit is used to receive a data processing task, determine the distribution location of the data shard according to the shard routing table, split the data processing task into multiple subtasks, build a dependency graph of the subtasks, calculate the priority of each subtask according to the dependency graph, assign the subtask to the corresponding target data node based on the priority, monitor the resource occupancy rate of the subtask, and when the resource occupancy rate exceeds a third preset threshold, suspend the execution of the subtask, select a data node with a node load value lower than a fourth preset threshold as a task reallocation node, migrate the subtask to the task reallocation node for continued execution, summarize the execution results of the subtasks, and generate a data processing result.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images