Distributed data analysis system and method for edge node model parameters

Abstract

The application relates to the technical field of data analysis, in particular to a distributed data analysis system and method for edge node model parameters, which comprises the following steps: analyzing the model parameter update urgency of each edge node based on local model parameter update data and node running state data of each edge node; analyzing the data transmission potential of each edge node under comprehensive resource constraints based on node running state data and network topology data; quantitatively evaluating the priority of model parameter transmission of each edge node according to the parameter update urgency and the data transmission potential; and scheduling and optimizing the data transmission between edge nodes based on the evaluation results and the to-be-transmitted model parameter update data, so that the node data transmission efficiency is optimized, and the intelligent scheduling of the model parameter transmission of the edge nodes is precise, efficient and dynamic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a distributed data analysis system and method for edge node model parameters. BACKGROUND

[0002] As an important part of the edge computing system, the edge node undertakes the key tasks of real-time data collection, local model iterative training, and parameter update synchronization. The efficient synchronization of the local model parameters of the edge node and the global model directly determines the convergence accuracy of the global model and the real-time performance of the edge service. For example, if the model parameter synchronization of the device fault detection model of the production line edge node in the industrial internet is delayed, it may cause delay in the identification of device abnormalities and lead to production downtime. If the model parameter synchronization of the environment perception model of the roadside edge node in the automatic driving is not synchronized, it may cause misjudgment of the vehicle decision and lead to safety risks. However, the edge node generally faces complex constraints such as wide-area distributed deployment, heterogeneous hardware resources, dynamic network topology, and diverse characteristics of the parameter data to be transmitted.

[0003] Therefore, the prior art attempts to determine the transmission priority by the model parameter update amplitude or to select the transmission node according to the CPU and memory usage of the node. However, these methods still have significant deficiencies and are difficult to adapt to the complex dynamic scenarios of edge computing. First, the prior art ignores the coordinated matching of transmission demand and transmission capacity. The emergency level is only measured by the parameter update amplitude, and the node with large parameter deviation is preferentially scheduled, but the node may be in a state of resource overload or network topology edge, and forced transmission may easily lead to data packet loss, dramatic increase in delay, or even transmission failure. Second, the prior art generally ignores the influence of network topology structure on transmission efficiency and only focuses on the resource state of the node itself, resulting in over-scheduling of nodes with abundant resources but poor topology. Finally, the prior art does not fully utilize the data characteristics of the model parameters to be transmitted, resulting in inefficient occupation of bandwidth resources. The model parameters generated by the edge node training often have significant sparsity and information density differences, but the existing scheduling methods treat the parameter data as homogeneous data streams without compressing the data volume through sparse coding and information entropy optimization, resulting in a large amount of redundant data occupying bandwidth.

[0004] To solve these problems, the present application designs a distributed data analysis system and method for edge node model parameters. SUMMARY

[0005] The purpose of the present application is to provide a distributed data analysis system and method for edge node model parameters, which realizes the coordinated evaluation of the parameter update emergency level and the resource carrying capacity at the node level, optimizes the transmission efficiency through parameter sparsity and information entropy analysis at the data level, and relies on the degree centrality to mine the structural advantages of the nodes at the network topology level, thereby realizing the precise, efficient, and dynamic intelligent scheduling of the transmission of edge node model parameters.

[0006] The present application is implemented as follows:

[0007] In a first aspect, the present application provides a distributed data analysis system for edge node model parameters, comprising an edge node data acquisition module, a parameter update urgency analysis module, a data transmission potential analysis module, a data transmission priority evaluation module, and a node data transmission scheduling optimization module. The edge node data acquisition module is configured to acquire local model parameter update data and node operating state data of each edge node, and simultaneously acquire network topology data and to-be-transmitted model parameter update data between each edge node. The parameter update urgency analysis module is configured to construct a parameter update urgency analysis model based on the local model parameter update data and the node operating state data of each edge node, and analyze the model parameter update urgency of each edge node. The data transmission potential analysis module is configured to construct a data transmission potential analysis model based on the node operating state data and the network topology data, and analyze the data transmission potential of each edge node under comprehensive resource constraints. The data transmission priority evaluation module is configured to construct a data transmission priority evaluation model based on the parameter update urgency analysis results and the data transmission potential analysis results, and quantitatively evaluate the priority of model parameter transmission of each edge node. The node data transmission scheduling optimization module is configured to dynamically schedule and optimize the data transmission sequence between edge nodes based on the quantitative evaluation results of the model parameter transmission priority and the to-be-transmitted model parameter update data.

[0008] In an implementation manner of the present application, the parameter update urgency analysis model is constructed based on the local model parameter update data and the node operating state data of each edge node, and the model parameter update urgency of each edge node is analyzed, including the following specific steps:

[0009] S21, obtaining the current local model parameter of each edge node in the local model parameter update data of each edge node and the last global model parameter, and taking the difference between the current local model parameter of each edge node and the last global model parameter as the model parameter difference vector of each edge node;

[0010] S22, calculating the Euclidean norm of the model parameter difference vector of each edge node as the model parameter update amplitude of each edge node, obtaining the mean value of the model parameter update amplitudes of all edge nodes, and performing division operation on the model parameter update amplitude of each edge node and the mean value of the model parameter update amplitudes of all edge nodes, and taking the division operation result as the parameter update amplitude difference degree of each edge node;

[0011] S23, obtaining the CPU usage, memory usage, and network bandwidth usage in the node operating state data of each edge node, and analyzing the resource tightness of each edge node;

[0012] S24, obtain the mean value of all edge node resource tightness, divide the resource tightness of each edge node by the mean value of all edge node resource tightness, and take the division result as the resource tightness difference degree of each edge node;

[0013] S25, multiply the parameter update amplitude difference degree of each edge node by the resource tightness difference degree, and take the multiplication result as the model parameter update urgency degree of each edge node.

[0014] In an implementation manner of the present application, based on the node running state data and the network topology data, a data transmission potential analysis model is constructed to analyze the data transmission potential of each edge node under the comprehensive resource constraint, including the following specific steps:

[0015] S31, extract the available network bandwidth, CPU idle rate and memory idle rate of each edge node from the node running state data of each edge node, and calculate the bandwidth index, CPU idle index and memory idle index of each edge node;

[0016] S32, perform geometric mean operation on the bandwidth index, CPU idle index and memory idle index of each edge node, and take the geometric mean operation result as the transmission resource index of each edge node;

[0017] S33, obtain the adjacency matrix of each edge node in the network topology data between the edge nodes, take the number of edges directly connected between each edge node in the adjacency matrix of each edge node as the degree centrality of each edge node, and take the ratio of the degree centrality of each edge node to the mean value of the degree centrality of all edge nodes as the network centrality of each edge node;

[0018] S34, multiply the transmission resource index and the network centrality of each edge node, and take the multiplication result as the data transmission potential of each edge node under the comprehensive resource constraint.

[0019] In an implementation manner of the present application, according to the parameter update urgency degree analysis result and the data transmission potential analysis result, a data transmission priority evaluation model is constructed to quantitatively evaluate the priority of model parameter transmission of each edge node; specifically including the following steps:

[0020] S41, obtain the model parameter update urgency degree analysis result of each edge node and the data transmission potential analysis result of each edge node under the comprehensive resource constraint obtained by analysis;

[0021] S42, multiply the model parameter update urgency degree analysis result of each edge node by the data transmission potential analysis result of each edge node under the comprehensive resource constraint, and take the multiplication result as the priority of model parameter transmission of each edge node.

[0022] In an implementation form of the present application, the data transmission sequence between edge nodes is dynamically scheduled and optimized based on the model parameter transmission priority quantization evaluation result and the to-be-transmitted model parameter update data; and specifically includes the following contents:

[0023] S51, obtain the priority of model parameter transmission of each edge node, and extract the to-be-transmitted model parameter update data on each edge node;

[0024] S52, analyze the data condensation degree of each edge node based on the to-be-transmitted model parameter update data;

[0025] S53, multiply the transmission priority of each edge node with the data condensation degree to obtain the comprehensive scheduling weight of each edge node;

[0026] S54, construct the data transmission sequence between edge nodes according to the order from high to low of each comprehensive scheduling weight, and sequentially perform data transmission on each edge node in the order of the data transmission sequence between edge nodes.

[0027] In an implementation form of the present application, in step S52, the data condensation degree of each edge node is analyzed based on the to-be-transmitted model parameter update data; and specifically includes the following steps:

[0028] S521, extract the to-be-transmitted model parameter matrix from the to-be-transmitted model parameter update data of each edge node, count the number of zero-value elements in the to-be-transmitted model parameter matrix, and take the ratio of the number of zero-value elements to the total number of elements in the to-be-transmitted model parameter matrix as the parameter sparsity of the to-be-transmitted model parameter matrix of each edge node;

[0029] S522, extract all non-zero parameter values in the to-be-transmitted model parameter matrix of each edge node, and establish the data distribution range of the to-be-transmitted model parameter matrix of each edge node according to the order from the minimum value to the maximum value of all non-zero parameter values;

[0030] S523, for the data distribution range of the to-be-transmitted model parameter matrix of each edge node, equally divide the data distribution range into multiple intervals, count the number of elements falling into each interval, take the ratio of the number of elements in each interval to the total number of all non-zero parameter values as the probability mass function value corresponding to each interval, and import the probability mass function values corresponding to all intervals into a Shannon entropy formula to calculate the information entropy of the to-be-transmitted model parameter matrix of each edge node;

[0031] S524, based on the information entropy of the model parameter matrix to be transmitted on each edge node obtained in step S523, performing division operation on the parameter sparsity of the model parameter matrix to be transmitted on each edge node and the information entropy, and taking the result of the division operation as the data condensation degree of the model parameter matrix to be transmitted on each edge node.

[0032] In a second aspect, the present application provides a distributed data analysis method for edge node model parameters, comprising the following specific steps:

[0033] S1, obtaining local model parameter update data and node running state data of each edge node, and obtaining network topology data and model parameter update data to be transmitted between each edge node;

[0034] S2, based on the local model parameter update data and the node running state data of each edge node, constructing a parameter update urgency analysis model to analyze the model parameter update urgency of each edge node;

[0035] S3, based on the node running state data and the network topology data, constructing a data transmission potential analysis model to analyze the data transmission potential of each edge node under the comprehensive resource constraint;

[0036] S4, according to the parameter update urgency analysis result and the data transmission potential analysis result, constructing a data transmission priority evaluation model to quantitatively evaluate the priority of model parameter transmission of each edge node;

[0037] S5, based on the quantitative evaluation result of the model parameter transmission priority and the model parameter update data to be transmitted, dynamically scheduling and optimizing the data transmission sequence between edge nodes.

[0038] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes the distributed data analysis method for edge node model parameters by calling the computer program stored in the memory.

[0039] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0040] The application is based on local model parameter update data and node running state data of each edge node, analyzes the model parameter update urgency of each edge node, is based on node running state data and network topology data, analyzes the data transmission potential of each edge node under comprehensive resource constraints, quantitatively evaluates the priority of model parameter transmission of each edge node according to the parameter update urgency and the data transmission potential, schedules and optimizes the data transmission between edge nodes based on the evaluation results and the to-be-transmitted model parameter update data, thereby optimizing the node data transmission efficiency and realizing the accurate, efficient and dynamic intelligent scheduling of edge node model parameter transmission. BRIEF DESCRIPTION OF DRAWINGS

[0041] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0042] Figure 1 It is a schematic diagram of the overall process of the distributed data analysis method of edge node model parameters of the application.

[0043] Figure 2 It is a structural schematic diagram of the distributed data analysis system of edge node model parameters of the application.

[0044] Figure 3 It is an analysis flowchart of step S2 of the distributed data analysis method of edge node model parameters of the application. DETAILED DESCRIPTION

[0045] The technical solutions of the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the application and the specific features in the embodiments are detailed descriptions of the technical solutions of the application, rather than limitations of the technical solutions of the application. In the case of no conflict, the technical features in the embodiments of the application and the embodiments can be combined with each other.

[0046] Embodiment 1

[0047] As shown in Figure 1 The embodiment provides a distributed data analysis method of edge node model parameters, which includes the following specific steps:

[0048] S1, obtaining local model parameter update data and node running state data of each edge node, and simultaneously obtaining network topology data and to-be-transmitted model parameter update data between each edge node;

[0049] S2, based on the local model parameter update data and the node running state data of each edge node, constructing a parameter update urgency analysis model, and analyzing the model parameter update urgency of each edge node;

[0050] S3, constructing a data transmission potential analysis model based on the node running state data and the network topology data, and analyzing the data transmission potential of each edge node under the comprehensive resource constraint;

[0051] S4, updating the emergency degree analysis result and the data transmission potential analysis result according to the parameters, constructing a data transmission priority evaluation model, and quantitatively evaluating the priority of the model parameter transmission of each edge node;

[0052] S5, based on the model parameter transmission priority quantitative evaluation result and the to-be-transmitted model parameter update data, dynamically scheduling and optimizing the data transmission sequence between the edge nodes.

[0053] In this embodiment, as shown in Figure 3 the parameter update emergency degree analysis model is constructed based on the local model parameter update data and the node running state data of each edge node in step S2, the model parameter update emergency degree of each edge node is analyzed, and the following specific steps are included:

[0054] S21, obtain the current local model parameters of each edge node in the local model parameter update data of each edge node and the last global model parameters, and take the difference between the current local model parameters of each edge node and the last global model parameters as the model parameter difference vector of each edge node; the embodiment calculates the parameter difference between the local model and the global model, quantifies the deviation degree between the two, and is the core basis for judging the necessity of parameter update. First, each edge node obtains the last version of global model parameters through regular communication with the center node. The parameters are usually stored in the distributed database of the center node, and the edge node obtains them in an incremental synchronization manner through the RESTful API interface, only downloads the parameter fragments inconsistent with the local cache, and reduces the data transmission amount. The current local model parameters are automatically exported by the model training module of the edge node after each training iteration is completed, including the weight matrix, bias vector and other core parameters of each layer of the model. For example, in a convolutional neural network (CNN), the filter weight of the convolution layer and the sampling parameter of the pooling layer are extracted. Then, the parameter difference vector is calculated by vector subtraction operation: for each parameter dimension, subtract the last global model parameter value from the current local model parameter value to form the difference value of the corresponding dimension, and finally combine to form a complete difference vector. For example, if the model of a certain edge node contains 1000 parameters, the parameter vector of the last global model is [w1, w2,..., w1000], and the current local model parameter vector is [wi1, wi2,..., wi1000], then the parameter difference vector is [wi1-w1, wi2-w2,..., wi1000-w1000]. The difference vector directly reflects the deviation degree of the local model in the training process. The greater the length of the difference vector, the worse the consistency between the local model and the global model, and the more necessary it is to realize synchronization through parameter update. It provides basic data support for subsequent emergency degree quantification, and avoids the limitations of subjective judgment of parameter update requirements.

[0055] S22, calculate the Euclidean norm of the model parameter difference vector of each edge node as the model parameter update amplitude of each edge node; obtain the mean of the model parameter update amplitudes of all edge nodes, divide the model parameter update amplitude of each edge node by the mean of the model parameter update amplitudes of all edge nodes, and take the division result as the parameter update amplitude difference degree of each edge node; in this embodiment, the multi-dimensional parameter difference vector is converted into a single-dimensional quantitative index by the Euclidean norm, and the influence of the parameter scale difference between nodes is eliminated by normalization processing, so as to realize the horizontal comparability of the update amplitude. First, the model parameter update amplitude is calculated: the calculation method of the Euclidean norm is to take the square root of the sum of squares of all elements in the parameter difference vector. For example, if the parameter difference vector of a node is [0.1, -0.2, 0.3], the Euclidean norm is about 0.374, which is the model parameter update amplitude of the node, and the greater the value, the more significant the deviation between the local model and the global model. Then, the mean of the update amplitudes of all edge nodes is calculated: after the center node collects the update amplitudes of all nodes, the mean is calculated by the arithmetic average formula. Finally, divide the update amplitude of each node by the mean to obtain the parameter update amplitude difference degree, for example, if the update amplitude of a node is 0.5 and the mean of all nodes is 0.25, the difference degree is 2, which represents that the parameter deviation of the node is 2 times the average level. In this embodiment, the multi-dimensional parameter difference is reduced to a single value index that can be directly compared, and the interference caused by different model parameter scales is eliminated by mean normalization processing, so as to ensure that the update amplitude difference degree of the node running any model can be measured on a unified scale, and provide a fair comparison benchmark for subsequent emergency degree analysis.

[0056] S23, obtain the CPU usage rate, memory usage rate and network bandwidth usage rate in the node running state data of each edge node; analyze the resource tightness of each edge node; specifically, the arithmetic mean of the CPU usage rate, memory usage rate and network bandwidth usage rate is taken as the resource tightness of each edge node; this embodiment quantifies the resource load pressure of the edge node by fusing the usage of the three core resources of CPU, memory and network, and provides the basis of node capability dimension for judging the parameter update urgency. First, the acquisition of resource usage rate data depends on the system monitoring agent of the edge node, which calculates the usage rate of each resource in real time by reading the kernel statistical information of the operating system (for example, the CPU usage is obtained through the / proc / stat file, the memory usage is obtained through the / proc / meminfo, and the network bandwidth usage is obtained through the / proc / net / dev): the CPU usage rate is the proportion of CPU busy time in total time in a unit time, the memory usage rate is the proportion of used memory capacity in total memory capacity, and the network bandwidth usage rate is the proportion of current actual transmission rate in maximum available bandwidth. For example, the CPU usage rate of a certain edge node is 80%, the memory usage rate is 70%, and the network bandwidth usage rate is 60%, so the resource tightness is 70% calculated by the arithmetic mean formula. In this embodiment, CPU, memory and network are the core resources of edge node running model training and parameter transmission, and the overload of any one resource may lead to node service degradation, therefore, by comprehensively considering the load of the three, the resource tightness of the node can be fully reflected, the higher the resource tightness, the less the remaining resources of the node, and the weaker the ability to process new tasks, at this time, if the parameter update is not processed in priority, parameter accumulation or node crash may occur. For example, in the vehicle networking edge computing scene, a roadside edge node processes real-time data of multiple vehicles at the same time, the CPU usage rate is 90%, the memory usage rate is 85%, and the network bandwidth usage rate is 80%, so the resource tightness is 85%, which indicates that the node is in a high load state, and its parameters need to be synchronized in priority to avoid affecting the real-time performance of vehicle data processing.

[0057] S24, obtaining the mean value of all edge node resource tightness, dividing the resource tightness of each edge node by the mean value of all edge node resource tightness, and taking the result of the division as the resource tightness difference degree of each edge node; in this embodiment, the resource tightness of each node is converted into a relative index through mean value normalization processing, realizing horizontal comparison of resource load pressure among different nodes, and providing a unified scale resource state basis for emergency degree analysis. First, the center node collects the resource tightness of all edge nodes through the data collection protocol, and then calculates the mean value. For example, assuming that there are 3 edge nodes, the resource tightnesses are 70%, 50% and 60% respectively, and the mean value is 60%. Then, divide the resource tightness of each node by the mean value to obtain the resource tightness difference degree: for example, the difference degree of the first node is about 1.167, that of the second node is about 0.833, and that of the third node is 1.0. In this embodiment, the resource tightness difference degree refers to the deviation degree of the resource tightness of the edge node relative to the overall average level: when it is greater than 1, it means that the resource load of the node is higher than the average level, and the larger the value, the more resource is tight; if it is less than 1, it means that it is lower than the average level, and the resource is relatively abundant. This embodiment eliminates the influence of the resource configuration difference of different edge node clusters, for example, the resource tightness of the edge node with lower configuration may be generally higher than that of the node with higher configuration, and if the absolute value is directly compared, the actual load pressure of the high-configuration node may be misjudged, and after mean value normalization, no matter how the node hardware is configured, the resource tightness difference degree can objectively reflect the relative load state of the node in the current cluster, ensuring the fairness and accuracy of the subsequent emergency degree analysis.

[0058] S25, multiply the parameter update amplitude difference degree of each edge node with the resource shortage difference degree, and take the multiplication result as the model parameter update urgency of each edge node. In this embodiment, by coupling the parameter update demand and the node resource state, a final urgency quantization index is constructed, and an explicit priority basis is provided for subsequent transmission scheduling. In this embodiment, the parameter update urgency depends on both the deviation size of the local model and the global model (the parameter update amplitude difference degree) and the resource load pressure of the node itself (the resource shortage difference degree). Only when both are at a high level, the parameter update demand of the node is most urgent. For example, the parameter update amplitude difference degree of a certain edge node is 1.8, its parameter deviation is significantly higher than the average level, the resource shortage difference degree is 1.5, and its resource load is significantly higher than the average level, so its urgency is 2.7; while the parameter update amplitude difference degree of another node is 2.0, its deviation is larger, but the resource shortage difference degree is 0.6, and its resource is abundant, so its urgency is 2.0*0.6=1.2. Obviously, the urgency of the former is higher, and it should be scheduled preferentially. In this embodiment, by means of multiplication coupling, the limitations of single-dimensional analysis are avoided: if only the parameter deviation is concerned, the non-urgent update of the resource-abundant node may be placed in front, resulting in the accumulation of parameters of the resource-shortage node; if only the resource shortage is concerned, the node with smaller parameter deviation may be scheduled preferentially, affecting the convergence efficiency of the global model. Through multiplication operation, this embodiment realizes accurate distinction between the nodes that need to be updated and the nodes that urgently need to be updated, and ensures that the nodes with high urgency can obtain transmission resources preferentially. In practical applications, for example, in the environmental monitoring edge cluster of smart city, a node responsible for PM2.5 monitoring has a large model parameter deviation due to sudden changes in sensor data, with an amplitude difference degree of 1.6, and is short of resources due to processing multiple monitoring tasks, with a shortage difference degree of 1.4. The urgency of the node is 2.24, and the node will obtain parameter synchronization resources preferentially, ensuring the accuracy of environmental monitoring data and avoiding warning errors caused by parameter lag.

[0059] In this embodiment, in step S3, based on the node running state data and the network topology data, a data transmission potential analysis model is constructed to analyze the data transmission potential of each edge node under the comprehensive resource constraint, including the following specific steps:

[0060] S31, extract the available network bandwidth, CPU idle rate, and memory idle rate of each edge node from the node running state data of each edge node; calculate the bandwidth index, CPU idle index, and memory idle index of each edge node; specifically, calculate the ratio of the available network bandwidth of each edge node to the average available network bandwidth of all edge nodes as the bandwidth index, calculate the ratio of the CPU idle rate of each edge node to the average CPU idle rate of all edge nodes as the CPU idle index, and calculate the ratio of the memory idle rate of each edge node to the average memory idle rate of all edge nodes as the memory idle index; the embodiment quantifies the remaining resource capacity of the node and performs normalization processing to construct a single-dimension evaluation index of transmission resources, providing a basis for subsequent calculation of the transmission resource index. First, the extraction of core resource data depends on the system monitoring module of the edge node: the available network bandwidth is obtained by monitoring the difference between the link transmission rate and the maximum bandwidth in real time; the CPU idle rate is the proportion of the CPU idle time in the total time in a unit of time; the memory idle rate is the proportion of the unused memory capacity in the total memory capacity. These data are collected periodically to ensure real-time, and after collection, they are uploaded to the center node through a data transmission protocol. Subsequently, the center node calculates the global average of each resource, including: the average available network bandwidth, the average CPU idle rate, and the average memory idle rate. Finally, the bandwidth index, CPU idle index, and memory idle index of each node are calculated. For example, the available bandwidth of a certain node is 80 Mbps, and the global average is 50 Mbps, so the bandwidth index is 1.6; the CPU idle rate is 30%, and the global average is 20%, so the CPU idle index is 1.5; the memory idle rate is 25%, and the global average is 20%, so the memory idle index is 1.25. The embodiment quantifies the remaining capacity of a certain resource of a node relative to the global average level by the bandwidth index, CPU idle index, and memory idle index of each node: when a certain index is greater than 1, it indicates that the remaining capacity of the resource is higher than the average level, and the larger the value, the more abundant the resource; when a certain index is less than 1, it indicates that it is lower than the average level, and the resource is relatively tight. The embodiment converts the resource remaining capacity of different dimensions into comparable relative indexes, eliminates the interference caused by the difference in node hardware configuration, ensures that each resource index is measured on a unified scale, and provides a fair basis for subsequent calculation of the comprehensive transmission resource index.

[0061] S32, geometrically average the bandwidth index, CPU idle index, and memory idle index of each edge node, and take the geometric average operation result as the transmission resource index of each edge node; this embodiment fuses multi-dimensional resource indexes by geometric average to construct a comprehensive transmission resource evaluation index, which accurately reflects the actual ability of the node to carry transmission tasks. The core reason for this embodiment to choose geometric average instead of arithmetic average lies in the need to conform to the short-board constraint characteristics of data transmission; in the data transmission process, bandwidth, CPU, and memory are indispensable, and any insufficient resource will become a transmission bottleneck, and geometric average can amplify the influence of the short board, avoiding the short-board covering by the long board in arithmetic average. Taking a node as an example, its bandwidth index is 1.4, CPU idle index is 1.5, and memory idle index is 1.5; first, the product of the three is calculated as 3.15, and then 3.15 is cubed to obtain a transmission resource index of about 1.47. If another node has a bandwidth index of 0.8, a CPU idle index of 1.6, and a memory idle index of 1.6, the product is 2.048, and the transmission resource index is about 1.27; although the latter two indexes are higher, the overall transmission resource index is significantly reduced due to the bandwidth short board, which conforms to the rule that insufficient bandwidth will directly limit the data transmission rate in the actual transmission scenario. Further, to verify the effectiveness of the transmission resource index, this embodiment compares the results of the simulation experiment: three nodes with transmission resource indexes of 1.47, 1.27, and 1.0 are selected to simultaneously transmit 100MB parameter data, and the results show that the node with index 1.47 takes 2.1 seconds to transmit, the node with index 1.27 takes 2.8 seconds, and the node with index 1.0 takes 3.5 seconds, which is negatively correlated with the index, proving that the transmission resource index can effectively reflect the transmission efficiency. At the same time, the transmission resource index not only integrates the synergistic effect of multi-dimensional resources, but also highlights the constraint influence of the short board resource, providing an accurate "resource capability portrait" for subsequent evaluation of transmission potential combined with network topology.

[0062] S33, obtain the adjacency matrix of each edge node in the network topology data between each edge node, take the number of edges directly connected to each edge node in the adjacency matrix of each edge node as the degree centrality of each edge node, and take the ratio of the degree centrality of each edge node to the average of the degree centralities of all edge nodes as the network centrality of each edge node; the embodiment evaluates the structural advantage of the node through network topology analysis, and quantifies the data forwarding and transmission efficiency potential of the node in the network. First, the construction of the adjacency matrix depends on periodic network detection: the central node detects the direct connection relationship between nodes by sending ICMP Echo request and Traceroute instruction to each edge node; if node A can directly receive the Ping response of node B, it is determined that A and B are directly connected. The detection period is set to 5 minutes to adapt to the dynamic change of the network topology. According to the detection result, the adjacency matrix is constructed: the matrix dimension is N x N, where N is the total number of nodes, if node i is directly connected to node j, then the matrix element a ij = 1, otherwise a ij = 0, and a ii = 0 (indicating that it is not connected to itself). For example, the adjacency matrix of 5 nodes may be: node 1 is [0, 1, 1, 0, 1], node 2 is [1, 0, 1, 1, 0], node 3 is [1, 1, 0, 1, 1], node 4 is [0, 1, 1, 0, 1], and node 5 is [1, 0, 1, 1, 0]; the degree centrality of a node is the number of "1"s in its corresponding matrix row (or column), representing the number of nodes directly connected to the node. Taking the above matrix as an example, the degree centrality of node 3 is 4 (directly connected to nodes 1, 2, 4 and 5), and the degree centrality of node 1 is 3 (connected to nodes 2, 3 and 5). If the degree centralities of the 5 nodes are 3, 3, 4, 3 and 3 respectively, the average of the global degree centralities is 3.2. The network centrality of node 3 is 1.25, and the network centrality of node 1 is about 0.9375. Among them, the network centrality greater than 1 indicates that the number of connections of the node is higher than the average level of the cluster, and the node is in the core position of the network, which can reduce the number of hops in data transmission and reduce the delay; the network centrality less than 1 is in the edge position, and the transmission efficiency is low. The embodiment converts the abstract network topology into a quantitative centrality index through adjacency matrix and degree centrality analysis, and provides an objective basis for evaluating the transmission structural advantage of the node.

[0063] S34, multiply the transmission resource index of each edge node with the network centrality, and take the multiplication result as the data transmission potential of each edge node under the comprehensive resource constraint. In this embodiment, by coupling the resource capability and the topological advantage, a data transmission potential quantification method is constructed to comprehensively evaluate the efficient transmission capability of the node under the comprehensive constraint. The multiplication operation realizes the synergy of hard resources and soft topology, that is, the data transmission potential of the edge node with abundant resources and topological core is the highest, and if only one of them is possessed, the data transmission potential is limited. In this embodiment, taking the actual edge nodes as an example, the transmission resource index of node A is 1.47, the network centrality is 1.25, and the data transmission potential is 1.8375; the transmission resource index of node B is 1.47, the network centrality is 0.9375, and the data transmission potential is about 1.378; the transmission resource index of node C is 1.27, the network centrality is 1.25, and the data transmission potential is 1.5875. By comparing the three, it can be seen that node A has a double advantage of resources and topology, and the data transmission potential is significantly higher than that of other nodes. At the same time, in order to verify the practicability of the data transmission potential, a comparative experiment is designed in this embodiment: 10 edge nodes are divided into high (>1.6), medium (1.2-1.6) and low (<1.2) groups according to the transmission potential, and 10 rounds of 100MB parameter data are transmitted in each group. The results show that the high potential group has an average transmission time of 2.0 seconds and a packet loss rate of 0.3%; the medium potential group has a time consumption of 2.6 seconds and a packet loss rate of 0.8%; and the low potential group has a time consumption of 3.8 seconds and a packet loss rate of 1.5%, which proves that the data transmission potential can effectively distinguish the transmission capability difference of the nodes. Based on the above content, this embodiment avoids the evaluation error of the disconnection of resources and topology, that is, if only the transmission resource index is considered, the capabilities of nodes A and B are equivalent, but after considering the topology, it can be seen that the actual transmission efficiency of node A is higher; if only the network centrality is considered, the topological advantages of nodes A and C are the same, but the resource support of node A is stronger.

[0064] In this embodiment, the emergency degree analysis result and the data transmission potential analysis result of the model parameter of each edge node are updated according to the parameters in step S4, and a data transmission priority evaluation model is constructed to quantitatively evaluate the priority of the model parameter transmission of each edge node; specifically including the following steps:

[0065] S41, obtaining the model parameter update emergency degree analysis result of each edge node obtained by analysis and the data transmission potential analysis result of each edge node under the comprehensive resource constraint;

[0066] S42, multiply the model parameter update urgency analysis result of each edge node and the data transmission potential analysis result of each edge node under the comprehensive resource constraint, and take the multiplication result as the priority of the model parameter transmission of each edge node. In this embodiment, the priority is generated by coupling the urgency and the transmission potential, so as to determine the transmission scheduling order of the node. In this embodiment, the multiplication is used to quantify the matching degree of the transmission demand and the transmission capacity; the higher the transmission demand and the stronger the transmission capacity of the edge node, the higher the priority. For example, the model parameter update urgency of node X is 2.2, the data transmission potential is 1.8, and the priority is 3.96; the model parameter update urgency of node Y is 1.5, the data transmission potential is 2.0, and the priority is 3.0. As can be seen from the comparison, node X has the highest priority due to the double advantages of demand and capacity, and should be scheduled first. In order to verify the scheduling effect of the priority of the model parameter transmission of each edge node, simulation experiments are performed in this embodiment: in a cluster containing 20 edge nodes, the parameter transmission is scheduled by using three strategies of priority sorting, urgency sorting and random sorting. The results show that the average transmission completion time of the priority sorting strategy is 15 seconds, which is increased by 31.8% compared with the urgency sorting only (22 seconds), and is increased by 57.1% compared with the random sorting (35 seconds), and the parameter synchronization success rate of the core node (the model parameter update urgency is greater than 2.0) is 99.2%, which is significantly higher than that of other strategies. In this embodiment, the optimal matching of demand and capacity is realized by simple and effective multiplication, the generated priority index is intuitive and quantifiable, which provides a clear decision basis for subsequent dynamic scheduling, and greatly improves the scheduling efficiency and reliability.

[0067] In this embodiment, the data transmission sequence between edge nodes is dynamically scheduled and optimized based on the model parameter transmission priority quantization evaluation result and the to-be-transmitted model parameter update data in step S5; specifically including the following contents:

[0068] S51, obtain the priority of the model parameter transmission of each edge node, and extract the to-be-transmitted model parameter update data on each edge node;

[0069] S52, based on the to-be-transmitted model parameter update data, analyze the data concentration degree of each edge node;

[0070] S53, multiply the transmission priority and the data concentration degree of each edge node to obtain the comprehensive scheduling weight of each edge node;

[0071] S54, construct the data transmission sequence between edge nodes according to the order from high to low of each comprehensive scheduling weight, and sequentially transmit data to each edge node according to the order of the data transmission sequence between edge nodes.

[0072] In the embodiment, the data concentration degree of each edge node is analyzed based on the model parameter update data to be transmitted in step S52; specifically including the following steps:

[0073] S521, extract the model parameter matrix to be transmitted from the model parameter update data to be transmitted of each edge node, count the number of zero-value elements in the model parameter matrix to be transmitted, and take the ratio of the number of zero-value elements to the total number of elements in the model parameter matrix to be transmitted as the parameter sparsity of the model parameter matrix to be transmitted on each edge node;

[0074] S522, extract all non-zero parameter values in the model parameter matrix to be transmitted of each edge node, and establish the data distribution range of the model parameter matrix to be transmitted of each edge node according to the order from the minimum value to the maximum value in all non-zero parameter values;

[0075] S523, for the data distribution range of the model parameter matrix to be transmitted of each edge node, divide the data distribution range into multiple intervals at equal intervals, count the number of elements falling into each interval, take the ratio of the number of elements in each interval to the total number of all non-zero parameter values as the probability mass function value corresponding to each interval, and import the probability mass function values corresponding to all intervals into the Shannon entropy formula to calculate the information entropy of the model parameter matrix to be transmitted of each edge node;

[0076] S524, obtain the information entropy of the model parameter matrix to be transmitted on each edge node based on step S523; divide the calculated parameter sparsity of the model parameter matrix to be transmitted on each edge node by the information entropy, and take the division result as the data concentration degree of the model parameter matrix to be transmitted on each edge node.

[0077] Exemplarily, the embodiment combines model parameter transmission priority with data characteristics to be transmitted, considers the data characteristics to be transmitted, and realizes efficient and reliable parameter synchronization through dynamic optimization of data transmission sequence. First, in the data preparation stage, the system synchronously obtains the transmission priority evaluation results of each edge node and the model parameter update data to be transmitted; the model parameter update data to be transmitted is extracted from the to-be-transmitted queue of each node, preprocessed by LZ4 compression to reduce the transmission amount, accompanied by an MD5 check code for subsequent integrity verification, and expired data generated more than 1 hour ago is filtered to avoid invalid transmission. Subsequently, the data characteristic analysis link is entered, the transmission efficiency is evaluated by calculating the data condensation degree, the parameter matrix to be transmitted is extracted, the proportion of zero-value elements is counted to obtain the parameter sparsity, and the compressible space on the data structure is reflected; the non-zero parameters are extracted, the data distribution range is determined and equally divided into 10 intervals, the Shannon entropy is calculated through the probability mass function, and the information density of the data is measured; finally, the data condensation degree is obtained by the ratio of the sparsity and the information entropy, and it should be noted that since the non-zero parameters are distributed in at least one interval, the probability mass function value is greater than 0, so the information entropy is always greater than 0, so the ratio is always meaningful, and the higher the data condensation degree, the easier the data is compressed and the higher the transmission efficiency. On this basis, the system multiplies the transmission priority and the data condensation degree to obtain the comprehensive scheduling weight, which reflects the matching degree of the node parameter update urgency and the transmission capacity, and also considers the data transmission efficiency, to ensure that the nodes with high priority and easy transmission obtain priority scheduling right. Finally, the sequence construction and transmission execution stage is entered, the transmission sequence is constructed from high to low according to the comprehensive scheduling weight, and the sequence adopts a 5-second rolling update mechanism; further, if the weight of a node decreases due to changes in resource load, the system will dynamically adjust the order and pause the transmission of low-weight nodes to process high-weight nodes first; during transmission, a connection is established through the TCP protocol, the link delay and packet loss rate are monitored in real time, and if the link quality is poor, the adjacent node is switched to forward; after transmission, the data integrity is confirmed through MD5 check, and if the check fails, up to 3 retransmissions are triggered, and if it still fails, it is marked as an exception and an alarm is given. Through the above, the embodiment can effectively solve the problem of imbalance between priority and efficiency in scheduling, ensure timely synchronization of core node parameters, maximize the use of network and node resources, and significantly improve the overall efficiency and reliability of distributed synchronization of edge cluster model parameters.

[0078] Embodiment 2

[0079] As Figure 2As shown, the embodiment provides a distributed data analysis system for edge node model parameters, comprising: an edge node data acquisition module, a parameter update urgency analysis module, a data transmission potential analysis module, a data transmission priority evaluation module, a node data transmission scheduling optimization module; wherein the edge node data acquisition module is used to acquire the local model parameter update data and node running state data of each edge node, and simultaneously acquire the network topology data and the to-be-transmitted model parameter update data between each edge node; the parameter update urgency analysis is used to construct a parameter update urgency analysis model based on the local model parameter update data and the node running state data of each edge node, and analyze the model parameter update urgency of each edge node; the data transmission potential analysis module is used to construct a data transmission potential analysis model based on the node running state data and the network topology data, and analyze the data transmission potential of each edge node under the comprehensive resource constraint; the data transmission priority evaluation module is used to construct a data transmission priority evaluation model according to the parameter update urgency analysis result and the data transmission potential analysis result, and quantitatively evaluate the priority of model parameter transmission of each edge node; the node data transmission scheduling optimization module is used to dynamically schedule and optimize the data transmission sequence between edge nodes based on the model parameter transmission priority quantitative evaluation result and the to-be-transmitted model parameter update data.

[0080] The steps of implementing the respective functions of each parameter and each unit module in the distributed data analysis system for edge node model parameters of the application described above can refer to each parameter and step in the embodiments of the distributed data analysis method for edge node model parameters described above, and will not be repeated here.

[0081] Embodiment 3

[0082] An electronic device according to an embodiment of the application comprises a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes the distributed data analysis method for edge node model parameters by calling the computer program stored in the memory. It should be noted that all computer programs of the distributed data analysis method for edge node model parameters are implemented using C language, wherein the edge node data acquisition module, the parameter update urgency analysis module, the data transmission potential analysis module, the data transmission priority evaluation module, and the node data transmission scheduling optimization module are all controlled by a remote server.

[0083] In the description of the specification, reference to "one embodiment", "an example", "a specific example" or the like means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the phrases "in one embodiment", "an example", "a specific example" or the like in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0084] The preferred embodiments of the application disclosed above are only to help explain the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments described. Obviously, many modifications and variations can be made in light of the contents of the specification. The specification selects and specifically describes these embodiments in order to better explain the principles and practical application of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A distributed data analysis system for edge node model parameters, characterized in that, The edge node data acquisition module, the parameter update urgency analysis module, the data transmission potential analysis module, the data transmission priority evaluation module, and the node data transmission scheduling optimization module are included. The edge node data acquisition module is configured to acquire local model parameter update data and node operating state data of each edge node, and simultaneously acquire network topology data and to-be-transmitted model parameter update data between each edge node. The parameter update urgency analysis module is configured to construct a parameter update urgency analysis model based on the local model parameter update data and the node operating state data of each edge node, and analyze the model parameter update urgency of each edge node. The data transmission potential analysis module is configured to construct a data transmission potential analysis model based on the node operating state data and the network topology data, and analyze the data transmission potential of each edge node under comprehensive resource constraints. The data transmission priority evaluation module is configured to construct a data transmission priority evaluation model based on the parameter update urgency analysis result and the data transmission potential analysis result, and quantitatively evaluate the priority of model parameter transmission of each edge node. The node data transmission scheduling optimization module is configured to dynamically schedule and optimize the data transmission sequence between edge nodes based on the quantitative evaluation result of the model parameter transmission priority and the to-be-transmitted model parameter update data.

2. The distributed data analytics system of edge node model parameters according to claim 1, characterized in that, The parameter update urgency analysis model is constructed based on the local model parameter update data and the node operating state data of each edge node, and the model parameter update urgency of each edge node is analyzed, including the following specific steps: S21, the current local model parameter of each edge node in the local model parameter update data of each edge node is obtained, and the difference between the current local model parameter and the previous global model parameter is obtained as the model parameter difference vector of each edge node; S22, the Euclidean norm of the model parameter difference vector of each edge node is calculated as the model parameter update amplitude of each edge node; the mean value of the model parameter update amplitudes of all edge nodes is obtained, and the model parameter update amplitude of each edge node is divided by the mean value of the model parameter update amplitudes of all edge nodes; the division result is taken as the parameter update amplitude difference degree of each edge node; S23, the CPU usage, memory usage, and network bandwidth usage in the node operating state data of each edge node are obtained; and the resource tightness of each edge node is analyzed; S24, the mean value of the resource tightness of all edge nodes is obtained, and the resource tightness of each edge node is divided by the mean value of the resource tightness of all edge nodes; the division result is taken as the resource tightness difference degree of each edge node; S25, the parameter update amplitude difference degree and the resource tightness difference degree of each edge node are multiplied; and the multiplication result is taken as the model parameter update urgency of each edge node.

3. The distributed data analytics system of edge node model parameters according to claim 2, characterized in that, The node running state data and the network topology data are used to construct a data transmission potential analysis model to analyze the data transmission potential of each edge node under the comprehensive resource constraint, including the following specific steps: S31, extracting the available network bandwidth, CPU idle rate and memory idle rate of each edge node from the node running state data of each edge node; calculating the bandwidth index, CPU idle index and memory idle index of each edge node; S32, performing geometric mean operation on the bandwidth index, CPU idle index and memory idle index of each edge node, and taking the geometric mean operation result as the transmission resource index of each edge node; S33, obtaining the adjacency matrix of each edge node in the network topology data between the edge nodes, taking the number of edges directly connected between each edge node in the adjacency matrix of each edge node as the degree centrality of each edge node, and taking the ratio of the degree centrality of each edge node to the average degree centrality of all edge nodes as the network centrality of each edge node; S34, performing multiplication operation on the transmission resource index and the network centrality of each edge node, and taking the multiplication operation result as the data transmission potential of each edge node under the comprehensive resource constraint.

4. The distributed data analytics system of edge node model parameters according to claim 3, characterized in that, The parameter updating emergency degree analysis result and the data transmission potential analysis result are used to construct a data transmission priority evaluation model to quantitatively evaluate the priority of model parameter transmission of each edge node; specifically including the following steps: S41, obtaining the model parameter updating emergency degree analysis result of each edge node and the data transmission potential analysis result of each edge node under the comprehensive resource constraint; S42, performing multiplication operation on the model parameter updating emergency degree analysis result of each edge node and the data transmission potential analysis result of each edge node under the comprehensive resource constraint, and taking the multiplication operation result as the priority of model parameter transmission of each edge node.

5. The distributed data analysis system of edge node model parameters according to claim 4, characterized in that, Based on the model parameter transmission priority quantitative evaluation result and the to-be-transmitted model parameter update data, the data transmission sequence between the edge nodes is dynamically scheduled and optimized; specifically including the following contents: S51, obtaining the priority of model parameter transmission of each edge node, and extracting the to-be-transmitted model parameter update data on each edge node; S52, analyzing the data condensation degree of each edge node based on the to-be-transmitted model parameter update data; S53, multiplying the transmission priority and the data condensation degree of each edge node to obtain the comprehensive scheduling weight of each edge node; S54, constructing the data transmission sequence between the edge nodes according to the order from high to low of the comprehensive scheduling weight, and sequentially transmitting data to each edge node according to the order of the data transmission sequence between the edge nodes.

6. The distributed data analytics system of edge node model parameters according to claim 5, characterized in that, In step S52, the data condensation degree of each edge node is analyzed based on the to-be-transmitted model parameter update data; specifically including the following steps: S521, extracting the to-be-transmitted model parameter matrix from the to-be-transmitted model parameter update data of each edge node, and counting the number of zero value elements in the to-be-transmitted model parameter matrix; taking the ratio of the number of zero value elements to the total number of elements in the to-be-transmitted model parameter matrix as the parameter sparsity of the to-be-transmitted model parameter matrix on each edge node; S522, extract all non-zero parameter values in each edge node to-be-transmitted model parameter matrix, and determine the data distribution range of each edge node to-be-transmitted model parameter matrix according to the order from the minimum value to the maximum value in all non-zero parameter values; S523, for the data distribution range of each edge node to-be-transmitted model parameter matrix, divide the data distribution range into multiple intervals at equal intervals, count the number of elements falling into each interval, and take the ratio of the number of elements in each interval to the total number of all non-zero parameter values as the probability mass function value corresponding to each interval; import the probability mass function values corresponding to all intervals into the Shannon entropy formula for calculation to obtain the information entropy of each edge node to-be-transmitted model parameter matrix; S524, based on step S523, obtain the information entropy of the to-be-transmitted model parameter matrix on each edge node; divide the calculated parameter sparsity of the to-be-transmitted model parameter matrix on each edge node by the information entropy, and take the division result as the data condensation degree of the to-be-transmitted model parameter matrix on each edge node.

7. A method for distributed data analysis of edge node model parameters, implemented on the basis of the distributed data analysis system of edge node model parameters according to any one of claims 1 to 6, characterized in that, The method comprises the following specific steps: Obtain local model parameter update data and node running state data of each edge node, and obtain network topology data and to-be-transmitted model parameter update data among the edge nodes; Based on the local model parameter update data and the node running state data of each edge node, construct a parameter update urgency analysis model to analyze the model parameter update urgency of each edge node; Based on the node running state data and the network topology data, construct a data transmission potential analysis model to analyze the data transmission potential of each edge node under the comprehensive resource constraint; According to the parameter update urgency analysis result and the data transmission potential analysis result, construct a data transmission priority evaluation model to quantitatively evaluate the priority of model parameter transmission of each edge node; Based on the quantitative evaluation result of the model parameter transmission priority and the to-be-transmitted model parameter update data, dynamically schedule and optimize the data transmission sequence among the edge nodes.

8. An electronic device comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes the distributed data analysis method of the edge node model parameter by calling the computer program stored in the memory according to claim 7.

Images