Transmission data gradient sparse method based on distributed reasoning

Abstract

The invention relates to the technical field of data gradient sparseness, in particular to a transmission data gradient sparseness method based on distributed reasoning, which comprises the following steps: confirming an original neural network model of a distributed computational node, performing gradient iterative computation on the original neural network model to obtain a plurality of model update gradient value sets, and performing gradient value aggregation on the plurality of model updating gradient value sets based on the to-be-trained model to obtain a local updating gradient value set, performing data packaging on the local updating gradient value set to obtain computational node gradient data, and updating the to-be-trained model in the reasoning main node by using the computational node gradient data to obtain a global updating model. According to the method, the training efficiency and the resource utilization rate of the global model in federated learning can be improved, and the communication efficiency between the main node and the computing node is improved.

Description

Technical Field

[0001] This invention relates to the field of data gradient sparsity technology, and in particular to a method for transmitting data gradient sparsity based on distributed inference. Background Technology

[0002] With the rapid development of the Internet of Things and edge computing, distributed machine learning has become a key technology for processing massive amounts of scattered data. Against this backdrop, the collaborative training of distributed inference nodes faces severe communication bottleneck challenges. The gradient sparsity technique for transmitting data aims to significantly reduce the amount of communication data between distributed nodes and the central server without significantly affecting the model's convergence accuracy. This is of vital practical significance for building an efficient and scalable distributed training system.

[0003] Traditional methods typically employ a fixed gradient sparsity ratio combined with a synchronous gradient aggregation mechanism. This is achieved by synchronizing all nodes with the server after each local iteration. However, a fixed sparsity strategy cannot adapt to the differences in importance between different layers in a neural network, which may lead to the over-pruning of key gradient information or redundant transmission of minor gradients. Furthermore, the strict synchronization mechanism requires nodes with high computing power to wait for slower nodes, resulting in severe idleness of cluster computing resources. Summary of the Invention

[0004] This invention provides a gradient sparsity method for transmitting data based on distributed inference and a computer-readable storage medium. Its main purpose is to improve the training efficiency and resource utilization of the global model in federated learning, and to enhance the communication efficiency between the master node and the computing node.

[0005] To achieve the above objectives, this invention provides a gradient sparsity method for transmitted data based on distributed inference, comprising: Obtain the inference master node, and identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes; The model to be trained is determined based on the inference master node, and the model to be trained is transmitted to each distributed inference node in the distributed inference node set to obtain the distributed computing node set; For each distributed computing node in the distributed computing node set, the following operations are performed to identify the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; Gradient iteration calculation is performed on the original neural network model to obtain multiple sets of model update gradient values; Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set; Pack the locally updated gradient value set to obtain the gradient data of the computing nodes, and summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes. The training model in the inference master node is updated using the gradient dataset of the computing nodes to obtain the global update model. The gradient sparsity of the transmission data under distributed inference is then achieved based on the global update model.

[0006] Optionally, the step of performing gradient iteration calculation on the original neural network model to obtain multiple sets of model update gradient values ​​includes: The original neural network model is updated locally to obtain the model update gradient value set. The parameters of the original neural network model are updated using the model update gradient value set to obtain the updated neural network model. Determine whether a preset global model update command has been received; If no global model update instruction is received, the updated neural network model will be used as the original neural network model, and the step of performing local gradient update on the original neural network model will be returned until a global model update instruction is received. If a global model update instruction is received, the model update gradient value set is aggregated to obtain multiple model update gradient value sets.

[0007] Optionally, the step of performing local gradient update on the original neural network model to obtain a set of model update gradient values ​​includes: In the original neural network model, network structure layers are extracted sequentially from multiple network structure layers, and the extracted network structure layers are denoted as gradient sparse layers, where the gradient sparse layer includes multiple layer parameters. Compression discrimination is performed on the gradient sparse layer to obtain the compression discrimination result, where the compression discrimination result is either compressible or incompressible; If the compression determination result is compressible, then the gradient sparse layer is recorded as the structure layer to be compressed; Gradient sparsification is performed on the structure layer to be compressed to obtain a sparse update gradient value set; Summarize the sparse update gradient value set to obtain multiple sparse update gradient value sets, and generate the model update gradient value set based on the multiple sparse update gradient value sets.

[0008] Optionally, the step of performing compression discrimination on the gradient sparse layer to obtain the compression discrimination result includes: Based on the original neural network model, update parameter statistics are performed to obtain the total number of parameters to be updated; Hierarchical information degree analysis is performed on the gradient sparse layer to obtain the hierarchical information degree; Gradient characteristic analysis is performed based on multiple layer parameters to obtain the gradient update importance factor; The gradient compression threshold is calculated based on the total number of parameters to be updated, the hierarchical information content, and the gradient update importance factor. The gradient compression threshold is expressed as: , in, Indicates the gradient compression threshold. This indicates the total number of parameters to be updated. This represents the gradient update importance factor. Indicates the degree of hierarchical information. This indicates the preset adjustment coefficient; The number of currently updated parameters is calculated based on the parameters of multiple layers in the gradient sparse layer. If the number of updated parameters is greater than the gradient compression threshold, then the compressibility is recorded as the compression judgment result; If the number of updated parameters is not greater than the gradient compression threshold, then incompressibility is recorded as the compression result.

[0009] Optionally, the step of performing hierarchical information analysis on the gradient sparse layer to obtain hierarchical information includes: In the original neural network model, determine the layer index of the gradient sparse layer, query the output neurons of the gradient sparse layer, and obtain the number of output neurons. The hierarchical information degree is calculated based on the layer index, the number of output neurons, and the number of currently updated parameters. The hierarchical information degree is expressed as: , in, Representation layer index, Indicates the number of output neurons. Indicates the number of parameters currently being updated. This represents a logarithmic function with the natural constant as its base.

[0010] Optionally, the step of performing gradient characteristic analysis based on multiple layer parameters to obtain the gradient update importance factor includes: Perform the following operation for each of the multiple layer parameters: Based on the layer parameters, the set of historical model parameters is determined in the training model of the inference master node. The set of historical model parameters includes multiple historical model parameters. The historical model gradient value set is queried based on the historical model parameter set. The historical model gradient value set includes multiple historical model gradient values, and each historical model gradient value corresponds one-to-one with a historical model parameter. Calculate relevant values ​​for model update based on historical model parameter set and historical model gradient value set; Summarize the model update-related values ​​corresponding to the parameters of each layer to obtain multiple model update-related values; Based on the original neural network model, obtain multiple current model gradient values ​​corresponding to multiple layer parameters, where each current model gradient value corresponds one-to-one with a layer parameter; Multiple layer parameter correlation values ​​are determined from multiple model update correlation values ​​based on multiple layer parameter correlation values, and gradient update importance factors are calculated based on multiple layer parameter correlation values ​​and multiple current model gradient values.

[0011] Optionally, the step of performing gradient sparsification on the structure layer to be compressed to obtain a sparse update gradient value set includes: Determine multiple parameters to be compressed in the structure layer to be compressed, and obtain multiple gradient values ​​to be compressed corresponding to the multiple parameters to be compressed; The dynamic sparsity ratio is calculated based on the gradient compression threshold and the preset reference sparsity ratio. Based on the dynamic sparsity ratio, multiple retained parameters are identified among multiple parameters to be compressed, and multiple retained gradient values ​​of multiple retained parameters are determined among multiple gradient values ​​to be compressed. By merging multiple retained gradient values ​​and multiple retained parameters, a sparse update gradient value set is obtained.

[0012] Optionally, before determining whether a preset global model update instruction has been received, the method further includes: Let the set of distributed computing nodes be denoted as the set of distributed communication nodes. Perform the following operation on each distributed communication node in the set of distributed communication nodes: The model update is determined based on the distributed communication nodes, and the model update result is obtained, which is either completed or not completed. If the model update result is that the update is not completed, then return to the step of judging the local model update based on the distributed communication node until the model update result is that the update is completed. If the model update result indicates that the update is complete, then a master node communication instruction is generated and returned to the inference master node. If the master node communication instructions generated by each distributed communication node are returned to the inference master node, then a global model update instruction is generated based on the inference master node.

[0013] Optionally, the step of aggregating gradient values ​​from multiple model update gradient value sets to obtain a local update gradient value set includes: Determine the set of parameters for the model to be updated based on the model to be trained; For each model parameter in the set of model parameters to be updated, perform the following operation: Identify multiple same-parameter update gradient values ​​corresponding to the model parameters to be updated in multiple model update gradient value sets, wherein the same-parameter update gradient values ​​correspond one-to-one with the model update gradient value set; The local update gradient value is obtained by summing multiple update gradient values ​​with the same parameters. Summarize the local update gradient values ​​corresponding to each model parameter to be updated to obtain the local update gradient value set.

[0014] To achieve the above objectives, the present invention also provides a gradient-sparse transmission data system based on distributed inference, comprising: The training model transmission module is used to obtain the inference master node, identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes, determine the model to be trained according to the inference master node, and transmit the model to be trained to each distributed inference node in the distributed inference node set to obtain the distributed computing node set. The local model determination module is used to perform the following operations on each distributed computing node in the distributed computing node set to determine the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; The gradient calculation module is used to perform gradient iterative calculation on the original neural network model to obtain multiple model update gradient value sets. Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set. The global model update module is used to package the local update gradient value set to obtain the gradient data of the computing nodes, summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes, and use the gradient dataset of the computing nodes to update the model to be trained in the inference master node to obtain the global update model.

[0015] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the gradient sparsity method for transmitted data based on distributed inference described above.

[0016] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned gradient sparsity method for transmitted data based on distributed inference.

[0017] To address the problems described in the background, this invention first performs gradient iteration calculations on the original neural network model to obtain multiple sets of model update gradient values. This step effectively solves the synchronization waiting problem caused by differences in node computing and communication capabilities in distributed training by introducing a global model update instruction mechanism. In traditional methods, fast nodes must passively wait for slow nodes after completing local computation, resulting in a large amount of idle computing resources. This solution allows nodes to continuously perform local gradient iterations before receiving instructions, converting waiting time into effective computation time, thereby significantly improving the overall computing resource utilization and training efficiency of the cluster. Furthermore, this step employs a dynamic adaptive gradient sparsity strategy instead of the traditional fixed-ratio compression. Traditional methods use a uniform sparsity rate for all network layers, ignoring the importance and gradient characteristics differences of different layers in the model, which can easily lead to the loss of key information or inadequate compression. This scheme dynamically calculates the compression threshold and sparsity ratio for each layer by comprehensively considering factors such as hierarchical information content and gradient importance. This achieves intelligent protection of key gradients while minimizing the amount of communication data, better balancing communication efficiency and model convergence accuracy. Furthermore, this scheme aggregates gradient values ​​from multiple model update gradient sets based on the model to be trained, obtaining a local update gradient set. This step aggregates the model update gradient sets obtained by the same distributed computing node in different iterations, enabling each distributed computing node to upload comprehensive gradient information representing multiple rounds of local iterative training in a single communication interaction with the inference master node. Compared to the traditional federated learning model, which requires communication with the server after each local iteration, this step significantly reduces the communication frequency between nodes and the master node, effectively reducing network bandwidth consumption and communication latency. Therefore, this invention can improve the training efficiency and resource utilization of the global model in federated learning, and enhance the communication efficiency between the master node and computing nodes. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a gradient sparsity method for transmitted data based on distributed inference, provided in an embodiment of the present invention; Figure 2 A functional block diagram of a gradient sparse transmission data system based on distributed inference provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the gradient sparsity method for transmitted data based on distributed inference, according to an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] This application provides a gradient sparsity method for transmitted data based on distributed inference. The executing entity of this method includes, but is not limited to, at least one electronic device configured to execute the method provided in this application, such as a server or a terminal. In other words, the gradient sparsity method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating a gradient sparsity method for transmitted data based on distributed inference, according to an embodiment of the present invention. In this embodiment, the gradient sparsity method for transmitted data based on distributed inference includes: S1. Obtain the inference master node, and identify the distributed inference node set based on the inference master node. The distributed inference node set includes multiple distributed inference nodes.

[0024] As is clear, the inference master node refers to a server used to coordinate and manage all distributed inference nodes in the distributed inference node set. For example, it is a parameter server responsible for maintaining the global model (such as the subsequent model to be trained) and performing global model updates. The distributed inference node set refers to a collection of multiple distributed inference nodes, where a distributed inference node refers to a terminal device that performs federated learning on the model to be trained sent by the inference master node. For example, it is an edge server or IoT device with local computing capabilities and storing private data.

[0025] S2. Determine the model to be trained based on the inference master node, and transmit the model to be trained to each distributed inference node in the distributed inference node set to obtain the distributed computing node set.

[0026] It should be explained that the "model to be trained" refers to the initial global machine learning model maintained by the inference master node and distributed to various distributed inference nodes for collaborative training, such as a ResNet-50 image classification model pre-trained on the ImageNet dataset. The "set of distributed computing nodes" refers to a collection of multiple distributed computing nodes, where a distributed computing node is a distributed inference node that receives the model to be trained and uses its stored local data to train the parameters of the model.

[0027] S3. Perform the following operations on each distributed computing node in the distributed computing node set to identify the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers.

[0028] Understandably, the original neural network model refers to the model to be trained in the distributed computing nodes. The network structure layers refer to the structural layers that constitute the original neural network model. Each network structure layer performs specific linear or nonlinear transformations on the input data to progressively extract features, such as convolutional layers, pooling layers, and fully connected layers in a Convolutional Neural Network (CNN), or self-attention layers and feedforward neural network layers in a Transformer model. The aforementioned operation of performing the following on each distributed computing node in the distributed computing node set refers to: identifying the original neural network model of the distributed computing node; performing gradient iterative calculations on the original neural network model to obtain multiple model update gradient value sets; and packaging the local update gradient value sets to obtain the computing node gradient data.

[0029] S4. Perform gradient iteration calculations on the original neural network model to obtain multiple sets of model update gradient values.

[0030] It is clear that the model update gradient value set refers to the set of gradient values ​​of the original neural network model collected in a certain iteration of the gradient iteration process. The model update gradient value set refers to the set composed of multiple model update gradient values. The specific meaning of the model update gradient will be explained later.

[0031] In detail, the step of performing gradient iterative calculation on the original neural network model to obtain multiple sets of model update gradient values ​​includes: The original neural network model is updated locally to obtain the model update gradient value set. The parameters of the original neural network model are updated using the model update gradient value set to obtain the updated neural network model. Determine whether a preset global model update command has been received; If no global model update instruction is received, the updated neural network model will be used as the original neural network model, and the step of performing local gradient update on the original neural network model will be returned until a global model update instruction is received. If a global model update instruction is received, the model update gradient value set is aggregated to obtain multiple model update gradient value sets.

[0032] Understandably, the model update gradient value set refers to the set of gradient values ​​generated during the local gradient update process of all model parameters in the original neural network model that need to be updated. For example, if an original neural network model includes m weight parameters that need to be updated, such as the weight matrix and bias vector of a fully connected layer, and the convolutional kernel tensor of a convolutional layer, then each weight parameter corresponds to a model update gradient value. The model update gradient value refers to the partial derivative (i.e., gradient value) of the loss function calculated through the backpropagation algorithm with respect to the weight parameter corresponding to that model update gradient value. This model update gradient value quantifies the impact of adjusting the corresponding weight parameters on reducing the loss of the original neural network model (i.e., the function value of the aforementioned loss function). The specific meaning and calculation method of this model update gradient value are existing concepts and methods and will not be elaborated here. The specific method for obtaining this model update gradient value set will be given later.

[0033] Furthermore, the updated neural network model refers to the original neural network model after parameter updates. The specific steps for updating the parameters of the original neural network model using the model update gradient value set are as follows: Extract the model update gradient values ​​sequentially from the model update gradient value set; determine the original model parameters corresponding to the model update gradient values ​​in the original neural network model; and calculate the updated model parameters based on the original model parameters and the model update gradient values ​​using the following formula: ,in, This indicates updating the model parameters. Represents the original model parameters. This represents the learning rate of the original neural network model during the local gradient update process. This represents the model update gradient values. The updated model parameters corresponding to each updated gradient value are summarized to obtain the updated model parameter set. Finally, each updated model parameter in the updated model parameter set replaces the corresponding original model parameters in the original neural network model. The replaced original neural network model is the updated neural network model. The global model update instruction refers to the instruction initiated by the inference master node to update the model to be trained.

[0034] Furthermore, due to the varying computational capabilities and communication efficiency between the distributed computing nodes and the inference master node, if a distributed computing node sends its model update gradient value set to the inference master node immediately after completing the calculation, the inference master node will experience differences in the time it takes to receive the gradient value sets from each distributed computing node. This results in some highly efficient and computationally powerful distributed computing nodes having to wait for other distributed computing nodes to complete their calculations and communication with the inference master node before they can receive subsequent instructions (such as receiving the model to be trained and performing the next round of gradient updates). This leads to a significant waste of computational resources for these highly efficient and computationally powerful distributed computing nodes. Therefore, this scheme introduces the aforementioned global model update instruction. This global model update instruction is generated by the inference master node only after all distributed computing nodes have completed the calculation of the model update gradient value set. The inference master node then sends the global model update instruction to each distributed computing node. If a distributed computing node does not receive the global model update instruction, it needs to continue to perform gradient updates locally, i.e., the aforementioned step of using the updated neural network model as the original neural network model and returning to perform local gradient updates on the original neural network model. Through this step, the resource idle problem caused by distributed computing nodes with high communication efficiency and strong computing power waiting for other distributed computing nodes is effectively avoided, thereby improving the utilization of computing resources and training efficiency.

[0035] Specifically, the local gradient update of the original neural network model to obtain the model update gradient value set includes: In the original neural network model, network structure layers are extracted sequentially from multiple network structure layers, and the extracted network structure layers are denoted as gradient sparse layers, where the gradient sparse layer includes multiple layer parameters. Compression discrimination is performed on the gradient sparse layer to obtain the compression discrimination result, where the compression discrimination result is either compressible or incompressible; If the compression determination result is compressible, then the gradient sparse layer is recorded as the structure layer to be compressed; Gradient sparsification is performed on the structure layer to be compressed to obtain a sparse update gradient value set; Summarize the sparse update gradient value set to obtain multiple sparse update gradient value sets, and generate the model update gradient value set based on the multiple sparse update gradient value sets.

[0036] Understandably, the layer parameters refer to the weight parameters that need to be updated in the gradient sparse layer. The compression discrimination result refers to the result obtained after performing compression discrimination on the gradient sparse layer. The compression discrimination result is either compressible or incompressible. Compressible means that the number of gradients in the gradient sparse layer (i.e., the number of subsequent updated parameters) is greater than the dynamically calculated gradient compression threshold, indicating that the gradient sparse layer has high communication consumption and is suitable for compression, so the gradient sparse layer can be compressed. Incompressible means that the number of gradients in the gradient sparse layer does not exceed the gradient compression threshold, indicating that the parameters of the gradient sparse layer are small or highly important, and the risk of compression is high, i.e., the gradient sparse layer cannot be compressed. The sparse update gradient value set refers to the set of multiple sparse update gradient values ​​after gradient sparsification. The sparse update gradient value set represents the set of model update gradient values ​​corresponding to all weight parameters (i.e., layer parameters) in a gradient sparse layer. The step of performing gradient sparsification on the structure layer to be compressed will be described in detail later. The process of generating a model update gradient value set based on multiple sparse update gradient value sets refers to placing all the sparse update gradient values ​​from the multiple sparse update gradient value sets into the same data set, which is the model update gradient value set.

[0037] In detail, the compression discrimination of the gradient sparse layer to obtain the compression discrimination result includes: Based on the original neural network model, update parameter statistics are performed to obtain the total number of parameters to be updated; Hierarchical information degree analysis is performed on the gradient sparse layer to obtain the hierarchical information degree; Gradient characteristic analysis is performed based on multiple layer parameters to obtain the gradient update importance factor; The gradient compression threshold is calculated based on the total number of parameters to be updated, the hierarchical information content, and the gradient update importance factor. The gradient compression threshold is expressed as: , in, Indicates the gradient compression threshold. This indicates the total number of parameters to be updated. This represents the gradient update importance factor. Indicates the degree of hierarchical information. This indicates the preset adjustment coefficient; The number of currently updated parameters is calculated based on the parameters of multiple layers in the gradient sparse layer. If the number of updated parameters is greater than the gradient compression threshold, then the compressibility is recorded as the compression judgment result; If the number of updated parameters is not greater than the gradient compression threshold, then incompressibility is recorded as the compression result.

[0038] Understandably, the total number of parameters to be updated refers to the number of all weight parameters in the original neural network model that need to be updated. The layer informationality refers to a value quantifying the network depth of the gradient sparse layer. The higher the layer informationality, the higher the importance of the gradient sparse layer, meaning the lower the probability of compressing it (and the higher the subsequent gradient compression threshold). The gradient update importance factor refers to a value quantifying the concentration of gradient values ​​and their historical importance in the gradient sparse layer. The higher the gradient update importance factor, the higher the probability that the gradient of the gradient sparse layer is dominated by key parameters, and the greater the potential risk of compression, meaning the gradient sparse layer is less suitable for compression (and the higher the subsequent gradient compression threshold). The gradient compression threshold refers to the critical criterion for whether to perform compression on the gradient sparse layer. The higher the gradient compression threshold, the higher the risk of compressing the gradient sparse layer, and the more inclined to not compress it. If the current number of updated parameters is greater than the gradient compression threshold, it means that the communication overhead caused by the parameter size of the gradient sparse layer exceeds the potential risk of compressing it. In this case, compressibility can be recorded as the compression judgment result. If the number of currently updated parameters is not greater than the gradient compression threshold, it indicates that the parameter size of the gradient sparse layer is not large, or that the compression risk of the gradient sparse layer is high. In this case, incompressibility can be recorded as a compression judgment result. The number of currently updated parameters refers to the number of layer parameters among multiple layer parameters.

[0039] Furthermore, in the above formula for calculating the gradient compression threshold, The term indicates that when the total number of parameters to be updated is larger (more data needs to be transmitted for communication, i.e., greater communication overhead) or the gradient update importance factor is larger (higher compression risk), the gradient compression threshold is tended to be increased to protect the gradient sparse layer. In this term, the gradient update importance factor is dominant. When the gradient sparse layer contains more information, i.e., the hierarchical information is greater, a certain factor is introduced. This approach protects the gradient sparse layer from over-compression, meaning that hierarchical information content is now the dominant factor.

[0040] In detail, the hierarchical information degree analysis of the gradient sparse layer to obtain the hierarchical information degree includes: In the original neural network model, determine the layer index of the gradient sparse layer, query the output neurons of the gradient sparse layer, and obtain the number of output neurons. The hierarchical information degree is calculated based on the layer index, the number of output neurons, and the number of currently updated parameters. The hierarchical information degree is expressed as: , in, Representation layer index, Indicates the number of output neurons. Indicates the number of parameters currently being updated. This represents a logarithmic function with the natural constant as its base.

[0041] Understandably, the layer index refers to the sequential position number of the gradient sparse layer in the original neural network model. For example, in a VGG-16 model with 10 hidden layers, the layer index of the first convolutional layer is 1, and the layer index of the last fully connected layer is 10. This layer index represents the absolute depth position of the gradient sparse layer in the network topology of the original neural network model. The larger the layer index, the deeper the gradient sparse layer is in the original neural network model, and the more abstract and higher-level the features it learns, i.e., the greater the layer information. The number of output neurons refers to the total number of elements in the feature vector or feature map output by the gradient sparse layer after forward propagation. The larger the number of output neurons, the higher the dimension of the output features of the gradient sparse layer, and the stronger its expressive power and computational complexity, i.e., the greater the layer information. The larger the number of currently updated parameters, the greater the proportion of parameters in the original neural network model that the gradient sparse layer accounts for, i.e., the greater the layer information.

[0042] In detail, the step of performing gradient characteristic analysis based on multiple layer parameters to obtain the gradient update importance factor includes: Perform the following operation for each of the multiple layer parameters: Based on the layer parameters, the set of historical model parameters is determined in the training model of the inference master node. The set of historical model parameters includes multiple historical model parameters. The historical model gradient value set is queried based on the historical model parameter set. The historical model gradient value set includes multiple historical model gradient values, and each historical model gradient value corresponds one-to-one with a historical model parameter. Calculate relevant values ​​for model update based on historical model parameter set and historical model gradient value set; Summarize the model update-related values ​​corresponding to the parameters of each layer to obtain multiple model update-related values; Based on the original neural network model, obtain multiple current model gradient values ​​corresponding to multiple layer parameters, where each current model gradient value corresponds one-to-one with a layer parameter; Multiple layer parameter correlation values ​​are determined from multiple model update correlation values ​​based on multiple layer parameter correlation values, and gradient update importance factors are calculated based on multiple layer parameter correlation values ​​and multiple current model gradient values.

[0043] Understandably, the historical model parameter set refers to a collection of multiple historical model parameters, where each historical model parameter represents the value of the weight parameter corresponding to the layer parameter in the model to be trained during previous training processes. The historical model gradient value set refers to a collection of multiple historical model gradient values, where each historical model gradient value represents the gradient value of a specific historical model parameter in the historical model parameter set during the corresponding training process. The model update correlation value refers to a numerical value that quantifies the correlation between the historical model parameter set and the historical model gradient value set. Optionally, the Pearson correlation coefficient between the historical model parameter set and the historical model gradient value set is calculated, and the Pearson correlation coefficient is used as the model update correlation value. The larger the model update correlation value, the more significant the influence of the historical model parameter on the direction and magnitude of the model update during previous training processes.

[0044] It should be explained that the current model gradient value refers to the gradient value of the layer parameters during the update of the original neural network model. The layer parameter correlation value refers to the model update correlation value corresponding to the weight parameters at parameter positions related to the layer parameters. The larger the layer parameter correlation value, the greater the potential impact of the current model gradient value corresponding to that layer parameter correlation value on the model (i.e., the original neural network model) update, and the higher its importance. Therefore, when calculating the gradient update importance factor, the current model gradient value should be given a larger weight. The formula for calculating the gradient update importance factor based on multiple layer parameter correlation values ​​and multiple current model gradient values ​​is as follows: , in, This indicates the number of layer parameter correlation values ​​among multiple layer parameter correlation values, or the number of current model gradient values ​​among multiple current model gradient values. This represents the first of multiple layer parameter correlation values. Each layer parameter related value, and These represent the first of multiple current model gradient values. The current model gradient value and the... The current model gradient value.

[0045] Specifically, the gradient sparsification of the structure layer to be compressed to obtain a sparse update gradient value set includes: Determine multiple parameters to be compressed in the structure layer to be compressed, and obtain multiple gradient values ​​to be compressed corresponding to the multiple parameters to be compressed; The dynamic sparsity ratio is calculated based on the gradient compression threshold and the preset reference sparsity ratio. Based on the dynamic sparsity ratio, multiple retained parameters are identified among multiple parameters to be compressed, and multiple retained gradient values ​​of multiple retained parameters are determined among multiple gradient values ​​to be compressed. By merging multiple retained gradient values ​​and multiple retained parameters, a sparse update gradient value set is obtained.

[0046] Understandably, the parameters to be compressed refer to the layer parameters in the structure layer to be compressed. The gradient value to be compressed refers to the current model gradient value of the parameters to be compressed. The reference sparsity ratio refers to a manually set percentage constant, and the dynamic sparsity ratio refers to the reference sparsity ratio adjusted according to the gradient compression threshold. Since different structure layers to be compressed have different importance and compression risks, the traditional method of uniformly sparsifying all structure layers to be compressed with a fixed sparsity ratio will lead to insufficient compression of low-risk layers (gradient sparse layers with small gradient compression thresholds) or over-compression of high-risk layers (gradient sparse layers with large gradient compression thresholds). Therefore, this scheme introduces the dynamic sparsity ratio, which adaptively scales the reference sparsity ratio according to importance and compression risk (i.e., gradient compression threshold).

[0047] Furthermore, the above-mentioned dynamic sparsity ratio is calculated as follows: ,in, Indicates the dynamic sparsity ratio. Indicates the reference sparsity ratio. Indicates the gradient compression threshold. This represents the average compression threshold, which is the average of the gradient compression thresholds corresponding to all gradient sparse layers in the original neural network model. When the gradient compression threshold is less than the average compression threshold, it indicates that the compression risk of the current gradient sparse layer is low. Therefore, the gradient sparse layer can be compressed more aggressively (i.e., retain fewer parameters to be compressed), that is, the dynamic sparsity ratio can be increased. If the gradient compression threshold is not less than the average compression threshold, it indicates that the compression risk of the current gradient sparse layer is high. A more conservative compression strategy should be adopted to protect key gradient information. In this case, the dynamic sparsity ratio needs to be reduced.

[0048] It should be explained that the "retained parameters" refer to the parameters to be compressed that are not compressed (i.e., the corresponding gradient values ​​to be compressed are retained), and the "compressible parameters" refer to the parameters to be compressed that need to be compressed (i.e., the corresponding gradient values ​​to be compressed are recorded as 0). The step of identifying multiple retained parameters and multiple compressible parameters among multiple parameters to be compressed based on the dynamic sparsity ratio refers to: multiplying the dynamic sparsity ratio by the total number of parameters to be compressed to obtain the number of retained parameters; then determining the top x largest gradient values ​​to be compressed among the multiple gradient values ​​to be compressed, where x is the number of retained parameters; and recording the x parameters corresponding to these top x largest gradient values ​​as multiple retained parameters; removing these multiple retained parameters from the multiple parameters to be compressed; the remaining multiple parameters to be compressed are the multiple compressible parameters. The retained gradient value refers to the gradient value to be compressed corresponding to the retained parameter. The gradient value to be compressed corresponding to the above-mentioned compressible parameter is recorded as the compressible gradient value. The following operations can be performed on the compressible parameter and the compressible gradient value: the compressible gradient value is recorded as 0 and no subsequent data packaging is generated, or the compressible gradient value is accumulated into the residual buffer corresponding to the compressible parameter. When updating the gradient of the local original neural network model in the future, the compressible gradient value accumulated in the residual buffer can be re-injected into the calculation process of the new round of model update gradient value set. The detailed construction and use of the residual buffer here are existing technologies and will not be described in detail here.

[0049] Specifically, before determining whether a preset global model update instruction has been received, the method further includes: Let the set of distributed computing nodes be denoted as the set of distributed communication nodes. Perform the following operation on each distributed communication node in the set of distributed communication nodes: The model update is determined based on the distributed communication nodes, and the model update result is obtained, which is either completed or not completed. If the model update result is that the update is not completed, then return to the step of judging the local model update based on the distributed communication node until the model update result is that the update is completed. If the model update result indicates that the update is complete, then a master node communication instruction is generated and returned to the inference master node. If the master node communication instructions generated by each distributed communication node are returned to the inference master node, then a global model update instruction is generated based on the inference master node.

[0050] Understandably, "completed update" means that the distributed communication node has completed updating its local original neural network model and obtained at least one set of model update gradient values. "Incomplete update" means that the distributed communication node has not yet obtained the set of model update gradient values. The master node communication instruction refers to a signal sent by the distributed communication node to the inference master node after completing its local model update, containing its node identifier and ready status, to notify the inference master node that it is ready to participate in this round of global gradient aggregation. If the inference master node receives master node communication instructions generated by all distributed communication nodes, it indicates that in the current training iteration, all distributed communication nodes in the distributed communication node set have completed their local training tasks and are in a synchronous waiting state. At this time, the inference master node can generate a global model update instruction and send it to all distributed communication nodes.

[0051] S5. Aggregate gradient values ​​from multiple model update gradient value sets based on the model to be trained to obtain a local update gradient value set.

[0052] Understandably, the local update gradient value set refers to the new model update gradient value set obtained by aggregating the gradient values ​​of multiple model update gradient value sets.

[0053] Specifically, the step of aggregating gradient values ​​from multiple model update gradient value sets to obtain a local update gradient value set includes: Determine the set of parameters for the model to be updated based on the model to be trained; For each model parameter in the set of model parameters to be updated, perform the following operation: Identify multiple same-parameter update gradient values ​​corresponding to the model parameters to be updated in multiple model update gradient value sets, wherein the same-parameter update gradient values ​​correspond one-to-one with the model update gradient value set; The local update gradient value is obtained by summing multiple update gradient values ​​with the same parameters. Summarize the local update gradient values ​​corresponding to each model parameter to be updated to obtain the local update gradient value set.

[0054] It is clear that the set of model parameters to be updated refers to the set of all weight parameters in the model to be trained that need to be updated. The same-parameter update gradient value refers to the model update gradient value corresponding to the layer parameter with the same parameter category as the model parameter to be updated in a certain model update gradient value set. The local update gradient value refers to the sum of multiple same-parameter update gradient values.

[0055] S6. Pack the locally updated gradient value set to obtain the gradient data of the computing nodes. Summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes.

[0056] Understandably, the gradient data of the computing nodes refers to standardized data units that are formed after data packaging and are suitable for efficient network transmission. Packaging the locally updated gradient value set means converting the locally updated gradient value set into a compact, serialized binary or specific protocol format (such as a sparse matrix representation based on the COO coordinate format).

[0057] S7. Use the gradient dataset of the computing nodes to update the model to be trained in the inference master node to obtain the global update model, and complete the transmission data gradient sparsity based on the global update model.

[0058] It is clear that the globally updated model refers to the updated model to be trained. Updating the model to be trained in the inference master node using the gradient dataset of the computing nodes is a prior art technique. For example, the inference master node aggregates (e.g., averages) the gradient data uploaded by all distributed computing nodes to obtain multiple global gradient values. Each global gradient value corresponds to a parameter of the model to be updated in the model to be trained. Then, using these multiple global gradient values, an optimizer (e.g., stochastic gradient descent, SGD) is used to update the parameter set of the model to be updated in the model to be trained. The updated model to be trained is the globally updated model, which will not be elaborated further here.

[0059] To address the problems described in the background, this invention first performs gradient iteration calculations on the original neural network model to obtain multiple sets of model update gradient values. This step effectively solves the synchronization waiting problem caused by differences in node computing and communication capabilities in distributed training by introducing a global model update instruction mechanism. In traditional methods, fast nodes must passively wait for slow nodes after completing local computation, resulting in a large amount of idle computing resources. This solution allows nodes to continuously perform local gradient iterations before receiving instructions, converting waiting time into effective computation time, thereby significantly improving the overall computing resource utilization and training efficiency of the cluster. Furthermore, this step employs a dynamic adaptive gradient sparsity strategy instead of the traditional fixed-ratio compression. Traditional methods use a uniform sparsity rate for all network layers, ignoring the importance and gradient characteristics differences of different layers in the model, which can easily lead to the loss of key information or inadequate compression. This scheme dynamically calculates the compression threshold and sparsity ratio for each layer by comprehensively considering factors such as hierarchical information content and gradient importance. This achieves intelligent protection of key gradients while minimizing the amount of communication data, better balancing communication efficiency and model convergence accuracy. Furthermore, this scheme aggregates gradient values ​​from multiple model update gradient sets based on the model to be trained, obtaining a local update gradient set. This step aggregates the model update gradient sets obtained by the same distributed computing node in different iterations, enabling each distributed computing node to upload comprehensive gradient information representing multiple rounds of local iterative training in a single communication interaction with the inference master node. Compared to the traditional federated learning model, which requires communication with the server after each local iteration, this step significantly reduces the communication frequency between nodes and the master node, effectively reducing network bandwidth consumption and communication latency. Therefore, this invention can improve the training efficiency and resource utilization of the global model in federated learning, and enhance the communication efficiency between the master node and computing nodes.

[0060] like Figure 2 The diagram shown is a functional block diagram of a gradient sparse transmission data system based on distributed inference provided in an embodiment of the present invention.

[0061] The distributed inference-based gradient sparse transmission data system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the distributed inference-based gradient sparse transmission data system 100 may include a training model transmission module 101, a local model determination module 102, an update gradient calculation module 103, and a global model update module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device. The training model transmission module 101 is used to obtain the inference master node, identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes, determine the model to be trained according to the inference master node, and transmit the model to be trained to each distributed inference node in the distributed inference node set to obtain a distributed computing node set. The local model determination module 102 is used to perform the following operation on each distributed computing node in the distributed computing node set to determine the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers. The updated gradient calculation module 103 is used to perform gradient iterative calculation on the original neural network model to obtain multiple model update gradient value sets, and to aggregate the gradient values ​​of the multiple model update gradient value sets based on the model to be trained to obtain a local updated gradient value set. The global model update module 104 is used to package the local update gradient value set to obtain the computing node gradient data, summarize the computing node gradient data corresponding to each distributed computing node to obtain the computing node gradient dataset, and use the computing node gradient dataset to update the model to be trained in the inference master node to obtain the global update model.

[0062] In detail, the modules in the distributed inference-based gradient sparse transmission data system 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The technique used is the same as the gradient sparsity method for transmitted data based on distributed inference described in the previous section, and can produce the same technical effect, so it will not be repeated here.

[0063] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a gradient sparsity method for transmitted data based on distributed inference, according to an embodiment of the present invention.

[0064] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a program for a gradient sparsity method for transmitted data based on distributed inference.

[0065] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for a data gradient sparse method program based on distributed inference, but also to temporarily store data that has been output or will be output.

[0066] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., programs based on distributed inference-based gradient sparse data transmission methods) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0067] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0068] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0069] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0070] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0071] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0072] The program for the gradient sparsity method for transmitted data based on distributed inference, stored in the memory 11 of the electronic device 1, is a combination of multiple instructions. When run in the processor 10, it can achieve the following: Obtain the inference master node, and identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes; The model to be trained is determined based on the inference master node, and the model to be trained is transmitted to each distributed inference node in the distributed inference node set to obtain the distributed computing node set; For each distributed computing node in the distributed computing node set, the following operations are performed to identify the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; Gradient iteration calculation is performed on the original neural network model to obtain multiple sets of model update gradient values; Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set; Pack the locally updated gradient value set to obtain the gradient data of the computing nodes, and summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes. The training model in the inference master node is updated using the gradient dataset of the computing nodes to obtain the global update model. The gradient sparsity of the transmission data under distributed inference is then achieved based on the global update model.

[0073] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0074] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0075] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Obtain the inference master node, and identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes; The model to be trained is determined based on the inference master node, and the model to be trained is transmitted to each distributed inference node in the distributed inference node set to obtain the distributed computing node set; For each distributed computing node in the distributed computing node set, the following operations are performed to identify the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; Gradient iteration calculation is performed on the original neural network model to obtain multiple sets of model update gradient values; Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set; Pack the locally updated gradient value set to obtain the gradient data of the computing nodes, and summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes. The training model in the inference master node is updated using the gradient dataset of the computing nodes to obtain the global update model. The gradient sparsity of the transmission data under distributed inference is then achieved based on the global update model.

[0076] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0077] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0079] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A gradient sparsity method for transmitted data based on distributed inference, characterized in that, The method includes: Obtain the inference master node, and identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes; The model to be trained is determined based on the inference master node, and the model to be trained is transmitted to each distributed inference node in the distributed inference node set to obtain the distributed computing node set; For each distributed computing node in the distributed computing node set, the following operations are performed to identify the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; Gradient iteration calculation is performed on the original neural network model to obtain multiple sets of model update gradient values; Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set; Pack the locally updated gradient value set to obtain the gradient data of the computing nodes, and summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes. The training model in the inference master node is updated using the gradient dataset of the computing nodes to obtain the global update model. The gradient sparsity of the transmission data under distributed inference is then achieved based on the global update model.

2. The gradient sparsity method for transmitted data based on distributed inference as described in claim 1, characterized in that, The gradient iteration calculation of the original neural network model yields multiple sets of model update gradient values, including: The original neural network model is updated locally to obtain the model update gradient value set. The parameters of the original neural network model are updated using the model update gradient value set to obtain the updated neural network model. Determine whether a preset global model update command has been received; If no global model update instruction is received, the updated neural network model will be used as the original neural network model, and the step of performing local gradient update on the original neural network model will be returned until a global model update instruction is received. If a global model update instruction is received, the model update gradient value set is aggregated to obtain multiple model update gradient value sets.

3. The gradient sparsity method for transmitted data based on distributed inference as described in claim 2, characterized in that, The process of performing local gradient updates on the original neural network model to obtain a set of model update gradient values ​​includes: In the original neural network model, network structure layers are extracted sequentially from multiple network structure layers, and the extracted network structure layers are denoted as gradient sparse layers, where the gradient sparse layer includes multiple layer parameters. Compression discrimination is performed on the gradient sparse layer to obtain the compression discrimination result, where the compression discrimination result is either compressible or incompressible; If the compression determination result is compressible, then the gradient sparse layer is recorded as the structure layer to be compressed; Gradient sparsification is performed on the structure layer to be compressed to obtain a sparse update gradient value set; Summarize the sparse update gradient value set to obtain multiple sparse update gradient value sets, and generate the model update gradient value set based on the multiple sparse update gradient value sets.

4. The gradient sparsity method for transmitted data based on distributed inference as described in claim 3, characterized in that, The compression discrimination of the gradient sparse layer to obtain the compression discrimination result includes: Based on the original neural network model, update parameter statistics are performed to obtain the total number of parameters to be updated; Hierarchical information degree analysis is performed on the gradient sparse layer to obtain the hierarchical information degree; Gradient characteristic analysis is performed based on multiple layer parameters to obtain the gradient update importance factor; The gradient compression threshold is calculated based on the total number of parameters to be updated, the hierarchical information content, and the gradient update importance factor. The gradient compression threshold is expressed as: ， in, Indicates the gradient compression threshold. This indicates the total number of parameters to be updated. This represents the gradient update importance factor. Indicates the degree of hierarchical information. This indicates the preset adjustment coefficient; The number of currently updated parameters is calculated based on the parameters of multiple layers in the gradient sparse layer. If the number of updated parameters is greater than the gradient compression threshold, then the compressibility is recorded as the compression judgment result; If the number of updated parameters is not greater than the gradient compression threshold, then incompressibility is recorded as the compression result.

5. The gradient sparsity method for transmitted data based on distributed inference as described in claim 4, characterized in that, The hierarchical information degree analysis of the gradient sparse layer to obtain the hierarchical information degree includes: In the original neural network model, determine the layer index of the gradient sparse layer, query the output neurons of the gradient sparse layer, and obtain the number of output neurons. The hierarchical information degree is calculated based on the layer index, the number of output neurons, and the number of currently updated parameters. The hierarchical information degree is expressed as: ， in, Representation layer index, Indicates the number of output neurons. Indicates the number of parameters currently being updated. This represents a logarithmic function with the natural constant as its base.

6. The gradient sparsity method for transmitted data based on distributed inference as described in claim 5, characterized in that, The step of performing gradient characteristic analysis based on multiple layer parameters to obtain gradient update importance factors includes: Perform the following operation for each of the multiple layer parameters: Based on the layer parameters, the set of historical model parameters is determined in the training model of the inference master node. The set of historical model parameters includes multiple historical model parameters. The historical model gradient value set is queried based on the historical model parameter set. The historical model gradient value set includes multiple historical model gradient values, and each historical model gradient value corresponds one-to-one with a historical model parameter. Calculate relevant values ​​for model update based on historical model parameter set and historical model gradient value set; Summarize the model update-related values ​​corresponding to the parameters of each layer to obtain multiple model update-related values; Based on the original neural network model, obtain multiple current model gradient values ​​corresponding to multiple layer parameters, where each current model gradient value corresponds one-to-one with a layer parameter; Multiple layer parameter correlation values ​​are determined from multiple model update correlation values ​​based on multiple layer parameter correlation values, and gradient update importance factors are calculated based on multiple layer parameter correlation values ​​and multiple current model gradient values.

7. The gradient sparsity method for transmitted data based on distributed inference as described in claim 6, characterized in that, The gradient sparsification of the structure layer to be compressed, to obtain a sparse update gradient value set, includes: Determine multiple parameters to be compressed in the structure layer to be compressed, and obtain multiple gradient values ​​to be compressed corresponding to the multiple parameters to be compressed; The dynamic sparsity ratio is calculated based on the gradient compression threshold and the preset reference sparsity ratio. Based on the dynamic sparsity ratio, multiple retained parameters are identified among multiple parameters to be compressed, and multiple retained gradient values ​​of multiple retained parameters are determined among multiple gradient values ​​to be compressed. By merging multiple retained gradient values ​​and multiple retained parameters, a sparse update gradient value set is obtained.

8. The gradient sparsity method for transmitted data based on distributed inference as described in claim 7, characterized in that, Before determining whether a preset global model update instruction has been received, the method further includes: Let the set of distributed computing nodes be denoted as the set of distributed communication nodes. Perform the following operation on each distributed communication node in the set of distributed communication nodes: The model update is determined based on the distributed communication nodes, and the model update result is obtained, which is either completed or not completed. If the model update result is that the update is not completed, then return to the step of judging the local model update based on the distributed communication node until the model update result is that the update is completed. If the model update result indicates that the update is complete, then a master node communication instruction is generated and returned to the inference master node. If the master node communication instructions generated by each distributed communication node are returned to the inference master node, then a global model update instruction is generated based on the inference master node.

9. The gradient sparsity method for transmitted data based on distributed inference as described in claim 8, characterized in that, The step of aggregating gradient values ​​from multiple model update gradient value sets to obtain a local update gradient value set includes: Determine the set of parameters for the model to be updated based on the model to be trained; For each model parameter in the set of model parameters to be updated, perform the following operation: Identify multiple same-parameter update gradient values ​​corresponding to the model parameters to be updated in multiple model update gradient value sets, wherein the same-parameter update gradient values ​​correspond one-to-one with the model update gradient value set; The local update gradient value is obtained by summing multiple update gradient values ​​with the same parameters. Summarize the local update gradient values ​​corresponding to each model parameter to be updated to obtain the local update gradient value set.

10. A gradient-sparse transmission data system based on distributed inference, characterized in that, The system includes: The training model transmission module is used to obtain the inference master node, identify the distributed inference node set based on the inference master node, wherein the distributed inference node set includes multiple distributed inference nodes, determine the model to be trained according to the inference master node, and transmit the model to be trained to each distributed inference node in the distributed inference node set to obtain the distributed computing node set. The local model determination module is used to perform the following operations on each distributed computing node in the distributed computing node set to determine the original neural network model of the distributed computing node, wherein the original neural network model includes multiple network structure layers; The gradient calculation module is used to perform gradient iterative calculation on the original neural network model to obtain multiple model update gradient value sets. Based on the model to be trained, the gradient values ​​of multiple model update gradient value sets are aggregated to obtain the local update gradient value set. The global model update module is used to package the local update gradient value set to obtain the gradient data of the computing nodes, summarize the gradient data of the computing nodes corresponding to each distributed computing node to obtain the gradient dataset of the computing nodes, and use the gradient dataset of the computing nodes to update the model to be trained in the inference master node to obtain the global update model.

Images