Model processing method, system, and non-transitory storage medium

Abstract

The application discloses a model processing method and system and a nonvolatile storage medium. The method comprises the following steps: determining a pruning ratio corresponding to each edge node according to an available resource state corresponding to each edge node; performing pruning processing on a global model based on the pruning ratio corresponding to each edge node to obtain a local model corresponding to each edge node; sending the local model corresponding to each edge node to the corresponding edge node; receiving a local parameter training result of the local model corresponding to each edge node; and performing parameter updating on the global model based on the local parameter training result of the local model corresponding to each edge node to obtain a parameter updating result of the global model. The application solves the technical problem of low model training accuracy and poor applicability in the related art model training and processing method based on federated learning.

Description

Technical Field

[0001] This invention relates to the field of edge computing, and more specifically, to a model processing method, system, and non-volatile storage medium. Background Technology

[0002] Federated learning, as a distributed collaborative learning framework, trains a local model using the private data of multiple edge nodes with computing capabilities, and then aggregates the data to a specific edge server to update the global model. This local training method can fully utilize the distributed computing resources of multiple nodes, reduce the pressure on system communication bandwidth, and break the "data silo" dilemma while ensuring privacy and security.

[0003] Traditional federated learning faces several key challenges in practical implementation, such as efficient model training on resource-constrained devices, ensuring resource utilization without compromising training accuracy, reducing communication and transmission pressure between nodes, and addressing data and hardware heterogeneity. These problems often become more complex when applied to resource-constrained, heterogeneous edge computing scenarios. Furthermore, most federated learning methods aim to train a single global model, but due to a lack of generalization guarantees, they typically cannot guarantee high inference accuracy across multiple edge nodes with varying data distributions.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a model processing method, system, and non-volatile storage medium to at least solve the technical problems of low model training accuracy and poor applicability in related technologies based on federated learning model training and processing methods.

[0006] According to one aspect of the present invention, a model processing method is provided, comprising: determining pruning ratios for multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratios are used to indicate the proportion of pruning processing performed on a global model; performing the pruning processing on the global model based on the pruning ratios for each edge node to obtain local models for each edge node; sending the local models for each edge node to the corresponding edge nodes; receiving local parameter training results of the local models for each edge node sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training the corresponding local models based on local data; and updating the parameters of the global model based on the local parameter training results of the local models for each edge node to obtain parameter update results of the global model.

[0007] Optionally, the above-mentioned pruning process on the global model based on the pruning ratios corresponding to the multiple edge nodes to obtain the local models corresponding to the multiple edge nodes includes: determining a first arrangement order of the pruning ratios corresponding to the multiple edge nodes, wherein the first arrangement order is in ascending order; and pruning the global model according to the first arrangement order based on the pruning ratios corresponding to the multiple edge nodes to obtain the local models corresponding to the multiple edge nodes.

[0008] Optionally, the above-mentioned pruning process, based on the pruning ratios corresponding to the multiple edge nodes and in the first arrangement order, is used to prune the global model to obtain the local models corresponding to the multiple edge nodes. This includes obtaining the local models corresponding to the multiple edge nodes in the following manner, in the first arrangement order: using the global model as the first initial model, taking the local model corresponding to the first edge node among two adjacent edge nodes as the initial model for the pruning process of the second edge node, and performing the pruning process on the initial model to obtain the local model corresponding to the second edge node.

[0009] Optionally, the above-mentioned parameter update of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model includes: obtaining a second sorting order of the model sizes of the local models corresponding to the multiple edge nodes, wherein the second sorting order is an ascending sorting order, and the model sizes of the local models corresponding to the multiple edge nodes are inversely proportional to the corresponding pruning ratios; obtaining a first parameter update result of the local models corresponding to the multiple edge nodes according to the second sorting order based on the local parameter training results of the local models corresponding to the multiple edge nodes; and summing the first parameter update results of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model.

[0010] Optionally, the above-mentioned method of obtaining the first parameter update result of the local model corresponding to each of the multiple edge nodes according to the second arrangement order, based on the local parameter training results of the local model corresponding to each of the multiple edge nodes, includes: obtaining the first parameter update result of the local model corresponding to each of the multiple edge nodes according to the second arrangement order in the following manner: taking any model as the base model, determining the weight values ​​of the local models corresponding to each of the multiple edge nodes; determining the first parameter training result in the local parameter training results of the local models corresponding to each of the multiple edge nodes that corresponds to the predetermined model parameters included in any model; and determining the first parameter update result of any model based on the weight values ​​of the local models corresponding to each of the multiple edge nodes and the first parameter training results of the local models corresponding to each of the multiple edge nodes.

[0011] Optionally, determining the weight values ​​of the local models corresponding to the plurality of edge nodes includes: using any one of the models as the baseline model, and based on the training results of the local parameters of the local models corresponding to the plurality of edge nodes, determining the gradient direction distance between the local models corresponding to the plurality of edge nodes and any one of the models; and determining the weight values ​​of the local models corresponding to the plurality of edge nodes based on the gradient direction distance between the local models corresponding to the plurality of edge nodes and any one of the models.

[0012] Optionally, determining the weight values ​​of the local models corresponding to the multiple edge nodes based on the gradient direction distances between the local models corresponding to the multiple edge nodes and any one of the models includes: determining the sorting number of any one of the models in the second arrangement order, and the sorting number of the local models corresponding to the multiple edge nodes in the second arrangement order; determining the weight values ​​of the local models corresponding to the multiple edge nodes based on the size relationship between the sorting numbers of the local models corresponding to the multiple edge nodes in the second arrangement order and the sorting numbers of any one of the models in the second arrangement order, and the gradient direction distances between the local models corresponding to the multiple edge nodes and any one of the models.

[0013] Optionally, determining the weight values ​​of the local models corresponding to the multiple edge nodes based on the relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one of the models, and the gradient direction distance between the local models corresponding to the multiple edge nodes and any one of the models, includes: determining, based on the relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one of the models, a first model whose sorting number is less than the sorting number of any one of the models, and a second model whose sorting number is greater than or equal to the sorting number of any one of the models; determining that the weight value of the first model is zero; and determining the weight value of the second model based on the gradient direction distance between the second model and any one of the models.

[0014] Optionally, the above-mentioned parameter update of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model includes: determining whether a preset termination condition has been met, wherein the preset termination condition is at least one of the following: the accuracy of the global model reaches a preset expected value, the number of training iterations reaches a preset iteration threshold, and the loss value of the global model is less than or equal to a preset loss threshold; if the preset termination condition is met, the parameters of the global model are updated based on the local parameter training results obtained from the last training of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model.

[0015] According to one aspect of the present invention, another model processing method is also provided, comprising: an edge server determining a pruning ratio for each of the multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratio is used to indicate the proportion of pruning processing performed on the global model; the edge server pruning the global model based on the pruning ratio for each of the multiple edge nodes to obtain local models for each of the multiple edge nodes; sending the local models for each of the multiple edge nodes to the corresponding edge nodes; the multiple edge nodes training their respective local models based on local data to obtain local parameter training results for the local models for each of the multiple edge nodes, and sending the local parameter training results for the local models for each of the multiple edge nodes to the edge server; and the edge server updating the parameters of the global model based on the local parameter training results for the local models for each of the multiple edge nodes to obtain parameter update results for the global model.

[0016] According to another aspect of the present invention, a model processing system is also provided, comprising: an edge server, configured to determine pruning ratios for multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratios are used to indicate the proportion of pruning processing performed on a global model; pruning the global model based on the pruning ratios for each edge node to obtain local models for each edge node; sending the local models for each edge node to their respective edge nodes; the multiple edge nodes being connected to the edge server, configured to train their respective local models based on local data to obtain local parameter training results for the local models for each edge node, and sending the local parameter training results for the local models for each edge node to the edge server; the edge server further configured to update the parameters of the global model based on the local parameter training results for the local models for each edge node to obtain parameter update results for the global model.

[0017] According to another aspect of the present invention, a non-volatile storage medium is also provided, wherein the non-volatile storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor and executed any one of the above-described model processing methods.

[0018] In this embodiment of the invention, the pruning ratios of multiple edge nodes are determined based on their respective available resource states. These pruning ratios indicate the proportion of pruning performed on the global model. Based on these pruning ratios, the global model is pruned to obtain local models corresponding to each edge node. These local models are then sent to their respective edge nodes. The local parameter training results of the local models sent by the edge nodes are received, where the local parameter training results are obtained by training the corresponding local models using local data. Based on the local parameter training results of the local models, the global model's parameters are updated to obtain the updated parameters. This approach effectively incorporates the heterogeneity of different edge nodes and the dynamic changes in runtime resources for efficient model training. It preserves the personalized training results of different edge nodes while maintaining model training accuracy, thereby improving model training accuracy and applicability. This solves the technical problems of low model training accuracy and poor applicability in federated learning-based model training and processing methods in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a schematic diagram of a model processing method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of an optional model pruning according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of an optional global model construction process according to an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the framework of an optional model processing method according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of another model processing method according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of a model processing system according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] With the development of cloud computing, artificial intelligence, the Internet of Things, and edge computing technologies, edge intelligence, by integrating edge computing functions and moving intelligent computing from cloud computing centers to the edge, can meet key needs such as agile connectivity, real-time business, intelligent applications, and security and privacy protection, and has broad application prospects. However, the traditional centralized machine learning paradigm, which aggregates data transmissions from distributed edge nodes to a central server to train machine learning models, brings problems to edge intelligence systems such as increased transmission bandwidth load, a surge in computing pressure on central nodes, and a lack of data security and privacy guarantees.

[0029] Federated learning, as a distributed collaborative learning framework, trains a local model using the private data of multiple edge nodes with computing capabilities, and then aggregates the data to a specific edge server to update the global model. This local training method can fully utilize the distributed computing resources of multiple nodes, reduce system communication bandwidth pressure, and break the "data silo" dilemma while ensuring privacy and security. Traditional federated learning faces several key challenges in practical implementation, such as efficient model training on resource-limited devices, ensuring resource utilization without affecting training accuracy, reducing communication transmission pressure between nodes, and addressing data and hardware heterogeneity. These problems often become more complex when applied to resource-limited heterogeneous edge computing scenarios. Furthermore, most federated learning aims to train a single global model, but due to the lack of generalization guarantees, it usually cannot guarantee high inference accuracy across multiple edge nodes with different data distributions.

[0030] Among related technologies, the HeteroFL framework breaks the traditional federated learning limitation that requires different nodes to use the same model for training and aggregation. It distributes heterogeneous local models to different nodes, then averages and aggregates the different local training results to update the global model. It also proposes Static Batch Normalization and Scalar modules to improve the robustness of model training on Non-IID data. Related experimental results verify that this framework can significantly improve computational and communication efficiency in federated learning. However, this approach distributes local models matching the resources of different nodes for training, and updates the global model by aggregating heterogeneous models. Although the local models on different nodes belong to the same model class, their structures and parameters are independent, making it impossible for them to mutually promote each other to improve training effectiveness and efficiency. Furthermore, the final global model aggregation strategy also has room for improvement. In addition, this approach aims to train a single global model and does not support personalized model training.

[0031] In related technologies, Hermes uses model pruning techniques to adapt heterogeneous local models to different nodes. During federated training, only local models with sparse structures are trained and transmitted, significantly reducing computational and communication pressure during training. Furthermore, during model aggregation, only the overlapping parameters of different local models are averaged to preserve the individual training results of each local model. This approach considers the resource capabilities of different nodes, using model pruning to adapt local models that match their resources. However, it does not consider that the available resources of a node are dynamically changing, thus failing to guarantee model training performance in dynamic systems. Moreover, the local models of different edge nodes are independent, making it impossible to guarantee sufficient overlap to ensure convergence speed and stability during the aggregation phase. Additionally, Hermes uses a set of global model parameters to store the individual learning results of different nodes, lacking generalization ability.

[0032] Based on the above problems, this invention provides a method embodiment for model processing. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] Figure 1 This is a flowchart of a model processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0034] Step S102: Determine the pruning ratio for each edge node based on the available resource status of each edge node.

[0035] Optionally, the pruning ratio mentioned above indicates the proportion of pruning performed on the global model, with multiple edge nodes corresponding to different pruning ratios. These multiple edge nodes correspond to multiple edge devices, which may include, but are not limited to, mobile terminal devices (such as smartphones, tablets, computers, etc.).

[0036] By taking into account the dynamic changes in the available resource status of multiple edge nodes during runtime, the model pruning ratio is adaptively adjusted, and structured model pruning is used to generate a local model for each edge node that matches its available resource status, thereby improving model applicability, overall resource utilization, and robustness.

[0037] Step S104: Based on the pruning ratios corresponding to the multiple edge nodes, perform the pruning process on the global model to obtain the local models corresponding to the multiple edge nodes.

[0038] It is understandable that the pruning ratios corresponding to the multiple edge nodes are determined based on the available resource status of the multiple edge nodes. Therefore, the local model obtained based on the pruning ratios corresponding to the multiple edge nodes fully considers the available resource status of the multiple edge nodes and is a local model that is adapted to the available resource status of each edge node.

[0039] In one optional embodiment, the above-mentioned pruning process on the global model based on the pruning ratios corresponding to the plurality of edge nodes to obtain the local models corresponding to the plurality of edge nodes includes: determining a first arrangement order of the pruning ratios corresponding to the plurality of edge nodes, wherein the first arrangement order is in ascending order; and pruning the global model according to the first arrangement order based on the pruning ratios corresponding to the plurality of edge nodes to obtain the local models corresponding to the plurality of edge nodes.

[0040] By employing the above method, a matching pruning ratio is selected for each edge node based on its available resource status, and this ratio can be dynamically adjusted as the system runs. The pruning ratios are sorted by size, and the global model is used as the initial model. Structured pruning is then performed progressively from smallest to largest, resulting in local models corresponding to multiple edge nodes. In essence, by progressively pruning the global model based on the ascending order of the pruning ratios for each edge node (i.e., the first order), nested local models are obtained for each edge node, thereby improving model pruning efficiency.

[0041] In an optional embodiment, the above-mentioned pruning process of the global model based on the pruning ratios corresponding to the plurality of edge nodes and in the first arrangement order to obtain the local models corresponding to the plurality of edge nodes includes: taking the global model as the first initial model, and in the first arrangement order, obtaining the local models corresponding to the plurality of edge nodes in the following manner: taking the local model corresponding to the first edge node in two adjacent edge nodes as the initial model for the pruning process of the second edge node, and performing the pruning process on the initial model to obtain the local model corresponding to the second edge node.

[0042] Using the above method, for a given series of pruning ratios, the global model is used as the first initial model, and structured pruning is performed progressively from small to large pruning ratios. For each model pruning (except for the first pruning operation), the model after the previous pruning is used as the initial model for the current pruning operation. Pruning is then performed based on this initial model. As the pruning ratio increases, the structure of the corresponding pruned model becomes sparser, until a series of nested derived models are generated, serving as the local models corresponding to multiple edge nodes. Figure 2 This is a schematic diagram of an optional model pruning according to an embodiment of the present invention, such as... Figure 2 As shown, a set of local models arranged in ascending order of pruning ratio is: local model N, ..., local model 1 (corresponding to edge node 1, ..., local model N respectively), with corresponding pruning ratios of α. n , ..., α1, the local models N to 1 are pruned sequentially in ascending order as described above. The model after the previous pruning is used as the initial model for the current pruning operation. Pruning is then performed on the basis of this initial model, resulting in a series of nested local models.

[0043] Optionally, during the pruning process, the importance of filter parameters, feature channel numbers, or neurons in different layers of the corresponding local model can be ranked according to the weight values ​​of each edge node, and then the parts of each layer of the model that are below a certain threshold are deleted, thereby generating a sparse derived model; different derived models are sent as local models to specific edge nodes.

[0044] Step S106: Send the local models corresponding to the above multiple edge nodes to the corresponding edge nodes.

[0045] Using the above method, after pruning the global model according to the pruning comparison and obtaining the local models corresponding to multiple edge nodes, the local models corresponding to multiple edge nodes are sent to the corresponding edge nodes. Each edge node uses local data to perform personalized model training on the received local model based on the received local model and training parameters.

[0046] Step S108: Receive the local parameter training results of the corresponding local models sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training the corresponding local models based on local data.

[0047] It can be understood that the local parameter training results of the corresponding local models sent by the multiple edge nodes are the local training parameters obtained by the multiple edge nodes based on their local data. In other words, the local parameter training results integrate the differential features of each edge node.

[0048] Step S110: Based on the local parameter training results of the local models corresponding to the multiple edge nodes, the parameters of the global model are updated to obtain the parameter update results of the global model.

[0049] In this way, the edge server receives the local parameter training results (i.e., local parameter training results) returned by different edge nodes. Based on the local parameter training results of the local models corresponding to multiple edge nodes, the global model is updated with parameters. This allows the updated global model parameters (i.e., parameter update results) to integrate the personalized features of multiple edge nodes. On the one hand, it can utilize the complementarity of computing resources and the sharing of data of different edge nodes for efficient model training. On the other hand, it can maximize the preservation of the local training results of different edge nodes through the parameter-sharing global model, enabling personalized learning without introducing additional parameters.

[0050] In one optional embodiment, the above-mentioned parameter update of the global model based on the local parameter training results of the local models corresponding to the plurality of edge nodes to obtain the parameter update result of the global model includes: obtaining a second sorting order of the model sizes of the local models corresponding to the plurality of edge nodes, wherein the second sorting order is an ascending sorting order, and the model sizes of the local models corresponding to the plurality of edge nodes are inversely proportional to the corresponding pruning ratios; obtaining a first parameter update result of the local models corresponding to the plurality of edge nodes according to the second sorting order based on the local parameter training results of the local models corresponding to the plurality of edge nodes; and summing the first parameter update results of the local models corresponding to the plurality of edge nodes to obtain the parameter update result of the global model.

[0051] It can be understood that the local parameter training results of the local models corresponding to multiple edge nodes are the parameter update results of the local models corresponding to multiple edge nodes in each edge node, and the first parameter update results of the local models corresponding to multiple edge nodes are the parameter update results of the local models corresponding to multiple edge nodes in the edge server. The first parameter update results are obtained by integrating the local parameter training results of the local models corresponding to multiple edge nodes.

[0052] It should be noted that since the local models corresponding to the multiple edge nodes are obtained by pruning the global model, the models are of different sizes and are nested. Smaller local models share model structure and parameters with larger local models. Therefore, by integrating the identical model parameters from the local parameter training results (i.e., local parameter training results) corresponding to the multiple edge nodes, starting with the smallest local model, and following the ascending order, the model parameters of the corresponding regions in the global model are gradually updated until all model parameters in the global model are updated, thus obtaining the new parameter update results (i.e., the first parameter update results) for the local models corresponding to the multiple edge nodes. As before... Figure 2 As shown, in Figure 2 In the model, the local model corresponding to edge node 1 is the smallest. The other local models (i.e., edge nodes 2 to N) all include the model parameters of edge node 1. Therefore, when updating the parameters of the local model 1 corresponding to edge node 1, the training results of the local parameters in other local models that correspond to the model parameters in local model 1 need to be taken into account. Finally, the parameter update result of local model 1 in the edge server (i.e., the first parameter update result) is obtained by integrating them.

[0053] Optionally, after obtaining the parameter update results (i.e., the first parameter update results) of the local models corresponding to multiple edge nodes in the edge server, the first parameter update results of the local models corresponding to multiple edge nodes are summed to finally obtain the parameter update results (i.e., the parameter update results) of the global model. Where W(t) represents the parameter update result corresponding to the global model, W k '(t) represents the update result of the first parameter of any one of the local models corresponding to multiple edge nodes, and N is the total number of multiple edge nodes.

[0054] In an optional embodiment, based on the local parameter training results of the local models corresponding to the plurality of edge nodes respectively, and in accordance with the second arrangement order, the first parameter update results of the local models corresponding to the plurality of edge nodes respectively are obtained. Based on the local parameter training results of the local models corresponding to the plurality of edge nodes respectively, and in accordance with the second arrangement order, the first parameter update results of the local models corresponding to the plurality of edge nodes are obtained in the following manner: taking any one model as the base model, determining the weight values ​​of the local models corresponding to the plurality of edge nodes respectively; determining the first parameter training results in the local parameter training results of the local models corresponding to the plurality of edge nodes that correspond to the predetermined model parameters included in the arbitrary model; and determining the first parameter update result of the arbitrary model based on the weight values ​​of the local models corresponding to the plurality of edge nodes and the first parameter training results of the local models corresponding to the plurality of edge nodes respectively.

[0055] Optionally, the aforementioned predetermined model parameters are the model parameters included in any model after sorting the local models corresponding to multiple edge nodes according to their model size from smallest to largest, excluding the model parameters included in the local model preceding that model. Figure 3 This is a schematic diagram of an optional global model construction process according to an embodiment of the present invention, such as... Figure 3 As shown, the edge server corresponds to multiple edge nodes (edge ​​node 1, ..., edge node N), and the corresponding local models are arranged in ascending order as local model 1, ..., local model N. Since local model 1 is the smallest local model, the model parameters included in local model 1 are the predetermined model parameters of this model; the predetermined model parameters corresponding to local model 2 are the other model parameters included in local model 2 besides the parameters included in local model 1.

[0056] Optionally, after sorting the local models corresponding to the multiple edge nodes in ascending order according to their model sizes to obtain a second sorting result, the model parameters of the local models corresponding to the multiple edge nodes are updated according to the second sorting result to obtain the first parameter update result of the local models corresponding to the multiple edge nodes. As before... Figure 3As shown, firstly, the predetermined model parameters included in local model 1 are updated. This is done by performing a weighted fusion calculation based on the first parameter training results corresponding to the model parameters included in local model 1 from the local parameter training results, and the weight values ​​of the local model corresponding to each edge node when local model 1 is used as the baseline model, to obtain the updated first parameter result (i.e., P1) corresponding to the predetermined model parameters included in local model 1. Next, the predetermined model parameters included in local model 2 are updated. This is done by performing a weighted fusion calculation based on the first parameter training results corresponding to the predetermined model parameters included in local model 2 from the local parameter training results, and the weight values ​​of the local model corresponding to each edge node when local model 2 is used as the baseline model, to obtain the updated first parameter result (i.e., P2) corresponding to the predetermined model parameters included in local model 2.

[0057] By using the above method, when updating the first parameter of the local model corresponding to the multiple edge nodes, any one of the local models corresponding to the multiple edge nodes is used as the benchmark model. The first parameter training results corresponding to the same model parameters as the benchmark model (i.e., any one of the above models) are selected from the local parameter training results of the local models corresponding to the multiple edge nodes. Then, the weight values ​​of the local models corresponding to the multiple edge nodes are determined. In this way, the same model parameters in the local parameter training results corresponding to the multiple edge nodes are weighted and integrated, so that the obtained first parameter update result integrates the differential features of multiple edge nodes and has stronger applicability.

[0058] Optionally, the weight values ​​for other models are determined based on the gradient direction distances between the local models corresponding to each of the multiple edge nodes and any one of the models. The weight values ​​of the local models corresponding to the multiple edge nodes are used to indicate the contribution of each of the multiple edge nodes to the parameter update of the baseline model (i.e., any one of the aforementioned models).

[0059] Optionally, the first parameter update result of any of the above models can be obtained by weighted summation based on the weight values ​​of the local models corresponding to each of the multiple edge nodes and the fourth parameters of the local models corresponding to each of the multiple edge nodes. That is, based on the weight values ​​of the local models corresponding to each of the multiple edge nodes and the training results of the first parameters of the local models corresponding to each of the multiple edge nodes, the first parameter update result of any of the above models can be determined in the following way:

[0060]

[0061] Among them, W k '(t) represents the update result of the first parameter of any one of the local models corresponding to multiple edge nodes; This represents the weight value of the local model corresponding to one of the multiple edge nodes, where any model k corresponding to any edge node is the baseline model. This represents the training result of the first parameter of the local model corresponding to one of the multiple edge nodes, where any model k corresponding to any edge node is the baseline model.

[0062] In one optional embodiment, determining the weight values ​​of the local models corresponding to the plurality of edge nodes includes: using any one of the models as the baseline model, determining the gradient direction distance between the local models corresponding to the plurality of edge nodes and any one of the models based on the local parameter training results of the local models corresponding to the plurality of edge nodes; and determining the weight values ​​of the local models corresponding to the plurality of edge nodes based on the gradient direction distance between the local models corresponding to the plurality of edge nodes and any one of the models.

[0063] Optionally, in each aggregation step, using any local model corresponding to any edge node as the baseline model, calculate the gradient direction distance between the local models corresponding to each of the multiple edge nodes and the arbitrary local model. Based on the local parameter training results corresponding to the arbitrary local model and the local parameter training results of the local models corresponding to the multiple edge nodes, determine the gradient direction distance between the local models corresponding to the multiple edge nodes and the arbitrary model as follows:

[0064]

[0065] Where, θ kj (t) represents the gradient direction distance between the local model corresponding to one of the above edge nodes j and any of the above models; W k W represents the local parameter training result of any local model corresponding to any edge node k. j This represents the local parameter training result of the local model corresponding to one of multiple edge nodes, node j. Let θ represent the gradient function, ||·|| represent the L2 norm, and <·> represent the inner product calculation. A smaller angle θ indicates that the gradient directions of the local models of the two edge nodes are more similar, meaning they have a higher correlation.

[0066] In an optional embodiment, determining the weight values ​​of the local models corresponding to the plurality of edge nodes based on the gradient direction distances between the local models corresponding to the plurality of edge nodes and any one of the models includes: determining the sorting number of any one of the models in the second arrangement order, and the sorting number of the local models corresponding to the plurality of edge nodes in the second arrangement order; determining the weight values ​​of the local models corresponding to the plurality of edge nodes based on the size relationship between the sorting numbers of the local models corresponding to the plurality of edge nodes in the second arrangement order and the sorting numbers of any one of the models in the second arrangement order, and the gradient direction distances between the local models corresponding to the plurality of edge nodes and any one of the models.

[0067] Optionally, when determining the weight values ​​of the local models corresponding to multiple edge nodes, it is necessary to consider the model size relationship between the local models corresponding to each edge node and the baseline model (i.e., any one of the aforementioned models). By sorting the local models corresponding to the multiple edge nodes in ascending order of model size, as described above, and obtaining the second sorting order, the ranking of the local models and the baseline model within this second sorting order reflects the model size relationship between them. Combining the gradient direction distances between the local models corresponding to each edge node and any one of the aforementioned models, and considering the model size relationship, the weight values ​​of the local models corresponding to the multiple edge nodes are determined.

[0068] In an optional embodiment, determining the weight values ​​of the local models corresponding to the multiple edge nodes based on the size relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one of the models, and the gradient direction distance between the local models corresponding to the multiple edge nodes and any one of the models, includes: determining, based on the size relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one of the models, a first model whose sorting number is less than the sorting number of any one of the models, and a second model whose sorting number is greater than or equal to the sorting number of any one of the models; determining that the weight value of the first model is zero; and determining the weight value of the second model based on the gradient direction distance between the second model and any one of the models.

[0069] Optionally, for a local model whose size is larger than any of the aforementioned models, it indicates that the local model whose size is larger than any of the aforementioned models includes all the model parameters of the aforementioned models. Therefore, when updating the parameters of any of the aforementioned models, the local model whose size is larger than any of the aforementioned models can be taken into account, and the weight value between the local model whose size is larger than any of the aforementioned models and the aforementioned models can be determined based on the gradient direction distance between the local model whose size is larger than any of the aforementioned models. For a local model whose size is smaller than any of the aforementioned models, it indicates that the local model whose size is larger than any of the aforementioned models does not include all the model parameters of the aforementioned models. Therefore, when updating the parameters of any of the aforementioned models, it is not necessary to take the local model whose size is smaller than any of the aforementioned models into account, and the weight value of the local model whose size is smaller than any of the aforementioned models is 0.

[0070] Optionally, based on the relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one of the models, and the gradient direction distance between the local models corresponding to the multiple edge nodes and any one of the models, the initial weights of the local models corresponding to the multiple edge nodes are determined as follows:

[0071]

[0072] Where e and α are preset constant coefficients, and θ kj (t) represents the gradient direction distance between the local model corresponding to node j among the multiple edge nodes and any of the multiple edge nodes; k corresponds to the sorting number of any model; j corresponds to the sorting number of the local model corresponding to node j among the multiple edge nodes.

[0073] Optionally, after obtaining the initial weights of the local models corresponding to the multiple edge nodes, the initial weights of the local models corresponding to the multiple edge nodes are normalized to obtain the weight values ​​of the local models corresponding to the multiple edge nodes. The normalization formula is as follows.

[0074]

[0075] In one optional embodiment, the above-mentioned updating of the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model includes: determining whether a preset termination condition has been reached; if the preset termination condition has been reached, then updating the parameters of the global model based on the local parameter training results obtained from the last training of the local models corresponding to the multiple edge nodes to obtain the parameter update result of the global model.

[0076] Optionally, the above-mentioned preset termination condition is at least one of the following: the accuracy of the above-mentioned global model reaches the preset expected value, the number of training iterations reaches the preset iteration threshold, and the loss value of the above-mentioned global model is less than or equal to the preset loss threshold.

[0077] Understandably, before updating the parameters of the global model, it is necessary to determine the model training status, i.e., whether the preset termination condition has been met (such as the global model's accuracy reaching the preset expected value, the number of training iterations reaching the preset iteration threshold, or the global model's loss value being less than or equal to the preset loss threshold, etc.). If the preset termination condition is met, the model training process is stopped, and the global model parameters are updated based on the local parameter training results of each local model obtained from the last training iteration, resulting in the updated global model parameters; otherwise, the model training operation continues until the preset termination condition is met.

[0078] The execution entity of the above steps S102 to S110 is the edge server. Through the above steps S102 to S110, the heterogeneity of different edge nodes and the dynamic changes of runtime resources can be taken into consideration to carry out efficient model training. While ensuring the model training accuracy, the personalized training results of different edge nodes can be preserved, thereby achieving the technical effect of improving the model training accuracy and model applicability. This solves the technical problems of low model training accuracy and poor applicability in federated learning-based model training and processing methods in related technologies.

[0079] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation method, which specifically includes:

[0080] In step S1, the user reaches an agreement on the goals, models, edge servers, and edge nodes participating in training for federated learning.

[0081] In step S2, the edge server receives the available resource status information of multiple edge nodes sent by multiple edge nodes, selects the corresponding pruning ratio accordingly, and then performs progressive model pruning on the global model according to the pruning ratio to generate a series of nested derived models as the local models of the corresponding edge nodes.

[0082] Step S3: The edge server sends the local models corresponding to multiple edge nodes to the corresponding edge nodes.

[0083] In step S4, each edge node uses local data to perform personalized training on the corresponding local model in parallel, based on the received local model and model training parameters (such as the number of local training iterations, training batch size, learning rate, etc.).

[0084] In step S5, each edge node sends the local parameter training results to the edge server.

[0085] Step S6: The edge server calculates the gradient direction distance between the local model parameters corresponding to different edge nodes based on the received local parameter training results of different edge nodes.

[0086] Step S7: Update the model parameters of the global model in a regional, progressive manner. During each aggregation step, using the local model of any edge node as the baseline model (starting from the least derived model), calculate the contribution (i.e., weight value) of different edge nodes in this round of update aggregation based on the gradient direction distance between the baseline model and the local models corresponding to multiple edge nodes. If other edge node models are smaller than the baseline model of the current step (without overlap), their corresponding contribution is zero. Based on the contribution values ​​of multiple edge nodes and the first parameter training result corresponding to the predetermined model parameters included in the baseline model from the local parameter training results of the local models corresponding to multiple edge nodes, perform weighted aggregation of the baseline model's parameters and update the parameters of the corresponding region in the global model. The predetermined model parameters are the model parameters included in the baseline model after sorting the local models corresponding to multiple edge nodes from smallest to largest model size, excluding the model parameters included in the local model preceding the baseline model.

[0087] Step S8: Repeat step S7 multiple times until all model parameters in the global model have been updated.

[0088] Step S9: Determine whether the training termination condition has been triggered, such as whether the global model accuracy has reached the preset expected value, the number of training iterations has reached the threshold, or the global model loss is less than or equal to the threshold. If yes, terminate the training and output the final parameter update result of the global model. If no, repeat steps S2 to S8.

[0089] Step S10: Return the parameter update results of the global model.

[0090] It is understood that the embodiments of the present invention are a model training method based on federated learning. This method coordinates multiple edge nodes with limited resources, heterogeneous hardware, dynamic system changes, and uneven data distribution to perform efficient model training. It takes into account the heterogeneity of different edge nodes and the dynamic changes of runtime resources, aiming to reduce the computation and transmission pressure during the edge federated learning process, improve the robustness of the system, enable personalized model training, and preserve the personalized training results of different edge nodes while ensuring the accuracy of model training. Specifically, it includes: (1) By taking into account the heterogeneity of different edge nodes and the dynamic changes of runtime resources, using structured model pruning to generate a local model for each edge node that matches its available resource status, and using a structure and parameter sharing mechanism to enable different local models to be combined into a compact multi-capacity model, and finally updating the global model according to the parameter output results of the trained local model (i.e., the first model parameters). (2) During system operation, a new model pruning scheme is used to prune the global model into a multi-capacity model containing a set of derived models, where different derived models have different complexities. The obtained derived models are sent as local models to specific edge nodes for local training. Unlike traditional model variants that are independent of each other, smaller derived models share model structure and parameters with larger derived models, embedding themselves into the larger derived models. Ultimately, all derived models are nested in a single global model, which can save the learning results of multiple models without introducing additional parameters. (3) A flexible pruning ratio is used instead of a fixed pruning ratio, so that the optimal pruning ratio is dynamically selected for each edge node to meet the model training requirements for available resources. (4) In the model aggregation phase, the training results of different local models are aggregated through a progressive parameter aggregation method, starting from the smallest derived model until the global model is updated. By adopting the above method, on the one hand, the complementarity of computing resources and the sharing of data of different edge nodes can be used to carry out efficient model training; on the other hand, the local training results of different edge nodes can be preserved to the greatest extent through the global model with shared parameters, enabling personalized learning without introducing additional parameters.

[0091] Based on the above embodiments and optional embodiments, the present invention proposes another optional implementation method. Figure 4 A schematic diagram of the framework of an optional model processing method according to an embodiment of the present invention is shown below. Figure 4As shown, this method mainly includes three key steps: progressive model pruning, personalized local training, and weighted model aggregation. Specifically, progressive model pruning involves: selecting a matching pruning ratio based on the available resources of each edge node, dynamically adjusting this ratio as the system runs; sorting the pruning ratios by size, and using the global model as the initial model for progressive, structured pruning from smallest to largest. For a given series of pruning ratios, the global model is used as the initial model, and structured pruning is performed progressively from smallest to largest. For each model pruning operation (except the first pruning operation), the model after the previous pruning is used as the initial model for the current pruning operation, and pruning is performed on top of it. As the pruning ratio increases, the structure of the pruned model becomes sparser, until a series of nested derived models are generated as local models corresponding to multiple edge nodes. During the pruning process, the importance of filter parameters, feature channels, or neurons in different layers of the model is ranked according to the weights. Then, the parts of each layer of the model that are below a certain threshold are deleted, thereby generating a sparse derived model. Different derived models are sent as local models to specific edge nodes.

[0092] The aforementioned personalized local training specifically includes: each edge node using local data to perform personalized model training on the received local model based on the received local model and training parameters.

[0093] The aforementioned weighted model aggregation specifically includes: the edge server receiving local parameter update results (i.e., the first model parameters) returned by different edge nodes; performing step-by-step aggregation based on the degree of overlap in the different parameter update results; starting from the part with the most overlap (the local model with the smallest model size), gradually updating the predetermined model parameters of the corresponding region in the global model until all parameters are updated. In each aggregation step, using the local model of any one of the multiple edge nodes (i.e., any local model) as the baseline model, the gradient direction distance between the local models of the multiple edge nodes and the arbitrary local model is calculated, and the contribution of different edge nodes in this round of update aggregation is determined accordingly, thus achieving weighted model aggregation.

[0094] According to an embodiment of the present invention, another model processing method is also provided. Figure 5 This is a flowchart of another model processing method according to an embodiment of the present invention, such as... Figure 5 As shown, the method includes:

[0095] Step S502: The edge server determines the pruning ratio corresponding to each edge node based on the available resource status of each edge node. The pruning ratio is used to indicate the proportion of pruning processing performed on the global model.

[0096] Step S504: The edge server performs pruning on the global model based on the pruning ratios corresponding to the multiple edge nodes to obtain local models corresponding to the multiple edge nodes; and sends the local models corresponding to the multiple edge nodes to the corresponding edge nodes.

[0097] Step S506: The aforementioned multiple edge nodes train their corresponding local models based on local data to obtain the local parameter training results of the local models corresponding to the aforementioned multiple edge nodes, and send the local parameter training results of the local models corresponding to the aforementioned multiple edge nodes to the aforementioned edge server.

[0098] In step S508, the edge server updates the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes, and obtains the parameter update results of the global model.

[0099] The execution entities of the above steps S502 to S508 are edge servers and multiple edge nodes. Through the above steps S502 to S508, the heterogeneity of different edge nodes and the dynamic changes of runtime resources can be taken into account to carry out efficient model training. While ensuring the model training accuracy, the personalized training results of different edge nodes can be preserved, thereby achieving the technical effect of improving the model training accuracy and model applicability. This solves the technical problems of low model training accuracy and poor applicability in federated learning-based model training and processing methods in related technologies.

[0100] According to an embodiment of the present invention, a system embodiment for implementing the above-described model processing method is also provided. Figure 6 This is a schematic diagram of the structure of a model processing system according to an embodiment of the present invention, such as... Figure 6 As shown, the above model processing system includes: an edge server 600 and multiple edge nodes 602, wherein:

[0101] The aforementioned edge server 600 is used to determine the pruning ratio of each edge node based on the available resource status of each edge node, wherein the pruning ratio is used to indicate the proportion of pruning processing performed on the global model; based on the pruning ratio of each edge node, the global model is pruned to obtain the local model corresponding to each edge node; and the local model corresponding to each edge node is sent to the corresponding edge node.

[0102] The aforementioned multiple edge nodes 602 are connected to the aforementioned edge server 600 to train the corresponding local model based on local data, obtain the local parameter training results of the local model corresponding to the aforementioned multiple edge nodes, and send the local parameter training results of the local model corresponding to the aforementioned multiple edge nodes to the aforementioned edge server.

[0103] The aforementioned edge server 600 is also used to update the parameters of the aforementioned global model based on the local parameter training results of the local models corresponding to the aforementioned multiple edge nodes, thereby obtaining the parameter update results of the aforementioned global model.

[0104] In this embodiment of the invention, by setting the aforementioned edge server 600, it is used to determine the pruning ratio corresponding to each of the multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratio is used to indicate the proportion of pruning processing performed on the global model; based on the pruning ratio corresponding to each of the multiple edge nodes, the global model is pruned to obtain the local models corresponding to each of the multiple edge nodes; the local models corresponding to each of the multiple edge nodes are sent to the corresponding edge nodes; the multiple edge nodes 602 are connected to the aforementioned edge server 600, and are used to train the corresponding local models based on local data to obtain the local parameter training results of the local models corresponding to each of the multiple edge nodes, and then... The local parameter training results of the local models corresponding to multiple edge nodes are sent to the aforementioned edge server. The aforementioned edge server 600 is also used to update the parameters of the aforementioned global model based on the local parameter training results of the local models corresponding to the multiple edge nodes, thereby obtaining the parameter update results of the aforementioned global model. This achieves the goal of taking into account the heterogeneity of different edge nodes and the dynamic changes of runtime resources to carry out efficient model training, ensuring the accuracy of model training while preserving the personalized training results of different edge nodes. This achieves the technical effect of improving the model training accuracy and model applicability, and solves the technical problems of low model training accuracy and poor applicability in federated learning-based model training and processing methods in related technologies.

[0105] It should be noted that in this application Figure 6 The specific structure of the model processing system shown is merely illustrative. In practical applications, the model processing system in this application can be more advanced than... Figure 6 The model processing system shown has more or less structure.

[0106] It should be noted that any optional or preferred model processing method in the above method embodiments can be executed or implemented in the model processing system provided in this embodiment.

[0107] Furthermore, it should be noted that the optional or preferred implementation methods of this embodiment can be found in the relevant descriptions in the method embodiments, which will not be repeated here.

[0108] This embodiment also provides a model processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module" and "apparatus" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0109] According to an embodiment of the present invention, an apparatus embodiment for implementing the above-described model processing method is also provided. The model processing apparatus includes: a first determining module, a first acquiring module, a first sending module, a first receiving module, and a second acquiring module, wherein:

[0110] The first determining module is used to determine the pruning ratio corresponding to each of the multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratio is used to indicate the proportion of pruning processing performed on the global model.

[0111] The first acquisition module is connected to the first determination module and is used to perform the pruning process on the global model based on the pruning ratios corresponding to the multiple edge nodes to obtain the local models corresponding to the multiple edge nodes.

[0112] The first sending module is connected to the first acquiring module and is used to send the local models corresponding to the multiple edge nodes to the corresponding edge nodes.

[0113] The first receiving module is connected to the first sending module and is used to receive the local parameter training results of the corresponding local models sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training the corresponding local models based on local data.

[0114] The second acquisition module is connected to the first receiving module and is used to update the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes, so as to obtain the parameter update results of the global model.

[0115] In this embodiment of the invention, the first determining module is configured to determine the pruning ratio of each edge node based on the available resource status of each edge node, wherein the pruning ratio indicates the proportion of pruning processing performed on the global model; the first acquiring module is connected to the first determining module and is configured to perform the pruning processing on the global model based on the pruning ratio of each edge node to obtain the local model corresponding to each edge node; the first sending module is connected to the first acquiring module and is configured to send the local models corresponding to each edge node to the corresponding edge nodes; the first receiving module is connected to the first sending module and is configured to receive the local parameter training results of the corresponding local models sent by the edge nodes. The result is that the local parameter training result is obtained by training the corresponding local model based on local data by the multiple edge nodes; the second acquisition module is connected to the first receiving module and is used to update the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes, so as to obtain the parameter update result of the global model. This achieves the goal of taking into account the heterogeneity of different edge nodes and the dynamic changes of runtime resources to carry out efficient model training, and preserves the personalized training results of different edge nodes while ensuring the model training accuracy. This achieves the technical effect of improving the model training accuracy and model applicability, and solves the technical problems of low model training accuracy and poor applicability of federated learning-based model training and processing methods in related technologies.

[0116] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0117] It should be noted that the first determining module, the first acquiring module, the first sending module, the first receiving module, and the second acquiring module mentioned above correspond to steps S102 to S110 in the embodiments. The instances and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in a computer terminal.

[0118] It should be noted that the optional or preferred implementation methods of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0119] The aforementioned model processing device may further include a processor and a memory. The first determining module, the first acquiring module, the first sending module, the first receiving module, the second acquiring module, etc., are all stored in the memory as program modules, and the processor executes the aforementioned program modules stored in the memory to realize the corresponding functions.

[0120] The processor contains a core that retrieves the corresponding program modules from memory. One or more cores may be configured. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip.

[0121] According to an embodiment of this application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein, when the program runs, it controls the device where the non-volatile storage medium is located to execute any of the above-mentioned model processing methods.

[0122] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals, and the non-volatile storage medium includes stored programs.

[0123] Optionally, during program execution, the device containing the non-volatile storage medium performs the following functions: determining the pruning ratio for each edge node based on the available resource status of each edge node, wherein the pruning ratio indicates the proportion of pruning performed on the global model; performing the pruning process on the global model based on the pruning ratio for each edge node to obtain local models for each edge node; sending the local models for each edge node to the corresponding edge node; receiving the local parameter training results of the local models sent by the edge nodes, wherein the local parameter training results are obtained by the edge nodes training the corresponding local models based on local data; and updating the parameters of the global model based on the local parameter training results of the local models for each edge node to obtain the parameter update results of the global model.

[0124] According to an embodiment of this application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein the program executes any of the above-described model processing methods during runtime.

[0125] According to an embodiment of this application, an embodiment of a computer program product is also provided, which, when executed on a data processing device, is adapted to execute a program that initializes the model processing method steps described above.

[0126] Optionally, when the aforementioned computer program product is executed on a data processing device, it is suitable to execute an initialization program with the following method steps: determining the pruning ratio corresponding to each of the multiple edge nodes based on the available resource status of each edge node, wherein the pruning ratio is used to indicate the proportion of pruning processing performed on the global model; performing the pruning processing on the global model based on the pruning ratio corresponding to each of the multiple edge nodes to obtain local models corresponding to each of the multiple edge nodes; sending the local models corresponding to each of the multiple edge nodes to the corresponding edge nodes; receiving the local parameter training results of the corresponding local models sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training the corresponding local models based on local data; and updating the parameters of the global model based on the local parameter training results of the local models corresponding to each of the multiple edge nodes to obtain the parameter update results of the global model.

[0127] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: determining pruning ratios for each of the multiple edge nodes based on their respective available resource states, wherein the pruning ratios indicate the proportion of pruning performed on the global model; performing the pruning process on the global model based on the pruning ratios for each of the multiple edge nodes to obtain local models corresponding to each of the multiple edge nodes; sending the local models corresponding to each of the multiple edge nodes to their respective edge nodes; receiving local parameter training results of the corresponding local models sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training their respective local models based on local data; and updating the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes to obtain parameter update results for the global model.

[0128] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of modules described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between modules, and may be electrical or other forms.

[0131] The modules described above as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0132] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0133] If the aforementioned integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0134] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A model data processing method, characterized in that, Applied to resource-constrained heterogeneous edge computing scenarios, including: The edge server determines the pruning ratio for each edge node based on the available resource status of each edge node. The pruning ratio indicates the proportion of pruning processing performed on the global model. The multiple edge nodes are multiple mobile terminal devices. The available resource status includes the heterogeneity of the resources of the corresponding edge nodes and the dynamic changes of runtime resources. The edge server performs the pruning process on the global model based on the pruning ratios corresponding to the multiple edge nodes, thereby obtaining the local models corresponding to the multiple edge nodes. The edge server sends the local models corresponding to the multiple edge nodes to the corresponding edge nodes; The edge server receives the local parameter training results of the corresponding local models sent by the multiple edge nodes, wherein the local parameter training results are obtained by the multiple edge nodes training the corresponding local models based on local data, and the local parameter training results integrate the differential features of the multiple edge nodes. The edge server updates the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes, and obtains the parameter update results of the global model. The step of pruning the global model based on the pruning ratios corresponding to the multiple edge nodes to obtain the local models corresponding to the multiple edge nodes includes: determining a first arrangement order of the pruning ratios corresponding to the multiple edge nodes, wherein the first arrangement order is in ascending order; and obtaining the local models corresponding to the multiple edge nodes according to the first arrangement order in the following manner: using the global model as the first initial model, taking the local model corresponding to the first edge node among two adjacent edge nodes as the initial model for the pruning process of the next edge node, and performing the pruning process on the initial model to obtain the local model corresponding to the next edge node.

2. The method according to claim 1, characterized in that, The parameter update of the global model is performed based on the local parameter training results of the local models corresponding to the multiple edge nodes, to obtain the parameter update results of the global model, including: Obtain a second sorting order of the model sizes of the local models corresponding to the multiple edge nodes, wherein the second sorting order is an ascending order, and the model sizes of the local models corresponding to the multiple edge nodes are inversely proportional to the corresponding pruning ratios; based on the local parameter training results of the local models corresponding to the multiple edge nodes, obtain the first parameter update results of the local models corresponding to the multiple edge nodes according to the second sorting order; The parameter update results of the local models corresponding to the multiple edge nodes are summed to obtain the parameter update results of the global model.

3. The method according to claim 2, characterized in that, The first parameter update results of the local models corresponding to the multiple edge nodes are obtained according to the second arrangement based on the local parameter training results of the local models corresponding to the multiple edge nodes, including: Based on the local parameter training results of the local models corresponding to the multiple edge nodes, and in accordance with the second arrangement order, the first parameter update results of the local models corresponding to the multiple edge nodes are obtained in the following manner: taking any one model as the base model, the weight values ​​of the local models corresponding to the multiple edge nodes are determined; the first parameter training results corresponding to the predetermined model parameters included in the local parameters training results of the local models corresponding to the multiple edge nodes are determined; based on the weight values ​​of the local models corresponding to the multiple edge nodes and the first parameter training results of the local models corresponding to the multiple edge nodes, the first parameter update result of the arbitrary model is determined.

4. The method according to claim 3, characterized in that, Determining the weight values ​​of the local model corresponding to each of the plurality of edge nodes includes: Using any one of the models as the baseline model, and based on the local parameter training results of the local models corresponding to the multiple edge nodes respectively, determine the gradient direction distance between the local models corresponding to the multiple edge nodes and the arbitrary model respectively; The weight values ​​of the local models corresponding to the multiple edge nodes are determined based on the gradient direction distance between the local models corresponding to the multiple edge nodes and any one of the models.

5. The method according to claim 4, characterized in that, The step of determining the weight values ​​of the local models corresponding to the multiple edge nodes based on the gradient direction distance between the local models corresponding to the multiple edge nodes and any one of the models includes: determining the sort number of the any one of the models in the second arrangement order, and the sort number of the local models corresponding to the multiple edge nodes in the second arrangement order. Based on the order number of the local models corresponding to the multiple edge nodes in the second arrangement order, the size relationship between the local models corresponding to the multiple edge nodes and the order number of any one model in the second arrangement order, and the gradient direction distance between the local models corresponding to the multiple edge nodes and any one model, the weight values ​​of the local models corresponding to the multiple edge nodes are determined.

6. The method according to claim 5, characterized in that, The determination of the weight values ​​of the local models corresponding to the multiple edge nodes, based on the relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one model, and the gradient direction distance between the local models corresponding to the multiple edge nodes and the any one model, includes: Based on the size relationship between the sorting numbers of the local models corresponding to the multiple edge nodes and the sorting number of any one model, a first model with a sorting number less than the sorting number of any one model and a second model with a sorting number greater than or equal to the sorting number of any one model are determined among the local models corresponding to the multiple edge nodes. The weight value of the first model is determined to be zero; The weight value of the second model is determined based on the gradient direction distance between the second model and any of the aforementioned models.

7. The method according to any one of claims 1 to 6, characterized in that, The parameter update of the global model is performed based on the local parameter training results of the local models corresponding to the multiple edge nodes, to obtain the parameter update results of the global model, including: Determine whether a preset termination condition has been met, wherein the preset termination condition is at least one of the following: the accuracy of the global model reaches a preset expected value, the number of training iterations reaches a preset iteration threshold, and the loss value of the global model is less than or equal to a preset loss threshold. If the preset termination condition is met, the parameters of the global model are updated based on the local parameter training results obtained from the last training of the local models corresponding to the multiple edge nodes, to obtain the parameter update results of the global model.

8. A model data processing system, characterized in that, Applied to resource-constrained heterogeneous edge computing scenarios, including: An edge server is used to determine the pruning ratio for each edge node based on the available resource status of each edge node, wherein the pruning ratio indicates the proportion of pruning processing performed on the global model; based on the pruning ratio for each edge node, the global model is pruned to obtain the local model corresponding to each edge node; the local model corresponding to each edge node is sent to the corresponding edge node, wherein the multiple edge nodes are multiple mobile terminal devices, and the available resource status includes: the heterogeneity of resources of the corresponding edge nodes and the dynamic changes of runtime resources; The plurality of edge nodes are connected to the edge server and are used to train the corresponding local model based on local data to obtain the local parameter training results of the local model corresponding to the plurality of edge nodes respectively, and send the local parameter training results of the local model corresponding to the plurality of edge nodes to the edge server. The local parameter training results integrate the differential features of the plurality of edge nodes. The edge server is also used to update the parameters of the global model based on the local parameter training results of the local models corresponding to the multiple edge nodes respectively, so as to obtain the parameter update results of the global model. The edge server is further configured to determine a first arrangement order of the pruning ratios corresponding to the plurality of edge nodes, wherein the first arrangement order is in ascending order; according to the first arrangement order, the local models corresponding to the plurality of edge nodes are obtained in the following manner: taking the global model as the first initial model, taking the local model corresponding to the first edge node in two adjacent edge nodes as the initial model for the pruning process of the second edge node, and performing the pruning process on the initial model to obtain the local model corresponding to the second edge node.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions, which are adapted to be loaded by a processor and executed by the model processing method according to any one of claims 1 to 7.

Images