Computing power network node knowledge migration method, device and system

Abstract

The present invention provides a method, device, and system for transferring knowledge of computing power network nodes, wherein the method includes: sending characteristic information of the weight matrix of the computing power feature neural network to a central node, so that the central node sends a target number of deduplicated weight matrices of the computing power feature neural network to each target edge node based on the characteristic information of the weight matrix of the computing power feature neural network sent by each target edge node; receiving a target number of deduplicated weight matrices of the computing power feature neural network; obtaining a fixed part of the weight matrix of a first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and obtaining a fine-tuning part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples, so as to obtain the weight matrix of the first model. The method, device, and system for transferring knowledge of computing power network nodes provided by the present invention can realize knowledge transfer between computing power nodes.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and system for migrating knowledge of computing power network nodes. Background Art

[0002] With the continuous development of services such as artificial intelligence, edge computing, the Industrial Internet, and the Internet of Vehicles, computing resources are becoming increasingly widespread across cloud, network, edge, and end. Furthermore, a digital, networked, and intelligent society generates massive amounts of data, and data processing requires a new computing infrastructure that collaborates between the cloud, edge, and end. However, computing power, algorithms, and data are diverse, widely distributed, and fragmented, mostly deployed within specific types of computing resources. This results in a significant reliance on central computing resources, leading to excessive network load and idle computing resources at the edge and end. Furthermore, computing power, algorithms, and data are difficult to coordinate, reuse, and migrate across different types and levels of computing resources, resulting in low overall computing resource utilization.

[0003] To address issues such as physical isolation of computing resources and data dispersion, solutions based on data segmentation and model segmentation are currently being used to address the problem of long-term idle edge computing resources and improve computing resource collaboration and knowledge transfer capabilities. The most representative of these is the federated learning algorithm.

[0004] The basic idea of ​​the federated learning algorithm is to first train local models on each local node. After the local models are optimized, they are weighted averaged to generate the overall model. During the federated learning training process, the computing network transmits the gradient information of each local model, which to a certain extent avoids the high network load and data security issues caused by data transmission. Furthermore, by distributing a large amount of model training processes to distributed computing nodes, the overall computing resource utilization of the computing network is relatively improved.

[0005] However, existing technologies cannot achieve knowledge transfer between nodes in scenarios such as new model training or computing network expansion. Summary of the Invention

[0006] The present invention provides a computing power network node knowledge migration method, device and system to solve the defects in the existing technology and achieve.

[0007] In a first aspect, the present invention provides a method for migrating knowledge of computing power network nodes, comprising:

[0008] Sending characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node;

[0009] Receiving the target number of weight matrices of the computing power feature neural network after deduplication;

[0010] Obtaining a fixed portion of the weight matrix of the first model based on the target number of weight matrices of the deduplicated computing power feature neural network, and performing training based on the target number of weight matrices of the deduplicated computing power feature neural network and training samples to obtain a fine-tuning portion of the weight matrix of the first model, thereby obtaining the weight matrix of the first model;

[0011] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0012] According to a computing power network node knowledge migration method provided by the present invention, the fixed part of the weight matrix of the first model is obtained based on the weight matrix of the computing power feature neural network after deduplication of the target number, and the fine-tuning part of the weight matrix of the first model is obtained based on the weight matrix of the computing power feature neural network after deduplication of the target number and training samples, thereby obtaining the weight matrix of the first model, further comprising:

[0013] Concatenating the weight matrix of the first model and the weight matrices of each historical neural network to obtain a concatenated matrix;

[0014] Performing singular value decomposition on the concatenated matrix to obtain a feature compression matrix corresponding to the weight matrix of the new computing power feature neural network;

[0015] The concatenated matrix is ​​compressed based on the feature compression matrix to obtain a new weight matrix of the computing power feature neural network.

[0016] A computing power network node knowledge migration method provided by the present invention further includes:

[0017] Representing the weight matrix of each of the historical neural networks as the product of a reconstruction matrix and a mapping matrix; wherein the reconstruction matrix is ​​generated based on a random seed;

[0018] Correspondingly, the characteristic information of the weight matrix of the computing power feature neural network includes the characteristic compression matrix corresponding to the weight matrix of the computing power feature neural network, each of the random seeds and each of the mapping matrices.

[0019] In a second aspect, the present invention further provides a method for migrating computing power network node knowledge, comprising:

[0020] Obtaining a weight matrix of a computing power feature neural network of each target edge node;

[0021] Sending the target number of deduplicated weight matrices of the computing power feature neural network to each of the target edge nodes, so that each of the target edge nodes obtains a fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and performs training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain a fine-tuning part of the weight matrix of the first model, thereby obtaining the weight matrix of the first model;

[0022] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0023] In a third aspect, the present invention further provides a computing power network node knowledge migration device, comprising:

[0024] A sending module is used to send characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node;

[0025] A receiving module, configured to receive the weight matrix of the target number of the computing power feature neural network after deduplication;

[0026] a training module, configured to obtain a fixed portion of the weight matrix of the first model based on the target number of weight matrices of the computing power feature neural network after deduplication, and perform training based on the target number of weight matrices of the computing power feature neural network after deduplication and training samples to obtain a fine-tuning portion of the weight matrix of the first model, so as to obtain the weight matrix of the first model;

[0027] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0028] In a fourth aspect, the present invention further provides a computing power network node knowledge migration device, comprising:

[0029] An acquisition module, configured to acquire a weight matrix of a computing power feature neural network of each target edge node;

[0030] A sending module is used to send the target number of deduplicated weight matrices of the computing power feature neural network to each of the target edge nodes, so that each of the target edge nodes obtains a fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and performs training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain a fine-tuning part of the weight matrix of the first model, so as to obtain the weight matrix of the first model;

[0031] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0032] In the fifth aspect, the present invention also provides a computing power network node knowledge migration system, comprising: the computing power network node knowledge migration device provided in the fourth aspect and multiple computing power network node knowledge migration devices provided in the third aspect.

[0033] In a sixth aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the computing power network node knowledge migration method as described in any one of the above is implemented.

[0034] In a seventh aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned computing power network node knowledge migration methods.

[0035] In an eighth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for migrating computing power network node knowledge.

[0036] The computing power network node knowledge migration method, device and system provided by the present invention solidify the global model parameters of each computing power node participating in the operation and store them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge migration between computing power nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is one of the flow charts of the computing power network node knowledge migration method provided by the present invention;

[0039] Figure 2 Schematic diagram of an application scenario of the computing power network node knowledge migration method provided by the present invention;

[0040] Figure 3 This is the second flow chart of the computing power network node knowledge migration method provided by the present invention;

[0041] Figure 4 This is the third flow chart of the computing power network node knowledge migration method provided by the present invention;

[0042] Figure 5 This is one of the structural diagrams of the computing power network node knowledge migration device provided by the present invention;

[0043] Figure 6 This is the second structural diagram of the computing power network node knowledge migration device provided by the present invention;

[0044] Figure 7 It is a structural diagram of the computing power network node knowledge migration system provided by the present invention;

[0045] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0047] In the description of the embodiments of the present invention, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance, and are not related to the order.

[0048] The following combination Figures 1 to 8 Describe the computing power network node knowledge migration method, device and system provided by the present invention.

[0049] Figure 1 This is one of the flow charts of the method for transferring computing power network node knowledge provided by the present invention. Figure 1 As shown, the execution subject of the computing power network node knowledge migration method provided by the embodiment of the present invention can be a computing power network node knowledge migration device, and the method includes: step 101, step 102 and step 103.

[0050] Specifically, Figure 2 This is a schematic diagram of an application scenario of the computing power network node knowledge migration method provided by the present invention. The computing power network node knowledge migration method provided by the embodiment of the present invention can be used for Figure 2 The scene shown.

[0051] like Figure 2 As shown, each first edge node in the computing power network obtains the weight matrix W1 of the first neural network model Model1 through model training, and each second edge node in the computing power network obtains the weight matrix W2 of the second neural network model Model2 through model training. It is necessary to train each edge node in the dotted line box in a new model training scenario to obtain a new neural network model Model3.

[0052] It can be understood that the hidden layer of a neural network can be regarded as a characteristic process of using matrix multiplication on the input data. By extracting information from the matrix, information such as data mining and feature screening can be obtained. Therefore, the neural network can be represented by its weight matrix.

[0053] In traditional solutions based on federated learning algorithms, the data processing structure is still "snowflake-style", that is, the data center (which can be the central node in the computing network) is used as the aggregation and coordination point for processing, and data analysis between computing nodes is often isolated. Knowledge transfer between nodes cannot be achieved in scenarios such as new model training or computing network expansion.

[0054] Figure 2The scenario shown is a new model training or computing power network expansion scenario, in which a mixture of some first edge nodes, some second edge nodes, and third edge nodes (edge ​​nodes that are neither first edge nodes nor second edge nodes) are used as target edge nodes. For this scenario, in the traditional solution based on the federated learning algorithm, the weight matrices W1 and W2 of the historical models (including Model1 and Model2) cannot assist in the model construction in the new scenario, that is, the historical model of the first edge node cannot assist in the new model training of the second edge node and the third edge node, and vice versa. The above problems have caused the loss of the value of historical mining data to a certain extent, which may lead to a decline in data mining quality in the new scenario. In addition, the independent model training in each scenario cannot solidify part of the model and feature engineering to the computing power node (referring to the nodes in the computing power network, including edge nodes and central nodes). The model update requires the repeated transmission of complete neural network parameters (i.e., the weight matrix of the neural network) in the computing power network. There is still the problem of high network load and idle computing resources as a whole.

[0055] It should be noted that each target edge node can serve as a computing power network node knowledge migration device that executes the computing power network node knowledge migration method in the embodiment of the present invention.

[0056] Step 101: Send characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained by the target edge node compressing the weight matrices of each historical neural network of the target edge node.

[0057] Specifically, the target edge nodes are edge nodes that participate in obtaining the first model. The first model is a neural network model.

[0058] After determining which edge nodes in the computing power network the target edge nodes are, each target edge node can send the characteristic information of the weight matrix of its own computing power characteristic neural network to the central node, but may not send the complete computing power characteristic neural network to reduce the amount of data to be transmitted and reduce the bandwidth load.

[0059] The characteristic information of the weight matrix of the computing power feature neural network is used to reconstruct the weight matrix of the computing power feature neural network. The central node can reconstruct the weight matrix of the computing power feature neural network based on the characteristic information of the weight matrix of the computing power feature neural network.

[0060] The central node can deduplicate the weight matrices of the reconstructed computing power feature neural networks, and then determine the target number of deduplicated weight matrices of the computing power feature neural networks based on the deduplicated weight matrices of the computing power feature neural networks.

[0061] Optionally, when the number of weight matrices of the deduplicated computing power feature neural network is greater than the preset number N, the target number can be the preset number N, and the central node can randomly extract N weight matrices of the deduplicated computing power feature neural network to reduce the number of pre-trained matrices of the computing power nodes; when the number of weight matrices of the deduplicated computing power feature neural network is less than or equal to the preset number N, the target number can be the number of weight matrices of the deduplicated computing power feature neural network, and the weight matrices of all deduplicated computing power feature neural networks can be determined as the target number of weight matrices of the deduplicated computing power feature neural network.

[0062] After determining the weight matrix of the target number of deduplicated computing power feature neural networks, the central node can send the weight matrix of the target number of deduplicated computing power feature neural networks to each target edge node.

[0063] It should be noted that before step 101, the global model parameters for each participating computing node are solidified and stored in the computing node to obtain the weight matrix of the historical neural network. Each edge node can compress the weight matrices of its own historical neural networks using any applicable feature compression method to obtain a weight matrix of the computing power feature neural network that retains the characteristics of the weight matrices of its own historical neural networks. The computing power feature neural network is the feature neural network of the target edge node.

[0064] Step 102: Receive the target number of weight matrices of the deduplicated computing power feature neural network.

[0065] Specifically, the target edge node can receive the target number of deduplicated weight matrices of the computing power feature neural network sent by the central node.

[0066] Step 103: Obtain the fixed part of the weight matrix of the first model based on the weight matrix of the target number of deduplicated computing power feature neural networks, and perform training based on the weight matrix of the target number of deduplicated computing power feature neural networks and training samples to obtain the fine-tuning part of the weight matrix of the first model to obtain the weight matrix of the first model.

[0067] Specifically, the target edge node can concatenate the weight matrices of the target number of deduplicated computing power feature neural networks as the weight matrix of its own pre-trained neural network. This pre-trained neural network is part of the first model. This pre-trained neural network is solidified during subsequent model training and does not participate in updates. Therefore, the weight matrix of this pre-trained neural network can be used as a fixed part of the weight matrix of the first model.

[0068] The portion of the weight matrix of the first model other than the fixed portion is the fine-tuning portion, and the value of the fine-tuning portion can be fine-tuned through model training. This fine-tuning process can utilize joint training methods such as federated learning algorithms. The first model is a global neural network, and the fine-tuning portion of the weight matrix of the first model represents the fine-tuned neural network in the first model.

[0069] After determining the value of the fine-tuning part in the weight matrix of the first model through fine-tuning, the fixed part and the fine-tuning part in the weight matrix of the first model can be combined to obtain the weight matrix of the first model, and then the first model represented by the weight matrix of the first model can be obtained. Figure 2 Model 3 in the illustrated scenario may be a second model obtained by averaging the first models of the target nodes.

[0070] Optionally, the weight matrix of the first model can be obtained through the following fine-tuning steps:

[0071] 1. Pre-structure the neural network parameters and express them as a low-rank matrix, that is, let W i mn =A i mk B i kn ; Among them A i mk is the reconstruction matrix, B i kn is the mapping matrix, where the reconstruction matrix A i mk In each model training scenario, a random seed Q is specified i generate.

[0072] Among them, W i m represents the weight matrix of the first model, the fine-tuning part of which is updated during the model training process until the training is completed; i is the number of the target edge node; m is the number of input nodes in the first model; n is the number of output nodes in the first model; k is a predetermined positive integer, k is less than m and k is less than n.

[0073] 2. In the target edge node, based on the localized data of the target edge node, B is generated according to the back propagation gradient and the gradient descent method.i kn Optimization.

[0074] 3. During the model training process, the target edge node will use the local random seed Q i and the local mapping matrix B i kn And the local error E i Transmit to the central node.

[0075] 4. The central node first generates A based on the random seed for the weight matrix i mk , for the weight matrix W i mn =A i mk *B i kn After recovery, the average of each target edge node is performed to generate the global matrix W g mn , for error E i The summation generates the global error.

[0076] 5. The central node converts the global matrix W g mn The data is sent to the computing power nodes, and the computing power nodes and the central node collaborate to repeat steps 1-4 for the next round of global model optimization until the central node error converges.

[0077] When the global error does not converge, the central node will convert the global matrix W g mn Sent to the computing power node; when the global error converges, the model training is completed, and the first model and Model3 are determined.

[0078] After deduplication and sampling of the computing power feature matrix, the central node sends and solidifies the computing power feature neural network, realizing the feature knowledge migration between computing power matrices and the sinking of computing power; feature engineering derivation and model fine-tuning are performed on the computing power node to generate a fine-tuned neural network for the new scenario, and interact with the central node for global optimization. Since the feature neural network contains historical scenario knowledge, the fine-tuned neural network is generally much smaller than the global neural network, which can effectively reduce the network load during the neural network transmission process.

[0079] The computing power network node knowledge migration method provided by the embodiment of the present invention is a computing power network knowledge migration solution for the coexistence of multi-level computing power resources (cloud computing, edge computing and end computing). It accumulates, extracts and compresses the historical model parameters of the computing power nodes to form a computing power feature neural network for the data characteristics of the computing power nodes; after the new scene arrives, the computing power feature neural network is deduplicated, sampled, broadcast and solidified for the parameters of some computing power nodes participating in the calculation, and only fine-tunes and optimizes the subsequent network in subsequent training. This realizes the refinement and accumulation of historical training knowledge and the sinking of computing power of feature engineering, effectively reduces network load, improves the overall utilization rate of the computing power network, and provides customers with a consistent experience and flexible service deployment.

[0080] The embodiment of the present invention solidifies the global model parameters of each computing power node participating and stores them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge transfer between computing power nodes.

[0081] Based on the content of any of the above embodiments, the fixed part of the weight matrix of the first model is obtained based on the weight matrix of the target number of deduplicated computing power feature neural networks, and the fine-tuning part of the weight matrix of the first model is obtained based on the weight matrix of the target number of deduplicated computing power feature neural networks and training samples. After obtaining the weight matrix of the first model, it also includes: splicing the weight matrix of the first model and the weight matrices of each historical neural network to obtain a spliced ​​matrix.

[0082] Specifically, after completing the iteration of the new scenario, the new matrix is ​​used to generate a new computing power node feature neural network to achieve knowledge accumulation.

[0083] After obtaining the first model, the first model becomes the new historical neural network. The new historical neural network of the target edge node can be spliced ​​with the original historical neural networks, and the entire historical neural network can be represented as W h =[(A i0 B i0 ) T ,(A i1 B i2 ) T …(A iN B iN ) T ] T. In the sense of neural networks, it is equivalent to splicing the hidden layers of each independent model's neural network.

[0084] Among them, the subscript i represents the number of the target edge node; W h represents the splicing matrix; the subscripts 0 to N are the numbers of the 0th to Nth historical neural networks respectively.

[0085] Perform singular value decomposition on the splicing matrix to obtain the feature compression matrix corresponding to the weight matrix of the new computing power feature neural network.

[0086] Specifically, the splicing matrix W h can be subjected to singular value decomposition (SVD, Singular Value Decomposition), and W h is expressed as W h T = USV T . Among them, U is the key feature compression matrix of W h [[ID=I22]] (which can be simply referred to as the "feature compression matrix").

[0087] Optionally, the singular value matrix S can be truncated by truncated singular value decomposition (Truncated SVD).

[0088] Optionally, the truncation criterion can be to gradually reduce the dimension of S, and reconstruct W h [[ID=I30]] according to the splicing steps of all the foregoing historical neural networks to generate the reconstructed matrix W hc . According to the target that the L2 relative distance between W hc and W h is < a, determine the best truncation dimension.

[0089] Compress the splicing matrix based on the feature compression matrix to obtain the weight matrix of the new computing power feature neural network.

[0090] Specifically, multiply the feature compression matrix U T with W h to generate the compressed weight matrix U T W h of the computing power feature neural network, realizing knowledge accumulation.

[0091] In the embodiment of the present invention, by performing singular value decomposition on the weight matrix of the first model and the splicing matrix of the weight matrices of each historical neural network, the weight matrix of the new computing power feature neural network is obtained, realizing strong feature extraction, compression, and duplicate removal of the parameter matrix, reducing the amount of data to be transmitted between the edge node and the central node, and reducing the bandwidth load. In practical applications, according to different compression ratios, the feature compression ratio can be above 70%, that is, the amount of data to be transmitted can be reduced by about 70%.

[0092] Based on the content of any of the above embodiments, the computing power network node knowledge migration method also includes: representing the weight matrix of each historical neural network as the product of a reconstruction matrix and a mapping matrix; the reconstruction matrix is ​​generated based on a random seed.

[0093] Specifically, the neural network parameters can be pre-structured and represented as a low-rank matrix, that is, let W i mn =A i mk B i kn ; Among them A i mk is the reconstruction matrix, B i kn is the mapping matrix, where the reconstruction matrix A i mk In each model training scenario, a random seed Q is specified i generate.

[0094] Among them, W i m represents the weight matrix of the first model, the fine-tuning part of which is updated during the model training process until the training is completed; i is the number of the target edge node; m is the number of input nodes in the first model; n is the number of output nodes in the first model; k is a predetermined positive integer, k is less than m and k is less than n.

[0095] Through pre-structured training, the computing power nodes can store random seeds and mapping matrices to achieve compressed storage of the neural network with a compression rate of m / k.

[0096] The problem of limited storage resources in computing power nodes can be solved by training and compressing the neural network parameters in computing power nodes through pre-structuring algorithms.

[0097] Correspondingly, the characteristic information of the weight matrix of the computing power feature neural network includes the characteristic compression matrix, each random seed and each mapping matrix corresponding to the weight matrix of the computing power feature neural network.

[0098] Accordingly, the target edge node can include the random seed Q ij and B ij , and the feature compression matrix U T The information included is used as the feature information of the weight matrix of its own computing power feature neural network and transmitted to the central node.

[0099] Wherein, subscript i is the number of the target edge node; subscript j is the number of the historical neural network of the target edge node.

[0100] By representing the weight matrix of each historical neural network as the product of a reconstruction matrix and a mapping matrix, this embodiment of the present invention achieves compressed storage of the neural network. This reduces the amount of data required to be transmitted between edge nodes and central nodes, thereby reducing bandwidth load. In practical applications, the compression rate of the neural network is over 60%.

[0101] Figure 3 This is the second flow chart of the method for transferring knowledge of computing network nodes provided by the present invention. Based on the content of any of the above embodiments, Figure 3 As shown, the execution subject of the computing power network node knowledge migration method provided by the embodiment of the present invention can be a computing power network node knowledge migration device, and the method includes: step 301 and step 302.

[0102] Specifically, the computing power network node knowledge migration device can be a central node in the computing power network.

[0103] Step 301: Obtain the weight matrix of the computing power feature neural network of each target edge node; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node.

[0104] Specifically, each edge node can compress the weight matrices of its own historical neural networks using any applicable feature compression method to obtain a weight matrix of a computing power feature neural network that retains the characteristics of the weight matrices of its own historical neural networks. The computing power feature neural network is the feature neural network of the target edge node.

[0105] After determining which edge nodes in the computing power network the target edge nodes are, each target edge node can send the characteristic information of the weight matrix of its own computing power characteristic neural network to the central node, but may not send the complete computing power characteristic neural network to reduce the amount of data to be transmitted and reduce the bandwidth load.

[0106] The characteristic information of the weight matrix of the computing power feature neural network is used to reconstruct the weight matrix of the computing power feature neural network. The central node can reconstruct the weight matrix of the computing power feature neural network based on the characteristic information of the weight matrix of the computing power feature neural network.

[0107] Step 302: Send the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node, so that each target edge node obtains the fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and performs training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain the fine-tuning part of the weight matrix of the first model, so as to obtain the weight matrix of the first model.

[0108] Specifically, the central node can deduplicate the weight matrices of each reconstructed computing power feature neural network, and then determine the target number of deduplicated weight matrices of the computing power feature neural network based on the deduplicated weight matrices of the computing power feature neural network.

[0109] Optionally, when the number of weight matrices of the deduplicated computing power feature neural network is greater than the preset number N, the target number can be the preset number N, and the central node can randomly extract N weight matrices of the deduplicated computing power feature neural network to reduce the number of pre-trained matrices of the computing power nodes; when the number of weight matrices of the deduplicated computing power feature neural network is less than or equal to the preset number N, the target number can be the number of weight matrices of the deduplicated computing power feature neural network, and the weight matrices of all deduplicated computing power feature neural networks can be determined as the target number of weight matrices of the deduplicated computing power feature neural network.

[0110] After determining the weight matrix of the target number of deduplicated computing power feature neural networks, the central node can send the weight matrix of the target number of deduplicated computing power feature neural networks to each target edge node.

[0111] The target edge node can receive the target number of deduplicated weight matrices of the computing power feature neural network sent by the central node. The target edge node can then concatenate these deduplicated weight matrices to form the weight matrix of its own pre-trained neural network. This pre-trained neural network, as part of the first model, is solidified during subsequent model training and does not participate in updates. Therefore, the weight matrix of this pre-trained neural network can serve as a fixed portion of the weight matrix of the first model.

[0112] The part other than the fixed part in the weight matrix of the first model is the fine-tuning part, and the value of the fine-tuning part can be fine-tuned through model training.

[0113] After determining the value of the fine-tuning part in the weight matrix of the first model through fine-tuning, the fixed part and the fine-tuning part in the weight matrix of the first model can be combined to obtain the weight matrix of the first model, and then the first model represented by the weight matrix of the first model can be obtained.

[0114] The embodiment of the present invention solidifies the global model parameters of each computing power node participating and stores them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge transfer between computing power nodes.

[0115] Figure 4 This is the third flow chart of the method for transferring knowledge of computing network nodes provided by the present invention. Based on the content of any of the above embodiments, Figure 4 As shown, the implementation process of the computing power network node knowledge transfer method may include the following steps:

[0116] Pre-structured neural network training and network compression:

[0117] Step 1 & 2: Edge local optimization

[0118] Step 1: Pre-structure the neural network parameters and express them as a low-rank matrix, that is, let W i mn =A i mk B i kn ; Among them A i mk is the reconstruction matrix, B i kn is the mapping matrix, where the reconstruction matrix A i mk In each model training scenario, a random seed Q is specified i generate.

[0119] The neural network can be represented as a low-rank pre-structured matrix and the random seeds and mapping matrices can be stored to achieve the reproduction of the neural network.

[0120] Step 2: In the computing power node, based on the localized data of the computing power node, perform B according to the back propagation gradient generation and gradient descent method. i kn Optimization.

[0121] Step 3. Upload random seeds and compress the network

[0122] During the model training process, the computing power node will use the local random seed Q i and the local mapping matrix B ikn And the local error E i Transmitted to the central node. Local mapping matrix B i kn It can represent a compressed neural network.

[0123] In the transmission of the global neural network, only the mapping matrix and random seeds need to be transmitted to achieve a compression ratio of m / k.

[0124] Step 4: Generate a global neural network on average

[0125] The center node weight matrix is ​​first generated according to the random seed A i mk , for the weight matrix W i mn =A i mk *B i kn After recovery, the average of multiple computing power nodes is performed to generate the global matrix W g mn , for error E i The summation generates the global error. The global matrix W g mn It can be the weight matrix of the global neural network.

[0126] Step 5: Send global neural network & Step 6: Iterate until convergence

[0127] The central node sends the global neural network and iterates repeatedly to generate the global optimal neural network.

[0128] The central node converts the global matrix W g mn The data is sent to the computing power nodes, and the computing power nodes and the central node collaborate to repeat Steps 1-4 for the next round of global model optimization until the central node error converges.

[0129] SVD-based feature neural network generation:

[0130] Step 7. Neural Network Splicing

[0131] In each model training scenario, repeat steps 1-5 to generate the computing power neural network matrix of each model, including the random seed Q ij and B ij (Where i represents the number of the computing power node, and j represents the number of the historical model that the computing power node participated in generating.) The historical model is the historical neural network. The computing power node stores the random seed and mapping matrix to store and restore the historical network, reducing the storage burden by a ratio of m / k.

[0132] Concatenate the historical neural networks in the computing power nodes, that is, represent the historical neural networks as W h =[(A i0 B i0 ) T ,(A i1 B i2 ) T …(A iN B iN ) T T . In the sense of neural networks, it is equivalent to concatenating the hidden layers of each independent model neural network.

[0133] Step8, SVD decomposition

[0134] Perform SVD decomposition on the concatenated matrix W h , that is, represent W h as W h T =USV T , where U is the key feature compression matrix of W h .

[0135] Use the SVD method to extract features and reduce the dimension of the concatenated neural network matrix. This process realizes adding a new layer on the basis of the original concatenated neural network.

[0136] Step9&10, SVD truncation and compression

[0137] Truncate the singular value matrix S: The truncation criterion is to gradually reduce the dimension of S and reconstruct W h according to Step7 to generate the reconstructed matrix W hc . According to the L2 relative distance <a between W hc and W h , determine the optimal truncation dimension.

[0138] Multiply the feature compression matrix U T with W h to generate the compressed computing power feature neural network U T W h .

[0139] Pre-training neural network generation:

[0140] Step11, Upload the feature neural network

[0141] In the new model scenario, the computing power neural network needs to transmit the stored computing power feature neural network U T W h to the central node. To reduce storage, the random seed Q ij and B ij , U​T Transmitted to the central node, the central node performs U T W h Reconstruction.

[0142] Step 12: Deduplication and Sampling

[0143] The central node will calculate the U T W h To reduce the amount of pre-trained matrices of computing nodes, the U T W h Sampling is performed. The sampling standard is: if the number of computing power feature neural networks after deduplication is greater than N, then N are randomly sampled; otherwise, all are sent to the computing power nodes; N is flexibly determined based on the computing power and storage capacity of the computing power nodes, and can also be optimized as one of the hyperparameters of the global model.

[0144] Step 13: Network distribution

[0145] The central node sends N feature matrices. The computing nodes will splice the feature matrices received from the central node, W r =[(U0 T W h0 ) T ,(U1 T W h1 ) T …(U N T W hN ) T ] T , as the pre-trained neural network of the computing power node, the network is solidified in the subsequent model training process and does not participate in the update.

[0146] Fine-tuning and global neural network generation:

[0147] Step 14: Pre-training feature generation

[0148] Multiply the computing power node data with the pre-trained neural network to obtain the pre-trained features, that is, P = W r D, W r With the bare feature D of the computing power node. In this step, feature splicing can be done by adding W rnew Expressed as [I,(U0 T W h0 ) T ,(U1 T W h1 ) T …(U N T W hN ) T ] T accomplish.

[0149] Step 15. Repeat Step 1-4 to generate fine-tuning neural network and global neural network

[0150] Repeat the process of Step 1-4 to fine-tune the neural network and generate the fine-tuned neural network W under the new scenario gt , the trained global neural network is identified as W gn =W gt* W rnew .

[0151] Step 16: Knowledge accumulation

[0152] After completing the iteration of the new scene, W gn As the weight matrix of the new historical model, repeat Steps 7-10 to generate the feature neural network of the new computing power node to achieve knowledge accumulation.

[0153] An embodiment of the present invention is a computing power node knowledge migration scheme based on a local pre-structured training-fine-tuning framework. Historical neural networks are compressed and stored at the computing power nodes, and feature extraction is performed using the SVD method to generate a computing power feature neural network. The central node deduplicates and samples the computing power node feature network, and sends it to the computing power node for solidification to form a pre-trained feature matrix. In subsequent training, only a fine-tuning network is generated to achieve knowledge migration between computing power nodes and the sinking of feature engineering computing power.

[0154] Through feature neural networks, the extraction and accumulation of historical training knowledge is achieved; through knowledge transfer and fine-tuning, knowledge transfer and computing power sinking between computing nodes are achieved. Since the computing power nodes have undergone effective feature engineering, the subsequent network is often much smaller than the traditional global network, thus effectively reducing the network load.

[0155] The computing power network node knowledge migration device provided by the present invention is described below. The computing power network node knowledge migration device described below and the computing power network node knowledge migration method described above can be referenced to each other.

[0156] Figure 5 This is one of the structural diagrams of the computing power network node knowledge migration device provided by the present invention. Based on the content of any of the above embodiments, such as Figure 5 As shown, the apparatus includes a sending module 501, a receiving module 502 and a training module 503, wherein:

[0157] A sending module 501 is configured to send characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node;

[0158] A receiving module 502 is configured to receive a target number of weight matrices of the deduplicated computing power feature neural network;

[0159] A training module 503 is configured to obtain a fixed portion of the weight matrix of the first model based on the weight matrix of the target number of deduplicated computing power feature neural networks, and to perform training based on the weight matrix of the target number of deduplicated computing power feature neural networks and training samples to obtain a fine-tuning portion of the weight matrix of the first model, thereby obtaining the weight matrix of the first model;

[0160] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0161] Specifically, the device may be a target edge node. The sending module 501, the receiving module 502 and the training module 503 may be electrically connected in sequence.

[0162] The sending module 501 can send the characteristic information of the weight matrix of its own computing power characteristic neural network to the central node, but may not send the complete computing power characteristic neural network to reduce the amount of data to be transmitted and reduce the bandwidth load.

[0163] The receiving module 502 can receive the target number of deduplicated weight matrices of the computing power feature neural network sent by the central node.

[0164] The training module 503 can splice the weight matrices of the target number of deduplicated computing power feature neural networks as the weight matrix of its own pre-trained neural network. The pre-trained neural network is part of the first model. The pre-trained neural network is solidified during the subsequent model training process and does not participate in the update. Therefore, the weight matrix of the pre-trained neural network can be used as a fixed part of the weight matrix of the first model.

[0165] The portion of the weight matrix of the first model other than the fixed portion is a fine-tuning portion, and the value of the fine-tuning portion can be fine-tuned through model training. The training module 503 can determine the value of the fine-tuning portion in the weight matrix of the first model through fine-tuning, and combine the fixed portion and the fine-tuning portion in the weight matrix of the first model to obtain the weight matrix of the first model, thereby obtaining the first model represented by the weight matrix of the first model.

[0166] Optionally, the device may further include:

[0167] The compression module is used to splice the weight matrix of the first model and the weight matrices of each historical neural network to obtain a spliced ​​matrix; perform singular value decomposition on the spliced ​​matrix to obtain a feature compression matrix corresponding to the weight matrix of the new computing power feature neural network; compress the spliced ​​matrix based on the feature compression matrix to obtain the weight matrix of the new computing power feature neural network.

[0168] Optionally, the device may further include:

[0169] A pre-structured module for representing the weight matrix of each historical neural network as the product of a reconstruction matrix and a mapping matrix; the reconstruction matrix is ​​generated based on a random seed;

[0170] Correspondingly, the characteristic information of the weight matrix of the computing power feature neural network includes the characteristic compression matrix, each random seed and each mapping matrix corresponding to the weight matrix of the computing power feature neural network.

[0171] The computing power network node knowledge migration device provided in an embodiment of the present invention is used to execute the above-mentioned computing power network node knowledge migration method of the present invention. Its implementation method is consistent with the implementation method of the computing power network node knowledge migration method provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0172] The computing power network node knowledge migration device is used in the computing power network node knowledge migration method of the aforementioned embodiments. Therefore, the description and definition of the computing power network node knowledge migration method in the aforementioned embodiments can be used to understand the various execution modules in the embodiments of the present invention.

[0173] The embodiment of the present invention solidifies the global model parameters of each computing power node participating and stores them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge transfer between computing power nodes.

[0174] Figure 6 This is the second structural diagram of the computing power network node knowledge migration device provided by the present invention. Based on the content of any of the above embodiments, such as Figure 6 As shown, the device includes an acquisition module 601 and a delivery module 602, wherein:

[0175] An acquisition module 601 is configured to acquire a weight matrix of a target edge node's computing power feature neural network of each target edge node;

[0176] A sending module 602 is used to send the weight matrix of the target number of deduplicated computing power feature neural networks to each target edge node, so that each target edge node obtains a fixed part of the weight matrix of the first model based on the weight matrix of the target number of deduplicated computing power feature neural networks, and performs training based on the weight matrix of the target number of deduplicated computing power feature neural networks and training samples to obtain a fine-tuning part of the weight matrix of the first model, thereby obtaining the weight matrix of the first model;

[0177] Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0178] Specifically, the device may be a central node. The acquisition module 601 and the sending module 602 may be electrically connected.

[0179] The acquisition module 601 can reconstruct the weight matrix of the computing power feature neural network based on the feature information of the weight matrix of the computing power feature neural network.

[0180] The sending module 602 can deduplicate the weight matrices of the reconstructed computing power feature neural networks, and then determine the target number of deduplicated weight matrices of the computing power feature neural networks based on the deduplicated weight matrices of the computing power feature neural networks.

[0181] After determining the target number of weight matrices of the deduplicated computing power feature neural network, the sending module 602 may send the target number of weight matrices of the deduplicated computing power feature neural network to each target edge node.

[0182] Optionally, the device may include a model generation module for averaging multiple computing nodes after the weight matrix is ​​restored to generate a global matrix.

[0183] The computing power network node knowledge migration device provided in an embodiment of the present invention is used to execute the above-mentioned computing power network node knowledge migration method of the present invention. Its implementation method is consistent with the implementation method of the computing power network node knowledge migration method provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0184] The computing power network node knowledge migration device is used in the computing power network node knowledge migration method of the aforementioned embodiments. Therefore, the description and definition of the computing power network node knowledge migration method in the aforementioned embodiments can be used to understand the various execution modules in the embodiments of the present invention.

[0185] The embodiment of the present invention solidifies the global model parameters of each computing power node participating and stores them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge transfer between computing power nodes.

[0186] Figure 7 This is a schematic diagram of the structure of the computing network node knowledge migration system provided by the present invention. Based on the content of any of the above embodiments, Figure 7 As shown, the system includes a central node 701 and multiple edge nodes 702 .

[0187] Specifically, the central node 701 can be a computing power network node knowledge migration device; the edge node 702 is a target edge node, which can be another computing power network node knowledge migration device.

[0188] The process of implementing computing power network node knowledge migration by the central node 701 and the edge node 702 can be referred to the aforementioned embodiments and will not be repeated here.

[0189] The embodiment of the present invention solidifies the global model parameters of each computing power node participating and stores them in the computing power node to obtain the weight matrix of the historical neural network. Based on the scene accumulation, the weight matrix of each historical neural network is compressed to obtain the weight matrix of the computing power feature neural network. In the new scene or the expansion of the computing power network, the central node broadcasts the weight matrix of the computing power feature neural network of each target edge node to other target edge nodes to perform feature derivation. In the training process of the new model, the feature neural network is solidified, and only the subsequent neural network is trained to generate a fine-tuning neural network, which is collaboratively generated with the feature neural network to achieve knowledge transfer between computing power nodes.

[0190] Figure 8 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the computing power network node knowledge transfer method, which includes: sending feature information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the feature information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node; receiving the target number of deduplicated weight matrices of the computing power feature neural network; obtaining the fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain the fine-tuning part of the weight matrix of the first model to obtain the weight matrix of the first model; wherein the computing power feature neural network of the target edge node The weight matrix of the network is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node; or includes: obtaining the weight matrix of the computing power feature neural network of the target edge node of each target edge node; sending the weight matrix of the computing power feature neural network after the target number of deduplication to each target edge node, so that each target edge node obtains the fixed part of the weight matrix of the first model based on the weight matrix of the computing power feature neural network after the target number of deduplication, and trains based on the weight matrix of the computing power feature neural network after the target number of deduplication and training samples to obtain the fine-tuning part of the weight matrix of the first model to obtain the weight matrix of the first model; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node.

[0191] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0192] The processor 810 in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory 830. Its implementation method is consistent with the implementation method of the computing power network node knowledge migration method provided in the present application, and can achieve the same beneficial effects, which will not be repeated here.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute the computing power network node knowledge transfer method provided by the above methods, and the method includes: sending characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node; receiving the target number of deduplicated weight matrices of the computing power feature neural network; obtaining the fixed part in the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain the micro-part in the weight matrix of the first model. Adjust the part to obtain the weight matrix of the first model; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node; or includes: obtaining the weight matrix of the computing power feature neural network of the target edge node of each target edge node; sending the weight matrix of the target number of deduplicated computing power feature neural networks to each target edge node, so that each target edge node obtains the fixed part of the weight matrix of the first model based on the weight matrix of the target number of deduplicated computing power feature neural networks, and trains based on the weight matrix of the target number of deduplicated computing power feature neural networks and training samples to obtain the fine-tuning part in the weight matrix of the first model to obtain the weight matrix of the first model; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0194] When the computer program product provided in the embodiment of the present application is executed, the above-mentioned computing power network node knowledge migration method is implemented. Its specific implementation method is consistent with the implementation method recorded in the embodiment of the aforementioned method, and can achieve the same beneficial effects, which will not be repeated here.

[0195] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the above-mentioned computing power network node knowledge migration method, the method comprising: sending characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends the characteristic information of the weight matrix of the computing power feature neural network to each target edge node, and sends the target number of deduplicated weight matrices of the computing power feature neural network to each target edge node; receiving the target number of deduplicated weight matrices of the computing power feature neural network; obtaining a fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain a fine-tuning part in the weight matrix of the first model to obtain the weight matrix of the first model; Among them, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node; or includes: obtaining the weight matrix of the computing power feature neural network of the target edge node for each target edge node; sending the weight matrix of the computing power feature neural network after the target number of deduplication to each target edge node, so that each target edge node obtains the fixed part of the weight matrix of the first model based on the weight matrix of the computing power feature neural network after the target number of deduplication, and trains based on the weight matrix of the computing power feature neural network after the target number of deduplication and training samples to obtain the fine-tuning part of the weight matrix of the first model to obtain the weight matrix of the first model; wherein, the weight matrix of the computing power feature neural network of the target edge node is obtained after the target edge node compresses the weight matrices of each historical neural network of the target edge node.

[0196] When the computer program stored on the non-transitory computer-readable storage medium provided in the embodiment of the present application is executed, the above-mentioned computing power network node knowledge migration method is implemented. Its specific implementation method is consistent with the implementation method described in the embodiment of the aforementioned method, and can achieve the same beneficial effects, which will not be repeated here.

[0197] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0198] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for transferring knowledge of computing network nodes, characterized in that: include: Sending characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends a target number of deduplicated weight matrices of the computing power feature neural network to each target edge node based on the characteristic information of the weight matrix of the computing power feature neural network sent by each target edge node; Receiving the target number of weight matrices of the computing power feature neural network after deduplication; Obtaining a fixed portion of the weight matrix of the first model based on the target number of weight matrices of the deduplicated computing power feature neural network, and performing training based on the target number of weight matrices of the deduplicated computing power feature neural network and training samples to obtain a fine-tuning portion of the weight matrix of the first model, thereby obtaining the weight matrix of the first model; The weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node; The computing power characteristic neural network is a characteristic neural network of a target edge node; the target edge node is an edge node that participates in obtaining the first model; the first model is a neural network model; The weight matrix of the computing power feature neural network after deduplication based on the target number is used to obtain a fixed part of the weight matrix of the first model, including: The weight matrices of the target number of deduplicated computing power feature neural networks are spliced ​​as the weight matrix of its own pre-trained neural network; the weight matrix of the pre-trained neural network is the fixed part in the weight matrix of the first model.

2. The computing power network node knowledge migration method according to claim 1 is characterized in that: The method further comprises: obtaining a fixed portion of the weight matrix of the first model based on the weight matrix of the deduplicated computing power feature neural network of the target number, and obtaining a fine-tuning portion of the weight matrix of the first model based on the weight matrix of the deduplicated computing power feature neural network of the target number and training samples, thereby obtaining the weight matrix of the first model; Concatenating the weight matrix of the first model and the weight matrices of each historical neural network to obtain a concatenated matrix; Performing singular value decomposition on the concatenated matrix to obtain a feature compression matrix corresponding to the weight matrix of the new computing power feature neural network; The concatenated matrix is ​​compressed based on the feature compression matrix to obtain a new weight matrix of the computing power feature neural network.

3. The computing power network node knowledge migration method according to claim 2, characterized in that: Also includes: Representing the weight matrix of each of the historical neural networks as the product of a reconstruction matrix and a mapping matrix; The reconstruction matrix is ​​generated based on a random seed; Correspondingly, the characteristic information of the weight matrix of the computing power feature neural network includes the characteristic compression matrix corresponding to the weight matrix of the computing power feature neural network, each of the random seeds and each of the mapping matrices.

4. A method for transferring knowledge of computing network nodes, characterized in that: include: Obtaining a weight matrix of a computing power feature neural network of each target edge node; Sending the target number of deduplicated weight matrices of the computing power feature neural network to each of the target edge nodes, so that each of the target edge nodes obtains a fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and performs training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain a fine-tuning part of the weight matrix of the first model, thereby obtaining the weight matrix of the first model; The weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node; The target edge node is an edge node that participates in obtaining the first model; the first model is a neural network model; The weight matrix of the computing power feature neural network after deduplication based on the target number is used to obtain a fixed part of the weight matrix of the first model, including: The weight matrices of the target number of deduplicated computing power feature neural networks are spliced ​​as the weight matrix of its own pre-trained neural network; the weight matrix of the pre-trained neural network is the fixed part in the weight matrix of the first model.

5. A computing power network node knowledge migration device, characterized in that: include: A sending module is used to send characteristic information of the weight matrix of the computing power feature neural network to the central node, so that the central node sends a target number of deduplicated weight matrices of the computing power feature neural network to each target edge node based on the characteristic information of the weight matrix of the computing power feature neural network sent by each target edge node; A receiving module, configured to receive the weight matrix of the target number of the computing power feature neural network after deduplication; a training module, configured to obtain a fixed portion of the weight matrix of the first model based on the target number of weight matrices of the computing power feature neural network after deduplication, and perform training based on the target number of weight matrices of the computing power feature neural network after deduplication and training samples to obtain a fine-tuning portion of the weight matrix of the first model, so as to obtain the weight matrix of the first model; The weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node; The computing power characteristic neural network is a characteristic neural network of a target edge node; the target edge node is an edge node that participates in obtaining the first model; the first model is a neural network model; The weight matrix of the computing power feature neural network after deduplication based on the target number is used to obtain a fixed part of the weight matrix of the first model, including: The weight matrices of the target number of deduplicated computing power feature neural networks are spliced ​​as the weight matrix of its own pre-trained neural network; the weight matrix of the pre-trained neural network is the fixed part in the weight matrix of the first model.

6. A computing power network node knowledge migration device, characterized in that: include: An acquisition module, configured to acquire a weight matrix of a computing power feature neural network of each target edge node; A sending module is used to send the target number of deduplicated weight matrices of the computing power feature neural network to each of the target edge nodes, so that each of the target edge nodes obtains a fixed part of the weight matrix of the first model based on the target number of deduplicated weight matrices of the computing power feature neural network, and performs training based on the target number of deduplicated weight matrices of the computing power feature neural network and training samples to obtain a fine-tuning part of the weight matrix of the first model, so as to obtain the weight matrix of the first model; The weight matrix of the computing power feature neural network of the target edge node is obtained by compressing the weight matrices of each historical neural network of the target edge node by the target edge node; The target edge node is an edge node that participates in obtaining the first model; the first model is a neural network model; The weight matrix of the computing power feature neural network after deduplication based on the target number is used to obtain a fixed part of the weight matrix of the first model, including: The weight matrices of the target number of deduplicated computing power feature neural networks are spliced ​​as the weight matrix of its own pre-trained neural network; the weight matrix of the pre-trained neural network is the fixed part in the weight matrix of the first model.

7. A computing power network node knowledge migration system, characterized in that: include: The computing power network node knowledge migration device as described in claim 6 and multiple computing power network node knowledge migration devices as described in claim 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the computing power network node knowledge migration method as described in any one of claims 1 to 4 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computing power network node knowledge migration method as described in any one of claims 1 to 4 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computing power network node knowledge migration method as described in any one of claims 1 to 4 is implemented.

Images