Device-edge-cloud collaborative training and task reasoning method for hybrid privacy data

Abstract

The present invention discloses an end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, which comprehensively considers the difference in computing power of end-edge-cloud nodes and the privacy constraints of samples to be trained, realizes the segmentation of front and back layers of the neural network model, and redistribution of samples to be trained; adopts the end-edge-cloud distributed collaborative training method, performs front-layer neural network representation reasoning and back-layer neural network parameter training for privacy data on the object-end device and edge server respectively, and performs full-layer neural network parameter training for public data on the cloud center node; based on the above model parameters, performs hierarchical convergence and weighted fusion of the neural network model, and uses the iteratively converged neural network model to complete the intelligent reasoning task of multimodal data on the object-end device. The present invention realizes end-edge-cloud collaborative neural network training and task reasoning that are highly adaptable to hybrid privacy data from three aspects: model segmentation and sample redistribution, end-edge-cloud distributed collaborative training, and model hierarchical convergence and fusion.

Description

Technical Field

[0001] The present invention belongs to the field of distributed machine learning, and specifically relates to an end-edge-cloud collaborative training and task reasoning method for hybrid privacy data. Background Art

[0002] Artificial neural network (ANN), also known as neural network (NN), is the core of deep learning algorithm, which consists of an input layer, one or more hidden layers and an output layer. Neural networks have the characteristics of adaptability, nonlinearity and parallel processing, and are widely used in machine learning, pattern recognition, data mining and other fields. They have been widely used in speech recognition, visual analysis and natural language understanding. With the advancement of deep learning technology research and application, on the one hand, due to the continuous expansion of parameter scale and sample scale, the demand for computing power for neural network model training is also growing; on the other hand, with the deep integration of various intelligent physical devices into people's production and life, the coupling between sample data and user privacy is also closer, and public data and private data often coexist. Therefore, in the process of real model training and task reasoning, the demand for privacy protection is also increasing.

[0003] The development of computing power infrastructure is currently evolving from cloud-based to ubiquitous, with computing power distribution extending from the center to the edge and end-side, presenting a three-level computing power architecture of end, edge, and cloud. In the field of machine learning, new paradigms such as distributed learning and federated learning are proposed and applied accordingly. Distributed machine learning distributes computing and data to multiple computing nodes, each of which holds a complete or partial data set, and updates and optimizes the model through message passing or parameter synchronization; however, driven by computing efficiency as the first demand, user data faces the risk of global visibility and privacy leakage. The core idea of ​​federated learning is to conduct distributed model training between multiple data sources with local data, without the need to exchange local data, and only build a global model by exchanging model parameters or intermediate results; however, in federated learning for data on end-devices, model training mainly relies on the computing power of the end-devices themselves. In actual application, it faces challenges such as limited computing resources and long training cycles.

[0004] During the operation of existing IoT devices, a large amount of multimodal data with training value is accumulated, such as video data collected by surveillance cameras, log data stored in mobile devices, and users' instant messaging and content creation data. These data usually present a mixture of public data and private data, and the overall data scale and computing power required for training are huge. Therefore, simply relying on data aggregation training on the cloud, local training on the IoT, or computing offloading methods for edge collaboration cannot simultaneously meet the needs of privacy protection, efficient training, and global model fusion. Summary of the invention

[0005] The purpose of the present invention is to provide an end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, which realizes end-edge-cloud collaborative neural network training and task reasoning that are highly adaptable to hybrid privacy data from three aspects: model segmentation and sample redistribution, end-edge-cloud distributed collaborative training, and model layered aggregation and fusion.

[0006] The technical solution to achieve the purpose of the present invention is: construct the topology of the object-side device, edge server, and cloud center node, and uniformly describe the composition and network connection relationship of the three types of nodes: the object-side device, the edge server, and the cloud center;

[0007] The full-layer neural network model is divided into a front-layer neural network and a back-layer neural network, which are deployed on the physical device node and the edge server node respectively, and the full-layer neural network model is deployed on the cloud center node;

[0008] The front-layer neural network, the back-layer neural network and the full-layer neural network model are collaboratively iterated and trained in the following manner until the iteration termination condition is met, and the full-layer neural network model that converges through iteration is used as the global neural network model;

[0009] The front-layer neural network is used on the physical device to represent and reason about the local privacy samples, and the intermediate calculation results are uploaded to the connected edge server.

[0010] The edge server uses the intermediate calculation results of the received privacy samples to train the parameters of the later neural network.

[0011] The cloud center uses public samples to train the parameters of the full-layer neural network model, and according to the number of samples participating in the training, the obtained parameters of the neural network model trained by the edge server and the cloud center are weighted to obtain the full-layer neural network parameters of this round of iteration;

[0012] The global neural network model is used to complete the intelligent reasoning task of multimodal data on the physical device.

[0013] Compared with the prior art, the present invention has the following significant advantages:

[0014] (1) The model training framework for hybrid privacy data designed in this invention breaks through the limitation of traditional model training methods that only consider computing efficiency or privacy protection, and achieves the simultaneous satisfaction of efficient utilization of computing resources and user data privacy protection, which is highly original.

[0015] (2) The end-edge-cloud collaborative neural network hierarchical training method proposed in the present invention can fully utilize the computing advantages of the three-level computing power infrastructure of the end, edge, and cloud, and realize efficient parallelism at both the data and model levels during the model training process, which is highly innovative.

[0016] (3) The model hierarchical convergence and fusion strategy adopted by the present invention can realize the construction of a unified optimization model for global sample data by exchanging parameters and sharing features of the training models on distributed computing nodes, avoiding the constraints of data island problems and having high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the end-edge-cloud collaborative training method of the present invention.

[0018] Figure 2 It is a schematic diagram of the end-edge cloud node topology of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0020] A device-edge-cloud collaborative training and task reasoning method for hybrid privacy data. The typical applicable scenario is to collect data based on widely distributed end-of-things devices. According to the privacy and computing power characteristics of computing nodes at all levels of the device-edge-cloud, large-scale distributed training of neural networks is realized, including but not limited to the following scenarios:

[0021] During the operation of existing object-side devices, a large amount of multimodal data with training value is accumulated, such as video data collected by surveillance cameras, log data stored in mobile devices, and users' instant messaging and content creation data. In view of the above scenarios, the end-edge-cloud collaborative training and task reasoning method for hybrid privacy data proposed in the present invention first redistributes public samples to the cloud based on the privacy of the sample data to be trained, and retains the privacy samples locally on the object; secondly, for privacy samples, after feature extraction and privacy protection at the object node, the intermediate results and subsequent training tasks are unloaded to the edge node, so as to save the computing power of the object as much as possible and utilize the edge computing power; then, for the public samples gathered from the widely existing object-side devices to the cloud, the powerful computing and storage capabilities on the cloud are used to train the model; finally, the models generated by each round of training of the edge node and the cloud node are weighted fused, and the fused new model is distributed to each computing node. After multiple rounds of iterations, the training of a large-scale full-layer neural network model is finally completed.

[0022] As shown in the figure, as an embodiment, a device-edge-cloud collaborative training and task reasoning method for hybrid privacy data includes the following steps:

[0023] Step S1, neural network model segmentation and sample redistribution:

[0024] The end-edge-cloud collaborative neural network training method proposed in the present invention comprehensively considers the computing power of the three types of nodes, namely, the physical end device, the edge server, and the cloud center, as well as the privacy constraints of the sample data to be trained. Therefore, in this step, firstly, the end-edge-cloud node topology is constructed to uniformly describe the composition and network connection relationship of the three types of nodes, namely, the end-edge-cloud; secondly, the neural network model is divided into two parts, the front-layer neural network with low computing power requirements and the back-layer neural network with high computing power requirements, through pre-training model segmentation; finally, since the computing power of the three types of nodes, namely, the end, the edge, and the cloud, is significantly enhanced in turn, and the privacy protection of the sample data of the physical end device is reduced in turn, the cloud-side redistribution method of the training samples is as follows: the public samples of all physical end devices are shared to the cloud center, and the privacy samples of each physical end device are stored locally on the physical end device.

[0025] Step S101: construct the end-edge cloud node topology:

[0026] The edge cloud nodes involved in the present invention include three types, namely cloud center nodes, edge server nodes and physical device nodes. The network topology between the three types of nodes is a hierarchical tree structure, and the nodes are connected through wired or wireless networks. Figure 2 As shown: the types and quantities of the three types of nodes are 1 cloud center node, n edge server nodes, and m1+m2+...+m device nodes. n where C is the cloud center node, E1, E2, ..., En is the edge server node connected to C, It is the physical device node connected to E1. is the physical device node connected to E2, and so on. For E n The connected physical device node.

[0027] Step S102, pre-training model segmentation:

[0028] Before the formal training begins, the cloud center node C accumulates and builds a model library containing a series of offline pre-trained neural networks based on historical data and tasks. The user selects the initial pre-trained model M that is similar to the current intelligent reasoning task from the model library. 0 , where intelligent reasoning tasks include but are not limited to prompt word association tasks for text data and target recognition tasks for image and video data. 0 The total number of network layers is k. According to the transferability characteristics of the front layer network of the neural network model, M 0 Split into front layer neural network With the back layer neural network in M 0 The first k1 layers of the network, where M 0 The k-k1 layer network after that, k1 is determined by the user according to the model M 0 Network feature settings. M 0 The network parameter vector of the first k1 layers, M 0 The network parameter vector of the last k-k1 layers. 0 M 0 The computing power requirement of the rear-layer neural network is higher than that of the front-layer neural network.

[0029] Step S103, redistribution of training samples in the cloud:

[0030] Each physical device T ij (Indicates that the edge server E i The set of samples to be trained on the jth physical device connected to the physical device is denoted as S ij , S ij By the physical device T ij Acquired during operation, including but not limited to multimodal data such as text, images, and videos. Classify samples according to whether they belong to private samples or public samples. ij Divide into two subsets and in Contains Sij All privacy samples in Contains S ij All public samples in the set, the relationship between the sets is and Due to its privacy, The original sample data in the object device T can only be ij Local storage, the intermediate results of the original samples after being processed by the front-layer neural network can be shared to the edge server E i ; Due to its openness, The original samples in can be shared to the cloud center C.

[0031] Considering the significant increase in computing power of the three types of nodes, namely, end, edge, and cloud, as well as the privacy requirements of samples, the cloud redistribution method for training samples is as follows: ① all public samples of the physical end devices are shared to the cloud center; ② the private samples of each physical end device are stored locally on the physical end device. After sample redistribution, the set of samples to be trained on the cloud center C is Remember S C The number of samples included is N C ; Each physical device T ij The set of samples to be trained on is remember The number of samples included is

[0032] Step S2: End-edge-cloud collaborative training of the neural network model:

[0033] The end-edge-cloud collaborative training of the neural network model fully utilizes the computing resources of the end, edge, and cloud to perform distributed training of the neural network. Specifically, the front-layer neural network is first used on the physical end device to represent and infer the local privacy samples after the redistribution in step S103, and the intermediate calculation results are uploaded to the connected edge server; secondly, the edge server uses the intermediate calculation results of the received privacy samples to train the parameters of the back-layer neural network; finally, the cloud center performs full-layer neural network parameter training on the public samples after the redistribution in step S103. In steps S2 and S3, L is denoted as the total number of iterations, which refers to the number of iterations to complete steps S201 to S302, and L is a positive integer that increases by 1 each round starting from 1.

[0034] Step S201: The front-layer neural network reasoning and protection for privacy data performed by the object end:

[0035] From step S102, it can be seen that each object terminal device T ij The set of samples to be trained on is for Each training sample in After the front layer of neural network (The network parameter vector is ) The intermediate calculation result output is vector x', then:

[0036]

[0037] Among them, the front layer neural network Network parameters is the neural network parameter of the first k1 layers obtained from the last iteration of the full-layer neural network, F a is the mapping function between the input s and the output intermediate calculation result x', which is composed of the front layer neural network To avoid the risk of privacy leakage when the intermediate calculation results are directly transmitted between the physical device and the edge server, the intermediate calculation result x' needs to be processed for privacy protection. Specifically, the intermediate calculation result x after privacy protection processing is:

[0038]

[0039] Where N(0,σ 2 ) is a Gaussian random noise vector with a mean of 0 and a standard deviation of σ. Its vector dimension is the same as x', and α, β, σ>0, which are parameters set by the user. Therefore, each object-side device T ij All the samples to be trained pass through the front layer of neural network The intermediate calculation result set of the output is:

[0040]

[0041] Each physical device T ij After completing the above calculations, The intermediate calculation results of this device are uploaded to the connected edge server E i Therefore, from step S101, it can be seen that the edge server E i With m i Therefore, the edge server E i The intermediate calculation result set obtained above is:

[0042]

[0043] Since each edge server E i With m i The physical devices are connected to the edge server E i The number of intermediate calculation results included is

[0044] Step S202: training of the back-layer neural network parameters for privacy data at the edge:

[0045] remember For edge server E i After L rounds of iterative training, the neural network model with k-k1 layers is obtained. for The network parameter vector of the first k1 layers, for The network parameter vector of the last k-k1 layers.

[0046] Since the edge server E i The network parameters of the first k1 layers are not trained, so When L = 1, When L>1, The parameters of the first k1 layers of the neural network obtained in the previous iteration of the full-layer neural network are obtained by step S301 in the previous iteration.

[0047] The following calculation method is used to obtain:

[0048]

[0049] Where η is the training step size set by the user, f E The loss function of the neural network model for edge training set by the user, f E The cross entropy loss function can be used but is not limited to: Denotes the loss function f E The gradient of Obtained through step S201, when L=1, Obtained by step S102; when L>1, The parameters of the neural network of the last k-k1 layers obtained in the previous iteration of the full-layer neural network are obtained by step S301 in the L-1th iteration.

[0050] Step S203: Full-layer neural network parameter training for public data on the cloud:

[0051] remember The cloud center node C is trained to obtain the full k-layer neural network model. The following calculation method is used to obtain the network parameters obtained by the cloud center through full-layer neural network parameter training on public samples:

[0052]

[0053] Among them, η is the training step size set by the user, is the network parameter of the full-layer neural network model obtained after the current round of training of the cloud center node, f C The loss function set by the user for the neural network model trained on the cloud, fC The cross entropy loss function can be used but is not limited to: Denotes the loss function f C The gradient of S C is the set of samples to be trained on the cloud center C obtained in step S103. When L=1, W L-1 =W 0 , obtained in step S102; when L>1, W L-1 is the neural network parameter of the whole layer in the previous iteration.

[0054] Will Split into and for The network parameter vector of the first k1 layers, for The network parameter vector of the last k-k1 layers is and It will be used for calculation in step S301.

[0055] Step S3, collaborative training model convergence generation and object-side reasoning:

[0056] This step generates the total model parameter vector of this round of iteration based on the neural network model parameters trained by the edge server and the cloud center, and determines whether to perform a new iteration based on the stop iteration condition. Specifically, firstly, the neural network model parameters trained by the edge server and the cloud center are weighted according to the number of samples involved in the training to obtain the front-layer network and back-layer network parameter vectors of the total model of this round of iteration; then the stop iteration condition is determined. If the distance between the total model parameter vectors before and after this round of iteration is less than the user-set value, or the number of iterations reaches the user-set value, the iteration is stopped and the final neural network model parameter vector is output. Otherwise, the iterative process of steps S2 to S3 is repeated until the stop iteration condition is met. Finally, the full-layer neural network model after iterative convergence is used to complete the intelligent reasoning task of multimodal data on the physical end device.

[0057] Step S301: Layered aggregation of the end-edge-cloud model:

[0058] Remember M L is the Lth round of full-layer neural network model after layered aggregation, M L The network structure and M L-1 Same as M L-1 Have the same number of network layers and the number of network parameters in each layer. M L The network parameter vector of the first k1 layers, M L The network parameter vector of the last k-k1 layers is Depend on and It is calculated by weighting the number of samples participating in the training. Depend on and The weighted calculation is based on the number of samples participating in the training. The weighting method can be, but is not limited to, the direct weighted average method. In the case of using the direct weighted average method, and Obtained by the following calculation method:

[0059]

[0060]

[0061] Among them, N C Obtained from step S103, N Ei Obtained from step S201, and Obtained by step S202 (it can be known from step S202 Step S301 in the L-1th iteration) and Obtained by step S203. Therefore, the above two equations can be further written as:

[0062]

[0063]

[0064] So far, we can get the total model M of the Lth round after stratified aggregation L The network parameter vector of the first k1 layers And the network parameter vector of the k-k1 layers The parameters of the full-layer neural network in the Lth round of iteration are

[0065] Step S302, global model convergence verification and iterative training:

[0066] Repeat steps S201 to S301 until the total model parameter vector distance d before and after the Lth round of iterations is L When it is less than the user-set value ε, the model is considered to have converged and the iteration is stopped. The vector distance can be but is not limited to the Euclidean distance. For the case of using the Euclidean distance to calculate the parameter vector, when the conditions are met:

[0067]

[0068] When , stop the iteration; otherwise, Steps S201 to S302 are repeated iteratively until d is satisfied.L <εorL=L max When, the iteration stops, where L max The maximum number of total iteration rounds set by the user. When this step determines that the conditions for stopping iteration are met, record this round L = L * , then the L * The iterative round This is the parameter vector of the final full-layer neural network model, denoted by M * Parameter vector The corresponding full-layer neural network model, namely M * is the full-layer neural network after iterative convergence, called M * It is a global neural network model, or simply called the global model.

[0069] Step S303: Intelligent reasoning on the object side for multimodal data

[0070] Global Model M * With the initial model M 0 The network model has the same structure and is therefore applicable to the same data types and task types. As can be seen from step S102, the intelligent reasoning task types include but are not limited to prompt word association tasks for text data and target recognition tasks for image and video data.

[0071] When the task type is prompt word association for text data, each object-side device T ij After deploying the global model M adapted to the task * After that, for the text text input by the user, which can be private or public data, the associated word nextText is obtained as follows:

[0072]

[0073] Where F is the mapping function between the input text text and the output associated word nextText, which is composed of the neural network M * The network structure and parameter vector Decide.

[0074] When the task type is an object recognition task for image and video data, each object-side device T ij After deploying the global model M adapted to the task * After that, for each frame of the image data or video data input by the user, the target recognition result object in the image or video frame can be obtained as:

[0075]

[0076] Where F is the mapping function between the input image or video frame image and the output target recognition result object, which is composed of the neural network M * The network structure and parameter vector Decide.

[0077] The present invention can rely on the three-level computing power infrastructure of end-edge-cloud to take into account both efficient utilization of computing power resources and protection of user data privacy during model training, meet the urgent needs of distributed training of large-scale neural networks under the end-edge-cloud architecture, and ultimately use the model to complete the intelligent reasoning task of multimodal data on the physical end device.

Claims

1. A device-edge-cloud collaborative training and task reasoning method for hybrid privacy data. It is characterized in that The specific steps include: Construct the topology of the physical end devices, edge servers, and cloud center nodes, and uniformly describe the composition and network connection relationship of the three types of nodes: physical end devices, edge servers, and cloud centers; The full-layer neural network model is divided into a front-layer neural network and a back-layer neural network, which are deployed on the physical device node and the edge server node respectively, and the full-layer neural network model is deployed on the cloud center node; The front-layer neural network, the back-layer neural network and the full-layer neural network model are collaboratively iteratively trained in the following manner until the full-layer neural network model meets the iteration termination condition, and the full-layer neural network model that has converged through iteration is used as the global neural network model for intelligent reasoning on the object side for multimodal data; The front-layer neural network is used on the physical end device to perform representation reasoning and privacy protection on the local privacy sample to obtain an intermediate calculation result, and the intermediate calculation result is uploaded to the connected edge server. The network parameters of the front-layer neural network are the front-layer neural network parameters obtained by the previous round of iteration of the full-layer neural network; The specific method for obtaining the intermediate calculation results is: The local privacy samples on each physical device are output as intermediate calculation result vectors through the front-layer neural network; Perform privacy protection processing on the intermediate calculation result vector to obtain the intermediate calculation result of the front-layer neural network; The edge server uses the intermediate calculation results of the received privacy samples to train the parameters of the back-layer neural network. 1 The layer network parameters are the kk obtained after the last iteration of the full-layer neural network 1 Layer neural network parameters, k is the total number of layers of the full-layer neural network, k 1 is the number of layers of the front neural network; The cloud center uses public samples to train the parameters of the full-layer neural network model, and according to the number of samples participating in the training, the obtained parameters of the neural network model trained by the edge server and the cloud center are weighted to obtain the full-layer neural network parameters of this round of iteration; The global neural network model is used to complete the intelligent reasoning task of multimodal data on the physical device.

2. According to claim 1, the end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, It is characterized in that The cloud center nodes, edge server nodes and physical end device nodes form a hierarchical tree structure, and the nodes are connected through a wired or wireless network; one cloud center node is connected to n edge server nodes, and one edge server node is connected to m physical end device nodes.

3. According to claim 1, the end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, It is characterized in that The multimodal data obtained during the operation of each physical device node is used as a set of samples to be trained; Divide the sample set to be trained into a private sample subset and a public sample subset according to the sample attributes; The private sample subset can only be stored locally on the physical device, and the public sample subset is shared to the cloud center.

4. According to claim 1, the end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, It is characterized in that The specific formula for privacy protection of the intermediate calculation result vector is: In the formula, x' is the intermediate calculation result vector output by the front layer neural network, is the first k obtained from the previous iteration of the full-layer neural network 1 Layer neural network parameters, L is the number of iterations, F a is the mapping function between the input s and the output intermediate calculation result x', N(0,σ 2 ) is a Gaussian random noise vector with mean 0 and standard deviation σ, α, β, σ>0, which are all set parameters.

5. According to claim 1, the end-edge-cloud collaborative training and task reasoning method for hybrid privacy data, It is characterized in that The back layer neural network has the following kk after this round of iteration 1 Layer network parameters for: Among them, η is the training step size set by the user, f E is the loss function of the neural network model set for edge training, k is the number of network layers of the full-layer neural network, Denotes the loss function f E The gradient of E i is the i-th edge server node, For the edge node E i The set of intermediate calculation results obtained by the front-layer neural network after the set of samples to be trained on the connected physical device is is the final kk obtained from the last iteration of the full-layer neural network 1 Layer neural network parameters, L is the number of iterations.

6. The device-edge-cloud collaborative training and task reasoning method for hybrid privacy data according to claim 1, It is characterized in that The network parameters obtained by training the full-layer neural network parameters of the public samples in the cloud center are: In the formula, is the network parameter of the full-layer neural network model obtained after this round of training of the cloud center node, W L-1 is the full-layer neural network parameter of the previous iteration, L is the number of iterations, η is the set training step size, Denotes the loss function f C The gradient of Can be split into the first k 1 Layer network parameters After kk 1 Layer network parameters S C is the set of samples to be trained on the cloud center C.

7. The device-edge-cloud collaborative training and task reasoning method for hybrid privacy data according to claim 1, It is characterized in that The parameters of the full-layer neural network in this round of iteration are: In the formula, N C is the number of samples contained in the sample set to be trained in the cloud center, For edge server E i The number of intermediate calculation results included, n is the number of edge nodes, is the first k layers of the full-layer neural network obtained by the cloud center node in this round of training 1 Layer network parameters, kk is the full-layer neural network obtained by the current round of training of the cloud center node 1 Layer network parameters, It is decomposed from the network parameters of the full-layer neural network model obtained after the current round of training of the cloud center node. is the first k layers of the full-layer neural network in the previous iteration 1 Layer network parameters, is the kk obtained by the back layer neural network after this round of iteration 1 Layer network parameters, is the first k layers of the full-layer neural network after this round of iteration. 1 Layer network parameters, is the final kk of the full-layer neural network after this round of iteration 1 Layer network parameters, L is the number of iterations.

8. The device-edge-cloud collaborative training and task reasoning method for hybrid privacy data according to claim 1, It is characterized in that When the distance between the parameter vectors of the full-layer neural network model before and after the Lth round of iteration is less than the set value ε, the model is considered to have converged and the iteration is stopped.

9. The device-edge-cloud collaborative training and task reasoning method for hybrid privacy data according to claim 1, It is characterized in that The parameter vector distance of the full-layer neural network model before and after this round of iteration is calculated using the Euclidean distance. The specific formula is: In the formula, is the first k layers of the full-layer neural network in this iteration 1 Layer neural network parameters, is the back kk of the full-layer neural network in this round of iteration 1 Layer neural network parameters, is the first k layers of the full-layer neural network in the previous iteration 1 Layer network parameters, is the back kk of the full-layer neural network in the previous iteration 1 Layer network parameters, L is the number of iterations; When L <εorL=L max When , the iteration stops; where L max is the maximum number of total iterations set by the user, and ε is the minimum distance between the models before and after iteration set by the user; when the iteration stop condition is not met, Iterate and calculate the next round Parameter value, is the kk obtained by the back layer neural network after this round of iteration 1 Layer network parameters, is the network parameter of the full-layer neural network model obtained after the current round of training of the cloud center node, and then the conditions for stopping iteration are judged; when the conditions for stopping iteration are reached, W L =[W a L ,W b L ] is the full-layer neural network parameter after iterative convergence.

Images