A data processing method, apparatus, device, and computer program product

Abstract

This application provides a data processing method, apparatus, device, and computer program product. The method includes: acquiring one or more second data points obtained by processing first data using one or more client models; deploying one or more client models on distributed nodes; mixing the one or more second data points to obtain one or more third data points; training a server-side model using the one or more second data points and the one or more third data points to obtain one or more first loss functions; determining one or more first weights corresponding to one or more client models based on the one or more first loss functions; using the one or more first weights to update the first parameters of the one or more client models to obtain updated one or more client models; and using the updated one or more client models to process the first data. This method can improve the performance of model training.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and computer program product. Background Technology

[0002] Split Learning (SL) splits the model on a single node in Federated Learning (FL) into a client model and a server model. The client model is deployed on distributed nodes, while the server model is deployed in the cloud. However, in edge environments, the number of distributed nodes often changes due to factors such as network link failures and recovery, and node energy depletion and replenishment. This leads to unstable training results for the client model, which in turn affects the training of the overall model, resulting in a lack of node scalability. Furthermore, since distributed nodes are deployed in fixed locations, the data collected by the nodes is usually heterogeneous. Using this heterogeneous data as input for split learning model training will affect the model training results. Summary of the Invention

[0003] This application provides a method, apparatus, computer-readable storage medium, and computer program product that can improve the performance of model training.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method applied to a cloud environment with a server-side model; the method includes: One or more pieces of second data are obtained by processing the first data using one or more client models; the one or more client models are deployed on distributed nodes. The one or more second data are mixed to obtain one or more third data; The server-side model is trained using the one or more second data and the one or more third data to obtain one or more first loss functions; One or more first weights corresponding to the one or more client models are determined according to the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client models to obtain the updated one or more client models, and the updated one or more client models are used to process the first data.

[0005] This application provides a data processing apparatus applied in a cloud environment with a server-side model; including: An acquisition unit is used to acquire one or more pieces of second data obtained by one or more client models processing the first data; the one or more client models are deployed on distributed nodes. A mixing processing unit is used to mix the one or more second data to obtain one or more third data; The training unit is used to train the server-side model using the one or more second data and the one or more third data to obtain one or more first loss functions; A determining unit is configured to determine one or more first weights corresponding to the one or more client models based on the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client models to obtain an updated one or more client models, and the updated one or more client models are used to process the first data.

[0006] This application provides an electronic device, including: a processor and a memory for storing a computer program that can run on the processor, wherein the processor, when running the computer program, implements the data processing method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: By acquiring one or more second data points obtained from processing first data using one or more client models deployed on distributed nodes, and then mixing these second data points to obtain one or more third data points, a server-side model is trained using these second and third data points to obtain one or more first loss functions. By mixing these second data points, the impact of changes in the number of nodes on model training performance can be mitigated. Furthermore, one or more first weights corresponding to one or more client models are determined based on these first loss functions, and the first parameters of one or more client models are updated based on these first weights to obtain updated client models. By determining the weights corresponding to each client model, the impact of heterogeneous data on model training performance can be addressed, thereby improving model training performance. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 A schematic diagram of an exemplary distributed training architecture provided for an embodiment of this application; Figure 3 A schematic diagram illustrating an exemplary model training process provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] Edge computing is rapidly developing, and edge devices are generating increasingly more data. Deep learning models, which typically contain multiple layers and can quickly find sufficient features from large amounts of data, have been applied to edge computing-related tasks. Transmitting the massive amounts of data generated by edge devices to a centralized location for deep learning model training introduces data transmission latency and data privacy issues. Edge distributed nodes already possess considerable computing power, enabling the deployment of deep learning models and distributed training.

[0014] FL (Flexible Learning) is an emerging distributed training method. FL consists of multiple distributed nodes and a central server. Nodes train models using local data, and the central server updates global model parameters using parameters uploaded from the distributed nodes. FL avoids the transmission latency and data privacy issues associated with raw data in deep learning model training. However, FL requires placing a deep learning model on each node and training it. This typically places high demands on the storage and computational performance of individual nodes. SL (Splitting Learning) was proposed as an improved distributed training method for FL. SL splits the model on a single node in FL into client-side and server-side models. Split learning achieves secure collaboration by splitting the model, eliminating the need to share raw data, and only sharing intermediate computation results or model parameters. The split layers are placed as close as possible to the input layer. The client-side model, the less computationally demanding part of the deep learning model, is deployed on distributed nodes; the server-side model, the more computationally demanding part, is deployed in the cloud. In SL, the client-side model on a node uses local data as input to generate the representation of the split layer, i.e., shredded data. The node then sends this shredded data to the cloud, where the server-side model continues training on the shredded data to generate global model update parameters. On the one hand, SL offloads most of the computation to the cloud, effectively reducing the storage and computational performance requirements of a single distributed node for training deep learning models; on the other hand, SL only transmits the intermediate representation of the model without involving the original data, further reducing data transmission latency and the risk of privacy leaks.

[0015] In edge environments, the number of distributed nodes frequently changes due to factors such as network link failures and recovery, and node energy depletion and replenishment. Since the training of the server-side model in SL (Solving Skills) depends on the training of the client-side model, changes in the number of nodes can cause instability in the training results of the client-side models, thus affecting the overall model training—meaning SL's node scalability is insufficient. Currently, by combining FL (Flexible Learning) with SL, after SL backpropagation, FL can be applied across clients to mitigate the impact of client number changes on the server-side through aggregation. Furthermore, after SL backpropagation, local models can be trained in parallel on the client side to address the impact of client number changes on the overall model training. However, these methods do not consider the client-side model separation problem during SL forward propagation, resulting in relatively weak scalability.

[0016] Furthermore, because distributed nodes are deployed in fixed locations, the data collected by these nodes is often heterogeneous. Using this heterogeneous data as input for deep learning model training significantly impacts the training results of the SL model, i.e., the generalization ability of SL. Data-based methods can be used to reduce the statistical variability between client datasets, but this may introduce privacy risks. Model-based methods can also be used to customize personalized model structures or parameters for different clients, but the scale used to describe the similarity between models may be inaccurate, leading to suboptimal model performance.

[0017] Based on this, embodiments of this application provide a data processing method. Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, this method is applied to a cloud environment with a server-side model; the method includes: S101. Obtain one or more second data obtained by processing the first data using one or more client models.

[0018] One or more client models are deployed on distributed nodes.

[0019] It should be noted that the server-side model set up in the cloud is a model with high computational requirements; the client-side model deployed on the distributed nodes is a model with low computational requirements; the number of client-side models can be one or more, and the specific number can be determined according to the actual situation, and is not limited here.

[0020] It should be noted that the first data can be understood as the input data of each client model, or as the data collected by distributed nodes. The specific content of the first data can be determined according to the actual situation and is not limited here. Each client model has its corresponding first data. The second data can be understood as shredded data, that is, the shredded data generated by the client model training from the first data to the slicing layer. Each client model can obtain one or more second data from the first data through data processing, and the specific number can be determined according to the actual situation and is not limited here. Obtaining one or more second data obtained by one or more client models from the first data through data processing can be understood as obtaining the first data corresponding to each client model, and training the first data through each client model to generate the second data of the slicing layer. That is, each client model processes the corresponding first data to obtain the corresponding second data.

[0021] S102. Mix one or more second data to obtain one or more third data.

[0022] It should be noted that the third data can be understood as mixed and pulverized data. The number of third data can be one or more, and the specific number can be determined based on the actual situation; no limit is set here. Specifically, when there are multiple client models, each client model also corresponds to multiple third data. Mixing one or more second data to obtain one or more third data can be understood as mixing the second data of one client model with the second data of each of the other client models to obtain one or more third data (mixed and pulverized data) corresponding to that client model.

[0023] S103. Train the server-side model using one or more second data and one or more third data to obtain one or more first loss functions.

[0024] It should be noted that the first loss function can be understood as the loss function corresponding to each client model on the server. Training the server-side model using one or more second data points and one or more third data points to obtain one or more first loss functions can be understood as training the server-side model using one or more shredded data points and one or more mixed shredded data points corresponding to each client model to obtain the first loss function corresponding to that client model.

[0025] S104. Determine one or more first weights corresponding to one or more client models based on one or more first loss functions; the one or more first weights are used to update the first parameters of one or more client models to obtain updated one or more client models, and the updated one or more client models are used to process the first data.

[0026] It should be noted that the first weight can be understood as the weight matrix assigned to each client model, and the first parameter can be understood as the model parameters of the client model. The model parameters can be one or more layers; the specific number of layers can be determined based on the actual situation and is not limited here. One or more first weights are determined for one or more client models based on one or more first loss functions. These one or more first weights are used to update the first parameters of one or more client models, resulting in updated client models. The updated client models are then used to process the first data. This can be understood as follows: the first weights corresponding to each client model are determined based on the first loss function; the first weights are used to update the parameters of each layer of the corresponding client model, resulting in updated first parameters, i.e., the updated client models; and the updated client models are then used to process the first data. Each updated client model can also process other acquired data; the content of the data can be determined based on the actual situation and is not limited here.

[0027] By acquiring one or more second data points obtained from processing first data using one or more client models deployed on distributed nodes, and then mixing these second data points to obtain one or more third data points, a server-side model is trained using these second and third data points to obtain one or more first loss functions. By mixing these second data points, the impact of changes in the number of nodes on model training performance can be mitigated. Furthermore, one or more first weights corresponding to one or more client models are determined based on these first loss functions, and the first parameters of one or more client models are updated based on these first weights to obtain updated client models. By determining the weights corresponding to each client model, the impact of heterogeneous data on model training performance can be addressed, thereby improving model training performance.

[0028] In this embodiment of the application, the process of mixing one or more second data to obtain one or more third data specifically includes: mixing the second data corresponding to the first client model with the second data corresponding to each of the one or more second client models through linear interpolation to obtain one or more third data corresponding to the first client model; the first client model is any one of the one or more client models; the one or more second client models are other client models besides the first client model among the one or more client models.

[0029] It should be noted that the first client model is any one of the one or more client models; the second client model is any other client model besides the first client model; the number of second client models can be one or more, and the specific number can be determined according to the actual situation, without limitation here. The second data corresponding to the first client model is mixed with the second data corresponding to each of the one or more second client models through linear interpolation to obtain one or more third data corresponding to the first client model. This can be understood as mixing the second data corresponding to the first client model with the second data corresponding to each of the other second client models through linear interpolation to obtain one or more third data (mixed and shredded data) corresponding to the first client model.

[0030] For ease of understanding, here's an example, considering only two clients. and Once all the shredded data and its corresponding tags are uploaded to the server, the server will then process the data by mixing the shredded data with the corresponding tags. The first client's The first crushing data and the first The first client's Each grinding data point generates mixed grinding data. The method in this application embodiment can generate multiple mixed shredded data by mixing the shredded data of each client model, which effectively increases the model's sensitivity to noise and changes.

[0031] In this embodiment of the application, the process of training a server-side model using one or more second data and one or more third data to obtain one or more first loss functions specifically includes: training a server-side model using one or more second data to obtain one or more second loss functions; training a server-side model using one or more third data to obtain one or more third loss functions; and determining one or more first loss functions based on one or more second loss functions and one or more third loss functions.

[0032] It should be noted that the second loss function can be understood as the loss function obtained by training the server-side model using the second data (shredded data) corresponding to each client model; the third loss function can be understood as the loss function obtained by training the server-side model using the third data (mixed shredded data) corresponding to each client model. Training the server-side model using one or more second data sets yields one or more second loss functions; training the server-side model using one or more third data sets yields one or more third loss functions. This can be understood as training the server-side model using the second data (shredded data) corresponding to each client model to obtain the second loss function, and then training the server-side model using the third data (mixed shredded data) corresponding to each client model to obtain the third loss function. Determining one or more first loss functions based on one or more second loss functions and one or more third loss functions can be understood as determining the first loss function corresponding to the corresponding client model based on the second and third loss functions corresponding to each client model. As an example, in practical applications, the second loss function corresponding to client i can be denoted as... The third loss function corresponding to client i can be denoted as: The first loss function corresponding to client i can be denoted as: The first loss function for each client model can be determined by using the second and third loss functions for each client model. To express.

[0033] In this embodiment of the application, the process of training a server-side model using one or more third data to obtain one or more third loss functions specifically includes: obtaining one or more label data obtained by processing the first data using one or more client-side models; training the server-side model using one or more third data and one or more label data to obtain one or more fourth loss functions; and determining one or more third loss functions based on one or more fourth loss functions and one or more third data.

[0034] It should be noted that the number of labeled data corresponding to each client model can be one or more, and the specific number can be determined according to the actual situation, without limitation here. One or more labeled data are obtained by processing the first data using one or more client models. The labeled data can also be understood as the labels corresponding to the second data (shattered data). The fourth loss function can be understood as the loss function corresponding to each mixed shattered data. One or more fourth loss functions are obtained by training the server-side model using one or more third data and one or more labeled data. This can be understood as training the server-side model using the third data (mixed shattered data) and labeled data corresponding to each client model, using the mixed shattered data as the input to the server-side model and the labeled data as the corresponding expected output, to obtain the fourth loss function corresponding to each mixed shattered data. One or more third loss functions are determined based on one or more fourth loss functions and one or more third data. This can be understood as determining the third loss function based on the fourth loss function corresponding to each mixed shattered data and all the mixed shattered data corresponding to each client model. In practical applications, as an example, consider only two clients. and The server side uses a hybrid approach. The first client's The first crushing data and the first The first client's The first pulverization data is used to generate mixed pulverization data, and the first pulverization data is generated by mixing the second pulverization data. The first client's The first tag and the first The first client's Each tag generates a mixed tag, utilizing the mixed and shredded data. and mixed tags Training is performed on the server side. As input to the server-side model, As the corresponding expected output, the loss function for each mixed-pulverized data is denoted as: (i.e., the fourth loss function).

[0035] The solution in this application embodiment mixes the fragmented data generated by the client model, which can solve the problem of non-scalability in edge environments where the number of nodes is limited by split learning.

[0036] In this embodiment of the application, the process of determining one or more first weights corresponding to one or more client models based on one or more first loss functions specifically includes: obtaining one or more first vectors and one or more second parameters; the first vectors, second parameters, and client models are in one-to-one correspondence; the second parameters are hypernetwork (HN) parameters; determining one or more second weights corresponding to one or more client models based on one or more first vectors and one or more second parameters; and updating one or more second weights corresponding to one or more client models according to one or more first loss functions to obtain one or more first weights.

[0037] It should be noted that the first vector can be understood as the initial embedding vector corresponding to each client model; the second parameter can be understood as the initial HNs parameters; each client model has a corresponding first vector and second parameter, and the specific values ​​of the first vector and second parameter for each client model are randomly generated and determined according to the actual situation, without being limited here. As an example, the multiple first vectors of multiple client models can be denoted as... V represents the set of all first vectors, and N represents the number of client models. This represents the first vector corresponding to client model 1; multiple HNs parameters can be denoted as... , This represents the set of all HNs parameters, where N represents the number of client models. This represents the second parameter corresponding to client model 1.

[0038] It should be noted that the second weight can be understood as the initial weight corresponding to each client model. Determining one or more second weights corresponding to one or more client models based on one or more first vectors and one or more second parameters can be understood as determining the second weight (initial weight) corresponding to each client model through the hypernetwork based on the first vector and second parameters corresponding to each client model. As an example, the second weight of client i can be determined specifically through... To represent; among which, This represents the second weight of client i. It is a client Embedded vector, It is a client HNs parameters, For the client The corresponding hypernetwork.

[0039] In this embodiment, when the second weight is the initial weight, the first weight can be understood as the weight obtained after updating the second weight. The process of updating one or more second weights corresponding to one or more client models according to one or more first loss functions to obtain one or more first weights specifically includes: obtaining the first loss function and second weights corresponding to the first client model; updating the first parameters of each layer in the first client model based on the second weights corresponding to the first client model to obtain updated first parameters; determining the fifth loss function corresponding to the first client model based on the updated first parameters; and updating the second weights corresponding to the first client model based on the fifth loss function to obtain the first weights corresponding to the first client model.

[0040] It should be noted that the first client model can be any one of one or more client models. The first parameter can be understood as the parameter of each layer in client i. Updating the first parameter of each layer in the first client model based on the second weight corresponding to the first client model yields the updated first parameter. This can be understood as weighting the parameters of each layer in the first client model using the second weight corresponding to the first client model, resulting in the weighted first parameter (model parameter). Specifically, this can be achieved by multiplying the second weight corresponding to the first client model with the parameters of each layer in the first client model, resulting in the calculated parameters for each layer. The fifth loss function can be understood as the loss function obtained after updating the first parameter in the first client model; determining the fifth loss function corresponding to the first client model based on the updated first parameter can be understood as determining the fifth loss function based on the updated model parameters of the first client model. As an example, in practical applications, the fifth loss function can be denoted as... ;in, This represents the first parameter after the update. It consists of the mixed and pulverized data after mixing and the loss term corresponding to the mixing label; It is by The loss function corresponding to the weighting ratio coefficients.

[0041] In this embodiment of the application, the process of updating the second weight corresponding to the first client model based on the fifth loss function to obtain the first weight corresponding to the first client model specifically includes: determining the first gradient information corresponding to the first vector corresponding to the first client model and the second gradient information corresponding to the second parameter corresponding to the first client model based on the fifth loss function; updating the second weight corresponding to the first client model according to the first gradient information and the second gradient information to obtain the first weight corresponding to the first client model.

[0042] It should be noted that the first gradient information can be understood as the gradient information corresponding to the first vector; the second gradient information can be understood as the gradient information corresponding to the second parameter. As an example, in practical applications, the first gradient information can be denoted as... The second gradient information can be denoted as: Based on the fifth loss function, the first gradient information corresponding to the first vector of the first client model and the second gradient information corresponding to the second parameters of the first client model are determined. The second weights of the first client model are updated according to the first and second gradient information to obtain the first weights of the first client model. This can be understood as follows: based on the fifth loss function, the first gradient information corresponding to the first vector (embedding vector) and the second gradient information corresponding to the second parameters (HNs parameters) of the first client model are determined respectively; the first vector is updated according to the first gradient information; the second parameters are updated according to the second gradient information; and the second weights are updated based on the updated first vector and the updated second parameters to obtain the first weights.

[0043] The solution in this application uses HNs to train the server-side model parameters to obtain the weight allocation coefficients, thus solving the problem that the heterogeneity of edge node data is limited by the generalization of SL.

[0044] To facilitate understanding, an example of the above scheme is provided here. First, a split distributed training architecture is constructed, dividing the model into two parts: a client-side model with lower computational requirements and a server-side model with higher computational requirements, which are placed on distributed nodes and in the cloud, respectively. Second, a Mixup-based data shredding method is proposed, which mixes the shredded data of the model layers uploaded by the client to the server, and uses the mixed shredded data for server-side model training as a regularization term in the model's objective function, mitigating the impact of varying node numbers on model training performance. Next, a model weight allocation algorithm based on Hierarchical Neural Networks (HNs) is proposed. During the training of the server-side model using the shredded data from each client and the mixed shredded data, HNs is used to assign weights to the output weight matrix of each client, evaluating the importance of different clients and mitigating the impact of heterogeneous node data on model generalization performance.

[0045] The distributed training of the model in this application mainly includes three parts: a split distributed training architecture, a Mixup-based data shredding and mixing algorithm, and a HNs-based model weight allocation algorithm.

[0046] For ease of understanding, Figure 2 This application provides an exemplary schematic diagram of a distributed training architecture; as shown in the embodiments of this application. Figure 2As shown, this includes N clients and one server. The N clients are deployed on corresponding distributed nodes, and the server is deployed in the cloud. The clients can be understood as a client-side model, and the server as a server-side model.

[0047] In an edge environment, there are N distributed nodes and a cloud. Models requiring less computational power, such as the input layer and a single-layer reservoir, are deployed as client-side models on the distributed nodes. Models requiring more computational power, such as the multi-layer reservoir and the output layer, are deployed as server-side models in the cloud.

[0048] Mixup is a technique for enhancing the performance of deep learning models. It generates new samples and labels by performing linear interpolation between data and labels, where the new samples are a weighted average of the original samples, and the new labels are a weighted average of the original labels. Mixup helps reduce the model's sensitivity to noise and variations. Therefore, this embodiment utilizes Mixup to mix the shredded data and corresponding labels on each distributed node, obtaining new mixed shredded data and mixed labels, thus addressing the scalability issue of deep learning models caused by changes in the number of nodes in edge environments.

[0049] HNNs are deep neural networks that dynamically output the weights of input parameters during training tasks. HNNs maintain a personalized model for each client on the server side. Furthermore, HNNs do not require transmitting redundant parameters, avoiding additional communication costs. Therefore, this embodiment utilizes HNNs to train the server-side model parameters. During the training of each client model, the output weight matrices of all clients are combined to obtain weight allocation coefficients. By assigning these weight allocation coefficients to the client's output weight matrix, the impact of heterogeneous node data on model training performance is addressed.

[0050] The distributed training method for models based on improved split learning is introduced below. First, let the input data be... The input data is used to train the client-side model, generating fragmentation data for the cutting layer. ,in The activation function is used; after obtaining the shredded data from the cutting layer, it is uploaded to the server. On the server side, firstly, after receiving the shredded data from the client, the server uses Mixup to mix the shredded data and its labels using linear interpolation, resulting in new mixed shredded data and mixed labels. Then, the shredded data and mixed shredded data are used as input to the server-side model for training. During training, the embedding vectors are used as input to the HNs to evaluate the parameter weights of each layer in each model, assigning higher weight allocation coefficients to parameters of model layers with higher relevance. Finally, the weight allocation coefficients are weighted with the model parameters to obtain the personalized model parameters corresponding to each client.

[0051] For ease of understanding, Figure 3 This application provides an exemplary flowchart of model training; acquiring shredded data. and the mixing and pulverizing data obtained through Mixup. as well as ;in, , ;Will , as well as The embedding vector of client i is obtained by training the server-side model using a hypernetwork (HNs). and hyperparameters and through Determine the client's weight matrix New model parameters are obtained by weighting the parameters of each layer of client i. The specific weighting method can be obtained through Where l is the number of layers, These are the intermediate parameters for all layers of client i after training.

[0052] The specific process of the data shredding and mixing method based on Mixup is as follows. First, consider only two clients. and Once all the shredded data and its corresponding tags are uploaded to the server, the server will then process the data by mixing the shredded data with the corresponding tags. The first client's The first crushing data and the first The first client's The first pulverization data is used to generate mixed pulverization data, and the first pulverization data is generated by mixing the second pulverization data. The first client's The first tag and the first The first client's This method generates mixed labels from individual labels. It can achieve feature mixing in the sample space, producing new samples with a certain degree of diversity, effectively increasing the model's sensitivity to noise and variation. The mixing formula can be obtained through... and Let it be represented by; where the probability parameter .when hour, It follows a uniform distribution.

[0053] After the pulverized data is mixed, the mixed pulverized data is used and mixed tags Training is performed on the server side. As input to the server-side model, As the corresponding expected output, the loss function for each mixed-pulverized data is denoted as: Similarly, consider only two clients. and , No. The and the first The loss function of the mixed and pulverized data generated by the mixing of multiple clients is... .in, The size of the training data for the client. and This represents the training data set.

[0054] In consideration In the case of multiple clients, each shredded data is mixed with shredded data from other clients, thus each client's shredded data can generate... A loss function. Let... To remove Sampling without replacement in all nodes except the node The subset obtained from the elements. The loss function for the mixed pulverization data of each client can be denoted as: Then, this loss function is used as a regularization term in the final objective loss function to improve the node scalability of SL. The objective loss function is denoted as... .in, The loss function used to continue training on the server side after shredding the data.

[0055] The weight allocation method based on HNs is as follows. HNs consist of several fully connected layers, and the parameters of the HNs are... The input to HNs is an embedding vector that is automatically updated along with the model parameters. The output is a weight matrix. By maintaining a personalized model for each client on the server side, and leveraging hierarchical user similarity, clients with similar data distributions should have higher aggregation weights to enhance their mutual contribution. The weight matrix of each client is defined as follows: ;in, It is a client Embedded vector, It is a client HNs parameters, For the client The corresponding hypernetwork.

[0056] The server receives data from the client. Uploaded shredded data Used as input for server-side model training. Assume... It is a trained client. The corresponding model parameters, based on right The parameters of each layer are weighted to obtain new model parameters. It can be done To represent; among which, The number of layers in a deep learning model. For all layers of client i, the intermediate parameters are obtained after training. { }. Then the corresponding client on the server side. The loss function is: ,in, It consists of the mixed and pulverized data after Mixup mixing and the loss term corresponding to the mixing label. It is by The loss function corresponding to the assigned weight allocation coefficients. The weight allocation coefficients are calculated using HNs, by applying them to each client. By considering the weights of the output weight matrices of all clients, the impact of data heterogeneity on DESN training is mitigated.

[0057] Next, we will introduce and The update rules are derived from the above formula. and The corresponding gradients are as follows: ; .

[0058] in, This can be obtained from the local training of client i in each update. Because... and yes exist and The gradient in the direction can therefore be updated using the following general method. and : ; .

[0059] Based on the above formula, updates can be made in each communication round. and Then update the weight matrix. .

[0060] For ease of understanding, the algorithm flow is illustrated below.

[0061] Deployment: N distributed nodes and one cloud. Client-side models requiring less computing power are deployed on the distributed nodes, while server-side models requiring more computing power are deployed in the cloud.

[0062] Input: Input data Expected output ; Output: Weight matrix .

[0063] Initialization: Initialize client and server, and network parameters. and embedding vector ; Client: for do; Each client Execute client-side model training to obtain shredded data; Upload the shredded data and tags to the server.

[0064] Server-side: for do; ; ; according to renew ; renew ; end for; : for do; Perform a Mixup-based data mixing method to mix the shredded data and labels; Continue server-side model training using the mixed and pulverized data; The server-side model training continues using the shredded data, and the mixed shredded data is used as a regularization term to obtain the final target loss function, which is then calculated. ; ; return .

[0065] The solution in this application embodiment mixes feature information in the sample space through Mixup, generating new samples with greater diversity, thus solving the problem that the number of nodes in the edge environment is limited by the non-scalability of SL nodes; and it uses HNs to train the server-side model parameters to obtain weight ratio coefficients, thus solving the problem that the heterogeneity of edge node data is limited by the generalization of SL.

[0066] This application provides a data processing apparatus. Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; as shown below. Figure 4 As shown, it is applied to a cloud environment with a server-side model; the data processing device 400 includes: The acquisition unit 401 is used to acquire one or more pieces of second data obtained by one or more client models processing the first data; the one or more client models are deployed on distributed nodes. The mixing processing unit 402 is used to mix the one or more second data to obtain one or more third data. Training unit 403 is used to train the server-side model using the one or more second data and the one or more third data to obtain one or more first loss functions; The determining unit 404 is used to determine one or more first weights corresponding to the one or more client models according to the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client models to obtain an updated one or more client models, and the updated one or more client models are used to perform data processing on the first data.

[0067] In some embodiments, the mixing processing unit 402 is further configured to perform a mixing process on the second data corresponding to the first client model and the second data corresponding to each of the one or more second client models through linear interpolation to obtain the one or more third data corresponding to the first client model; the first client model is any one of the one or more client models; the one or more second client models are other client models besides the first client model among the one or more client models.

[0068] In some embodiments, the training unit 403 is further configured to train the server-side model using the one or more second data to obtain one or more second loss functions; train the server-side model using the one or more third data to obtain one or more third loss functions; and determine the one or more first loss functions based on the one or more second loss functions and the one or more third loss functions.

[0069] In some embodiments, the training unit 403 is further configured to acquire one or more label data obtained by the one or more client models processing the first data; train the server-side model using the one or more third data and the one or more label data to obtain one or more fourth loss functions; and determine the one or more third loss functions based on the one or more fourth loss functions and the one or more third data.

[0070] In some embodiments, the determining unit 404 is further configured to obtain one or more first vectors and one or more second parameters; the first vectors and the second parameters correspond one-to-one with the client model; the second parameters are hypernetwork HNs parameters; determine one or more second weights corresponding to the one or more client models based on the one or more first vectors and the one or more second parameters; update the one or more second weights corresponding to the one or more client models according to the one or more first loss functions to obtain the one or more first weights.

[0071] In some embodiments, the determining unit 404 is further configured to obtain a first loss function and a second weight corresponding to the first client model; update the first parameter of each layer in the first client model based on the second weight corresponding to the first client model to obtain an updated first parameter; determine a fifth loss function corresponding to the first client model based on the updated first parameter; and update the second weight corresponding to the first client model based on the fifth loss function to obtain the first weight corresponding to the first client model.

[0072] In some embodiments, the determining unit 404 is further configured to determine, based on the fifth loss function, the first gradient information corresponding to the first vector of the first client model and the second gradient information corresponding to the second parameter of the first client model; and update the second weight of the first client model according to the first gradient information and the second gradient information to obtain the first weight of the first client model.

[0073] This application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 503. Optionally, the electronic device 500 may also include a communication bus 502.

[0074] In specific embodiments, the processor 501 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0075] In this embodiment, the communication bus 502 is used to establish a connection and communication between the processor 501 and the memory 503; when the processor 501 executes the running program stored in the memory 503, it implements the following data processing method: One or more second data points are obtained by processing first data using one or more client-side models; the one or more client-side models are deployed on distributed nodes; the one or more second data points are mixed to obtain one or more third data points; the server-side models are trained using the one or more second data points and the one or more third data points to obtain one or more first loss functions; one or more first weights corresponding to the one or more client-side models are determined based on the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client-side models to obtain updated one or more client-side models, and the updated one or more client-side models are used to process the first data.

[0076] Furthermore, the processor 501 is also configured to perform linear interpolation to mix the second data corresponding to the first client model with the second data corresponding to each of the one or more second client models to obtain the one or more third data corresponding to the first client model; the first client model is any one of the one or more client models; the one or more second client models are other client models besides the first client model among the one or more client models.

[0077] Furthermore, the processor 501 is also configured to train the server-side model using the one or more second data to obtain one or more second loss functions; train the server-side model using the one or more third data to obtain one or more third loss functions; and determine the one or more first loss functions based on the one or more second loss functions and the one or more third loss functions.

[0078] Furthermore, the processor 501 is also configured to acquire one or more label data obtained by the one or more client models processing the first data; train the server-side model using the one or more third data and the one or more label data to obtain one or more fourth loss functions; and determine the one or more third loss functions based on the one or more fourth loss functions and the one or more third data.

[0079] Furthermore, the processor 501 is also configured to obtain one or more first vectors and one or more second parameters; the first vectors and the second parameters correspond one-to-one with the client model; the second parameters are HNs parameters; determine one or more second weights corresponding to the one or more client models based on the one or more first vectors and the one or more second parameters; update the one or more second weights corresponding to the one or more client models according to the one or more first loss functions to obtain the one or more first weights.

[0080] Furthermore, the processor 501 is also used to obtain a first loss function and a second weight corresponding to the first client model; update the first parameter of each layer in the first client model based on the second weight corresponding to the first client model to obtain the updated first parameter; determine a fifth loss function corresponding to the first client model based on the updated first parameter; and update the second weight corresponding to the first client model based on the fifth loss function to obtain the first weight corresponding to the first client model.

[0081] Furthermore, the processor 501 is also configured to determine the first gradient information corresponding to the first vector of the first client model and the second gradient information corresponding to the second parameter of the first client model based on the fifth loss function; and update the second weight of the first client model according to the first gradient information and the second gradient information to obtain the first weight of the first client model.

[0082] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the data processing method described above.

[0083] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the data processing method described above.

[0084] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0086] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applied to a cloud environment with a server-side model configured; the method includes: One or more pieces of second data are obtained by processing the first data using one or more client models; the one or more client models are deployed on distributed nodes. The one or more second data are mixed to obtain one or more third data; The server-side model is trained using the one or more second data and the one or more third data to obtain one or more first loss functions; One or more first weights corresponding to the one or more client models are determined according to the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client models to obtain the updated one or more client models, and the updated one or more client models are used to process the first data.

2. The method according to claim 1, characterized in that, The process of mixing the one or more second data to obtain one or more third data includes: The second data corresponding to the first client model is mixed with the second data corresponding to each of the one or more second client models through linear interpolation to obtain the one or more third data corresponding to the first client model; the first client model is any one of the one or more client models; the one or more second client models are other client models besides the first client model among the one or more client models.

3. The method according to claim 1, characterized in that, The server-side model is trained using the one or more second data and the one or more third data to obtain one or more first loss functions, including: The server-side model is trained using the one or more second data to obtain one or more second loss functions; The server-side model is trained using the one or more third data to obtain one or more third loss functions; The one or more first loss functions are determined based on the one or more second loss functions and the one or more third loss functions.

4. The method according to claim 3, characterized in that, The step of training the server-side model using the one or more third data to obtain one or more third loss functions includes: Obtain one or more tag data obtained by the one or more client models processing the first data; The server-side model is trained using the one or more third data and the one or more label data to obtain one or more fourth loss functions; The one or more third loss functions are determined based on the one or more fourth loss functions and the one or more third data.

5. The method according to claim 1, characterized in that, Determining one or more first weights corresponding to the one or more client models based on the one or more first loss functions includes: Obtain one or more first vectors and one or more second parameters; the first vectors and the second parameters correspond one-to-one with the client model; the second parameters are hypernetwork (HNs) parameters. Based on the one or more first vectors and the one or more second parameters, determine one or more second weights corresponding to the one or more client models; The one or more second weights corresponding to the one or more client models are updated according to the one or more first loss functions to obtain the one or more first weights.

6. The method according to claim 5, characterized in that, The step of updating one or more second weights corresponding to the one or more client models according to the one or more first loss functions to obtain the one or more first weights includes: Obtain the first loss function and second weights corresponding to the first client model; Based on the second weight corresponding to the first client model, the first parameter of each layer in the first client model is updated to obtain the updated first parameter. The fifth loss function corresponding to the first client model is determined based on the updated first parameter; The second weights corresponding to the first client model are updated based on the fifth loss function to obtain the first weights corresponding to the first client model.

7. The method according to claim 6, characterized in that, The step of updating the second weights corresponding to the first client model based on the fifth loss function to obtain the first weights corresponding to the first client model includes: Based on the fifth loss function, determine the first gradient information corresponding to the first vector of the first client model and the second gradient information corresponding to the second parameter of the first client model; The second weights corresponding to the first client model are updated based on the first gradient information and the second gradient information to obtain the first weights corresponding to the first client model.

8. A data processing apparatus, characterized in that, The device, applicable to a cloud environment with a server-side model, includes: An acquisition unit is used to acquire one or more pieces of second data obtained by one or more client models processing the first data; the one or more client models are deployed on distributed nodes. A mixing processing unit is used to mix the one or more second data to obtain one or more third data; The training unit is used to train the server-side model using the one or more second data and the one or more third data to obtain one or more first loss functions; A determining unit is configured to determine one or more first weights corresponding to the one or more client models based on the one or more first loss functions; the one or more first weights are used to update the first parameters of the one or more client models to obtain an updated one or more client models, and the updated one or more client models are used to process the first data.

9. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images