Image recognition method, distributed system, device and storage medium

Abstract

The present application provides a method, distributed system, device and storage medium for image recognition, which relates to but is not limited to the field of image recognition technology. The present application is applied to a distributed system composed of a parameter server and multiple client systems, wherein the client system includes at least one edge server deployed with a third sub-model of an image model and a client deployed with a first sub-model and a second sub-model of the image model; the client obtains first gradient data of the third sub-model to calculate second gradient data of N training samples; the client performs gradient clipping and noise superposition on the N second gradient data to obtain average gradient data so that the edge server updates model parameters and re-iterates training until a first convergence condition is reached; the model parameters are updated by the parameter server so that the client system is retrained to obtain an image model for image recognition; the embodiments of the present application can more efficiently and securely provide an image model for image recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the field of image recognition technology, and in particular to an image recognition method, distributed system, device, and storage medium. Background Art

[0002] In image recognition and detection applications, a distributed system is usually used to train the model before performing image recognition, so as to make full use of the computing resources in the network, realize efficient processing and model optimization of large-scale data, and improve the accuracy and efficiency of image recognition and detection. Specifically, taking the distributed network architecture of the federated network as an example, the federated network communicates with multiple clients through a parameter server. The parameter server updates the model parameters obtained by the convergence of multiple client training and re-issues them, triggering the client to retrain until the preset convergence conditions are met, thereby obtaining an image model with higher image recognition and detection rates. However, in related technologies, due to the limited performance of the client, the efficiency and accuracy of model training are incompatible. Although there is also a way to add an edge server to the client in related technologies, in this scenario, the interaction between the edge server and the client is prone to security and privacy leaks. Therefore, there is an urgent need for a way to provide an image model for image recognition in a more efficient and secure way. Summary of the Invention

[0003] The embodiments of the present application provide a method, distributed system, device, and storage medium for image recognition, aiming to provide an image model for image recognition more efficiently and securely.

[0004] In a first aspect, an embodiment of the present application provides an image recognition method, which is applied to a distributed system, wherein the distributed system is provided with a parameter server and multiple client systems communicatively connected to the parameter server, at least one of the client systems includes at least one edge server and at least one client communicatively connected to the edge server; the method includes:

[0005] Deploying a first sub-model and a second sub-model of a preset image model on each client, and deploying a third sub-model of the image model on the edge server, wherein the output of the first sub-model is connected to the third sub-model, and the output of the third sub-model is connected to the second sub-model; and the output of the second sub-model is the output of the image model;

[0006] The client obtains first gradient data of the third sub-model in the current iteration, and calculates second gradient data corresponding to N training samples in the current iteration based on the first gradient data;

[0007] The client performs gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data;

[0008] The client sends the average gradient data to the edge server to update the model parameters of the edge server and perform the next iterative training until the image model meets a preset first convergence condition;

[0009] When the image model satisfies a first convergence condition, sending the model parameters to the parameter server through the edge server to update the global model parameters;

[0010] The client and the edge server retrain according to the updated global model parameters until a second convergence condition is met to obtain a trained image model;

[0011] Image recognition is performed using the trained image model, and recognition data is output.

[0012] In a second aspect, an embodiment of the present application provides a distributed system, including:

[0013] Multiple client systems, at least one of which includes at least one edge server and at least one client in communication with the edge server; the client deploys a first sub-model and a second sub-model of a preset image model; the edge server deploys a third sub-model of the image model, wherein the output of the first sub-model is connected to the third sub-model, the output of the third sub-model is connected to the second sub-model; and the output of the second sub-model is the output of the image model;

[0014] The client is used to:

[0015] Obtaining first gradient data of the third sub-model in the current iteration, and calculating second gradient data corresponding to the N training samples in the current iteration based on the first gradient data;

[0016] Perform gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data;

[0017] The client sends the average gradient data to the edge server;

[0018] The edge server is used to:

[0019] Updating model parameters according to the average gradient data and performing next iterative training until the image model meets a preset first convergence condition;

[0020] When the image model satisfies a first convergence condition, sending the model parameters to the parameter server through the edge server;

[0021] The parameter server updates the global model parameters and causes the client and the edge server to retrain according to the updated global model parameters until a second convergence condition is met, thereby obtaining a trained image model;

[0022] The client is also used to perform image recognition using the trained image model and output recognition data.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0024] one or more processors;

[0025] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any method described in the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described in the first aspect.

[0027] According to the above-mentioned embodiments of the present application, at least the following beneficial effects are achieved: by splitting the image model, the network layer with a larger amount of computation is deployed on the edge server, thereby improving the training efficiency of the model. At the same time, the model parameters of the edge server are updated through the client based on the average gradient data, so that the interaction between the client and the edge server is only intermediate data, thereby improving the privacy of the data interaction between the client and the edge server, and the average gradient data is based on noise superposition processing, so that the probability of restoring the image data based on the gradient is lower, thereby further improving the security of the original image data during the image training process. And through the interaction of model parameters between the edge server and the parameter server, the global image model can be updated in the distributed system composed of the client system and the parameter server, which can take into account the efficiency and accuracy of the image model training. Therefore, compared with the related art, the embodiments of the present application can provide a method, distributed system, device and storage medium for image recognition, which can provide an image model for image recognition more efficiently and securely. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the architecture of a distributed system provided in one embodiment of the present application;

[0029] Figure 2 A flowchart of an image recognition method provided in one embodiment of the present application;

[0030] Figure 3A schematic diagram of the splitting of an image model provided in one embodiment of the present application;

[0031] Figure 4 A schematic diagram of the training process of the image recognition method provided in one embodiment of the present application;

[0032] Figure 5 Simulation results of the image recognition method provided in one embodiment of the present application and image recognition in the prior art;

[0033] Figure 6 The simulation results of the image recognition method provided in one embodiment of the present application under 10 clients and different edge servers;

[0034] Figure 7 The simulation results of the image recognition method provided in one embodiment of the present application on two edge servers and different clients;

[0035] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0038] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0039] The following is an explanation of the terms used in this application:

[0040] Federated learning (FL) is a distributed machine learning approach that enables multiple participants to exchange model parameters through secure mechanisms, without exchanging data, to achieve collaborative training. FL can effectively help multiple institutions utilize data and conduct machine learning modeling while meeting user privacy, data security, and government regulations. Applicable machine learning algorithms include logistic regression, neural networks, and random forests, and are expected to become the foundation for the next generation of collaborative AI algorithms and networks.

[0041] Split Learning (SL) is a distributed model training solution. The core idea is to split the network structure. Each device only retains a part of the network. The sub-network structures of all devices constitute a complete network model. During the training process, different devices only perform forward or reverse calculations on the local network structure and pass the calculation results to the next device.

[0042] Split Fed is a distributed algorithm that combines the weight mean calculation of FL with the model splitting features of SL. In the first phase, each client initiates forward propagation, and the results are sent sequentially to the server for processing. In the second phase, the server performs backward propagation and transmits the results to each client. Simultaneously, the federated server calculates the average weight of each client and sends it back to each client to update the weight.

[0043] Parallel Split Learning (PSL) splits a deep neural network into two parts. The first part is sent to different clients, and the second part is sent to a single server. All clients and the single server exchange intermediate data and gradients during training to complete the image recognition task.

[0044] It should be noted that image recognition and detection applications typically utilize distributed systems for model training before image recognition. This fully utilizes network computing resources, enabling efficient processing of large-scale data and model optimization, while also improving the accuracy and efficiency of image recognition and detection. Specifically, taking the distributed network architecture of a federated network as an example, the federated network communicates with multiple clients via a parameter server. The parameter server updates the model parameters obtained through convergence training by multiple clients and re-distributes them, triggering client retraining until the preset convergence criteria are met. This results in an image model with higher image recognition and detection rates. Furthermore, to further improve training efficiency, split federation and parallel split learning have been proposed based on federated learning. While Split Fed combines the weight mean calculation of FL and model splitting of SL, its most significant drawback is that the forward and backward propagation processes in the server are not scalable as the number of clients increases, compromising both recognition accuracy and efficiency. Furthermore, intercepting the intermediate data sent by the client to the server allows the entire learned model to be reproduced, resulting in security and privacy breaches. Parallel split learning also suffers from the aforementioned privacy breaches. At the same time, in related technologies, due to the limited performance of the client, the efficiency and accuracy of model training are incompatible. Although related technologies also add edge servers to the client to improve efficiency, in this scenario, the interaction between the edge server and the client is prone to security and privacy leaks. Therefore, there is an urgent need for a method that can balance the efficiency and accuracy of image model training and the security of image data. Based on this, this application proposes an image recognition method, distributed system, device and storage medium that can provide an image model for image recognition more efficiently and securely.

[0045] Reference Figure 1 The distributed system shown includes a federated learning network system consisting of a parameter server 300 and multiple client systems 400. At least one client system 400 includes at least one edge server 100 and at least one client 200 communicatively connected to the edge server 100. The client 200 deploys a first sub-model and a second sub-model of a preset image model. The edge server 100 deploys a third sub-model of the image model, wherein the output of the first sub-model is connected to the third sub-model, and the output of the third sub-model is connected to the second sub-model. The output of the second sub-model is the output of the image model.

[0046] Client 200 is used to:

[0047] Obtain the first gradient data of the third sub-model in the current iteration, and calculate the second gradient data corresponding to the N training samples in the current iteration based on the first gradient data;

[0048] Perform gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data;

[0049] The client 200 sends the average gradient data to the edge server 100;

[0050] The edge server 100 is used to:

[0051] Update the model parameters according to the average gradient data and perform the next iterative training until the image model meets the preset first convergence condition;

[0052] When the image model satisfies the first convergence condition, the model parameters are sent to the parameter server 300 via the edge server 100;

[0053] The parameter server 300 updates the global model parameters and causes the client 200 and the edge server 100 to retrain according to the updated global model parameters until a second convergence condition is met, thereby obtaining a trained image model.

[0054] The client 200 is also used to perform image recognition using a trained image model and output recognition data.

[0055] Therefore, by splitting the image model, the network layer with a larger amount of computation is deployed on the edge server 100, thereby improving the training efficiency of the model. At the same time, the model parameters of the edge server 100 are updated through the client 200 based on the average gradient data, so that the interaction between the client 200 and the edge server 100 is only intermediate data, thereby improving the privacy of the data interaction between the client 200 and the edge server 100, and the average gradient data is based on noise superposition processing, which makes the probability of restoring the image data based on the gradient lower, thereby further improving the security of the original image data during the image training process. And through the interaction of model parameters between the edge server 100 and the parameter server 300, the global image model can be updated in the distributed system composed of the client system 400 and the parameter server 300, which can take into account the efficiency and accuracy of image model training. Therefore, compared with the related art, the embodiment of the present application can provide an image model for image recognition more efficiently and securely.

[0056] It should be noted that by setting the first sub-model, the second sub-model and the third sub-model, a U-shaped network architecture is formed between the client 200 and the corresponding edge server 100. For the client system 400, there is no need to transmit the training data set between the client 200 and the corresponding edge server 100. Only intermediate data (such as the data processed by the first sub-model for the image to be recognized and the average gradient data) needs to be transmitted, so that the network communication overhead caused by the transmission between the client 200 and the edge server 100 is very small, while providing stronger privacy protection.

[0057] It should be noted that the parameter server 300 aggregates the model parameters of all client systems 400 to update the global parameters of the image model, thereby improving the overall accuracy.

[0058] It should be noted that the first convergence condition is the condition for the client 200 and the edge server 100 to stop local iterative training for the global model parameters sent by the parameter server 300, and the second convergence condition is the condition for stopping the update of the global model parameters. At this time, the accuracy and efficiency of each client system 400 meet the preset requirements.

[0059] It should be noted that, for one client system 400, multiple clients 200 may share one edge server 100 or multiple edge servers 100. Figure 1 As shown, clients K1~KN share an edge server K; Figure 1 As shown, each client B corresponds to one edge server 100. Preferably, in this application, five clients 200 share two edge servers 100 for local training of the image model.

[0060] It should be noted that the first sub-model, the second sub-model, and the third sub-model are obtained by real-time splitting based on the performance of the devices in the existing network architecture and the characteristics of the image model.

[0061] It is understandable that referring to Figure 2 As shown, the present application also provides an image recognition method, which is applied to a distributed system. The distributed system is provided with a parameter server 300 and multiple client systems 400 communicatively connected to the parameter server 300. At least one client system 400 includes at least one edge server 100 and at least one client 200 communicatively connected to the edge server 100. The method includes:

[0062] Step S100: deploying a first sub-model and a second sub-model of a preset image model on each client 200, and deploying a third sub-model of the image model on the edge server 100, wherein the output of the first sub-model is connected to the third sub-model, and the output of the third sub-model is connected to the second sub-model; the output of the second sub-model is the output of the image model;

[0063] Step S200: The client 200 obtains the first gradient data of the third sub-model in the current iteration, and calculates the second gradient data corresponding to the N training samples in the current iteration based on the first gradient data;

[0064] Step S300: The client 200 performs gradient clipping and noise superposition processing on N second gradient data to obtain average gradient data;

[0065] Step S400: The client 200 sends the average gradient data to the edge server 100 to update the model parameters of the edge server 100 and perform the next iterative training until the image model meets the preset first convergence condition;

[0066] Step S500: When the image model meets the first convergence condition, the model parameters are sent to the parameter server 300 through the edge server 100 to update the global model parameters;

[0067] In step S600, the client 200 and the edge server 100 retrain according to the updated global model parameters until a second convergence condition is met, thereby obtaining a trained image model.

[0068] Step S700: Perform image recognition using the trained image model and output recognition data.

[0069] Therefore, by splitting the image model, the network layer with a larger amount of computation is deployed on the edge server 100, thereby improving the training efficiency of the model. At the same time, the update of the model parameters of the edge server 100 by the client 200 is based on gradient data processing, thereby improving the privacy of the data interaction between the client 200 and the edge server 100, and the model parameters are based on the average gradient data after noise superposition processing, which makes the probability of restoring the image data based on the gradient lower, thereby further improving the security of the image data during the image training process. And through the interaction of model parameters between the edge server 100 and the parameter server 300, the global image model can be updated in the distributed system composed of the client system 400 and the parameter server 300. Therefore, compared with the related art, the embodiment of the present application can provide a method for image recognition that can more efficiently and securely provide an image model for image recognition.

[0070] It should be noted that if Figure 1 As shown, a U-shaped model training network architecture is formed by the first sub-model, the second sub-model, and the third sub-model. Starting from the client 200, the U-shaped model training network architecture is forward propagated until the client 200 generates a predicted label and calculates the value of the loss function. Then, the gradient is reversely propagated to the edge server 100 and finally returns to the client 200 to sequentially update the model parameters of each layer. The first gradient data is obtained by determining the output error of the third sub-model in the edge server 100 at the client 200, so that the second gradient data corresponding to the N training samples can be obtained and reversely propagated to the edge server 100. For example, it is assumed that after time t, the gradient value received by the client K is dA k,t , we can get each training sample x i The client 200 gradient value is Then each training sample xi The corresponding second gradient data g k,t (x i )satisfy

[0071] It should be noted that gradient clipping can be done in the following two ways:

[0072] (1) Determine a gradient norm as a threshold. If the norm of the gradient of the parameter exceeds the threshold, clipping is performed.

[0073] (2) Use a fixed threshold for cropping.

[0074] Preferably, the gradient clipping in this application adopts the first method, which is clipping based on the norm of the gradient.

[0075] It should be noted that the average gradient data can update the weights of the corresponding network layers of the image model, thereby updating the model parameters.

[0076] It should be noted that, in some embodiments, the first convergence condition is set to ε <c1b 2 When T and δ>0, it satisfies When , the differential network composed of the first sub-model, the second sub-model and the third sub-model can achieve the (ε,δ)-DP level. Among them, δ is used to represent the difference between the output distributions corresponding to adjacent inputs; ε is the differential privacy budget; b = n k / batch_size; where n k is the number of training samples, batch_size is the number of samples N in the sample set of each iteration; c1 and c2 are the thresholds set in the differential privacy function.

[0077] It is understandable that the N second gradient data are subjected to gradient clipping and noise superposition processing to obtain average gradient data, including:

[0078] Obtain a gradient norm bound, and perform gradient clipping on each of the N second gradient data according to the gradient norm bound to obtain one-to-one corresponding third gradient data;

[0079] Get the calibration noise;

[0080] Calculating the sum of N third gradient data and calibration noise respectively to obtain fourth gradient data;

[0081] Average gradient data is obtained according to the average value of the N fourth gradient data.

[0082] It should be noted that, assuming that the noise scale is represented by σ and the gradient norm is represented by C′, each gradient can be clipped using the l2 norm according to the following formula (1), which is as follows:

[0083]

[0084] Among them, g k,t (x i ) represents the training sample x on client K i The second gradient data.

[0085] It should be noted that, assuming the calibration noise is N(0,σ 2 C′ 2 1), then the fourth gradient data is Therefore, the average gradient data Satisfies the following formula:

[0086]

[0087] Among them, n k is the number of samples N on client K.

[0088] Therefore, based on the average gradient data, the updated model parameters can satisfy Among them, W k,t is the model parameter of the current iteration of client K, W k,t+1 is the model parameter for the next iteration, η t A preset constant.

[0089] It should be noted that, assuming σ is When , we can add the noise in formula (2) to obtain the (ε,δ)-DP of b, where b=n k / batch_size. That is, when all b-stage operations are completed, (bε,bδ)-DP will be achieved. <c1b k When T and δ>0, it satisfies Differential privacy will reach the (ε,δ)-DP level; where b = n k / batch_size;n k is the number of training samples, batch_size is the number of samples N in the sample set of each iteration; T represents the round of current iterative training; c1 and c2 are the thresholds set in the differential privacy function.

[0090] It is understandable that, according to the gradient norm bound, the N second gradient data are gradient clipped respectively to obtain one-to-one corresponding third gradient data, including:

[0091] Calculate the ratio of the norm of the N second gradient data to the gradient norm bound respectively;

[0092] When the ratio is greater than or equal to 1, the second gradient data is divided by the ratio to obtain one-to-one corresponding third gradient data;

[0093] When the ratio is less than or equal to 1, the second gradient data is used as the one-to-one corresponding third gradient data.

[0094] It should be noted that the third gradient data can be determined by the above formula (1). k,t (x i )||2>C′, ensure||g k,t (x i )||2 is conserved; otherwise, it will be scaled down to C′; thus reducing the impact of each individual instance on g in the differential privacy method. k,t restrictions.

[0095] It is understandable that after the edge server 100 deploys the third sub-model of the image model, the method further includes:

[0096] Get calibration noise;

[0097] The calibration noise is superimposed on the intermediate data output by the first sub-model, so that the intermediate data superimposed with the calibration noise is trained by the second sub-model.

[0098] It should be noted that, to ensure the security of the model architecture, although we have added differential privacy to the client 200 model in the previous part, there is still the possibility of split data leakage. Therefore, in order to avoid the leakage of original data and further enhance confidentiality and model adversarial properties, this application is based on PixelDP and adds a noise layer at the output of the first sub-model. It makes full use of the noise application mechanism in differential privacy and adds a calibration noise to the output of a single layer of the client 200 model (such as the activation vector), thereby improving the security of the interaction between the client 200 and the edge server 100.

[0099] It is understood that obtaining calibration noise includes:

[0100] Get the preset input distance metric and output distance metric;

[0101] The output distance metric is used as the norm subscript of the output change difference value of each training sample of the first sub-model to calculate the first norm;

[0102] The input distance metric is used as the norm subscript of the input change difference value of each training sample of the first sub-model to calculate the second norm;

[0103] The calibration noise is obtained by performing Laplace calculation based on the first norm and the second norm.

[0104] For example, assuming that A is a function that represents the maximum output change that can be produced by an input change, then given p-norm and q-norm as the input distance metric and output distance metric respectively, we have:

[0105]

[0106]

[0107] Among them, A k,i Represents the training sample x i The gradient value, minA k,j Indicates the gradient value corresponding to the smallest sample among the N samples trained in client K; x k minA k,j The corresponding training sample, A k,i -minA k,j Corresponding training sample x i Output change difference value; x i -x k Corresponding training sample x i The maximum output change ΔI of the first sub-model is obtained by finding the maximum output change difference value and the corresponding input change difference value. p,q ; ε is the differential privacy budget.

[0108] At this time, for the third gradient data, it can be updated according to the following formula:

[0109]

[0110] Similarly, taking the intermediate data as the average gradient data as an example, for any intermediate data, it can be superimposed Then it is sent to the edge server 100.

[0111] It can be understood that the first sub-model, the second sub-model and the third sub-model are determined by the following steps:

[0112] Get the network layer data of the image model;

[0113] Perform three-layer model combination splitting based on network layer data to obtain multiple network layer combination sets, each of which corresponds to a three-layer model combination mode of the image model;

[0114] Perform a preset number of iterative training on each network layer combination set to obtain performance parameters;

[0115] determining a target network layer combination set from a plurality of network layer combination sets according to the performance parameter;

[0116] According to the target network layer combination set, a first sub-model, a second sub-model and a third sub-model are obtained.

[0117] It should be noted that the performance parameters include performance-related indicators such as accuracy and execution time. You can choose the one with the best performance parameters, or you can choose the best of several performance parameters with the largest weights based on weights. This embodiment of the application will not go into too much detail.

[0118] For example, first determine the distributed organizational structure in which the parameter server 300, edge server 100, and client 200 are placed in the cloud, edge, and end layers respectively, and place the high-occupancy computing resources on the edge server 100 for processing to reduce the pressure on the cloud and client 200. Secondly, according to the characteristics of the model, select the splitting method that best suits the image model, that is, the optimal splitting point. Specifically, take the image model as the ResNet model as an example, Figure 3 As shown in the figure, the ResNet model has a total of 8 layers. Since the ResNet model needs to be split into a U-shaped architecture, that is, the data is forwarded from the current client 200, processed by the edge server 100, and then forwarded back to the current client 200. After the label result is verified, the back propagation is carried out along the original path. Therefore, we need to split the 8-layer model into 3 parts, placing them on the client 200, the edge server 100, and the client 200 respectively.

[0119] Next, we analyze each layer of the model: the first three layers of the ResNet model perform operations such as convolution and pooling on the dataset input; the 456th layer crops the images to the same size; and the final 78 layers compare labels. Layer 456 can be considered a sub-layer, which cannot be split (instead of splitting the image) and can only be placed on edge server 100 (due to the high computational complexity of these three layers). Therefore, we summarize the network layer combinations shown in Table 1:

[0120]

[0121] Table 1

[0122] As shown in Table 1, there are six possible splitting scenarios (Numbers: I to VI). The C1-layer and C2-layer are deployed on the same client 200, forming the first sub-model and the second sub-model, respectively. The C1-layer represents the client 200 in the first half of the entire model, and the C2-layer represents the client 200 in the second half of the entire model (i.e., Client-back). Since Layer 4, Layer 5, and Layer 6 (i.e., layers 4 / 5 / 6 of the model) do not undergo splitting (and experimentally verified that splitting has no effect on the results), the C1-layer only contains Layer 1, Layer 2, and Layer 3 (i.e., layers 1 / 2 / 3 of the model), and the C2-layer only contains Layer 7 and Labels. Therefore, there are only six possible scenarios as shown in Table 1. The average and maximum accuracy of the training set (Acc-Avg-Training, Acc-Max-Training), the average and maximum accuracy of the test set (Acc-Avg-Test, Acc-Max-Test), and the time (min) for running 200 epochs for the six cases were statistically analyzed to obtain the summary data in Table 1.

[0123] According to the summary data in Table 1, it is determined that the final split of Layer7 and Labels (i.e., the 7 / 8 layers of the model) has no effect on the accuracy, but the C2-layer only has the 8th layer (i.e., the Labels layer), which has a shorter running time than the 7 / 8 layers. Therefore, it can be determined that only the 8th layer is placed in the C2-layer. It can be determined that when the C1-layer has a 3-layer structure, the accuracy of both the training set and the test set is higher than that of cases I and II. At the same time, its running time is much lower than other cases. The entire model structure is determined to be C1-layer: 123, Server: 4567, C2-layer: 8, thereby determining that the first sub-model includes the first three layers of the network layer, the second sub-model includes the last layer of the network layer, and the remaining network layers are in the edge server 100.

[0124] It is understandable that the steps of the parameter server 300 updating the model parameters include:

[0125] Calculate the mean of the model parameters sent by the multiple edge servers 100 in communication with the parameter server 300 to obtain the average model parameters;

[0126] The average model parameters are sent to each edge server 100 to replace the model parameters of the edge server 100 with the average model parameters.

[0127] It should be noted that all clients 200 perform iterative training of the image model in parallel, and then aggregate the global model obtained for each global iteration to update the global parameters. When an edge server 100 corresponds to multiple clients 200, the forward and backward propagation of the model on the edge server 100 is processed sequentially according to the fragmented data of the client 200. The order of the clients 200 is randomly selected in the edge server 100 operation, and the model is updated in each forward and backward propagation. In addition, the edge server 100 will synchronously receive the fragmented data from all participating clients 200. The calculation results are back-propagated to update the gradients and model parameters of each sub-model in the entire client system 400, where the edge server 100 will clip and calibrate the received model parameters. After the image model of the client system 400 is updated, the model parameters of the image model are sent to the parameter server 300 to calculate the mean. At the same time, parameter server 300 performs FedAvg on the local model of client 200, using weighted averaging to aggregate the model parameters of each client system 400. The FedAvg result is then sent back to all participating clients 200 to obtain the global optimal model. These operations are not affected by the client 200 because the local client 200 model is aggregated using the weighted averaging method, namely FedAvg.

[0128] Below, refer to Figure 4 As shown, the image recognition method of the present application is described with a specific embodiment, as follows:

[0129] After determining the model distribution, forward propagation begins within the data collected from client 200. Client 200 executes step 1, sending only the first intermediate data, the activation result, to the corresponding edge server 100. Edge server 100 then continues the forward propagation phase, passing the second intermediate data to the output layer.

[0130] The second intermediate data is obtained by calculating and processing data through the third sub-model. The following is the process of obtaining the second intermediate data and passing it to the output layer:

[0131] Input data: First, the edge server 100 receives first intermediate data output from the client 200, which can be sensor data, user requests, or other forms of input.

[0132] Data processing: Next, the edge server 100 processes the first intermediate data. This involves calculations, algorithms, or models to extract useful features or perform specific tasks. The processing process includes data cleaning, feature extraction, model inference, etc.

[0133] Intermediate parameter generation: During data processing, the edge server 100 generates some secondary intermediate data. This secondary intermediate data is calculated based on the primary intermediate data. The secondary intermediate data can be numbers, vectors, matrices, or other data structures, representing features, model weights, biases, and so on.

[0134] Pass to output layer: The generated second intermediate data is further passed to the output layer.

[0135] The second intermediate data is a result obtained by processing and calculating the first intermediate data, which is transmitted and converted inside the edge server 100 and is ultimately used to generate an output result or transmitted to other systems for use.

[0136] Edge server 100 executes step 2 (2), passing the gradient back to client 200 via the U-shaped network architecture. Client 200 obtains the output, generates the predicted label, terminates forward propagation, calculates the loss function, and begins backpropagating the gradient to edge server 100, ultimately returning to client 200. The gradient backpropagated to edge server 100 in the reverse direction is the average gradient after the noise layer is superimposed.

[0137] Generating a predicted label involves forward propagation of data to the model's output layer. Depending on the model design and task type, client 200 generates a predicted label using the output layer's results. For example, for classification tasks, the category with the highest probability can be selected as the predicted label; for regression tasks, the output layer value can be used directly as the predicted label. However, for image recognition tasks in this patent, the image recognition accuracy rate is used as the predicted label.

[0138] After the processing is completed, all edge servers 100 execute the third step ③, passing the model parameters to the parameter server 300 to obtain the average value, and then sending it back to each edge server 100 to update the model parameters of the edge server 100; at this point, a global training is completed, which is a complete image processing. During the training phase, multiple rounds of global training are required to meet the test requirements.

[0139] It should be noted that on client system 400, the process runs sequentially from client 200 to server and back to client 200, and then the feedback process is started while running parameter server 300. Model aggregation on parameter server 300 is accomplished by applying a weighted average to all locally trained model parts. It should be noted that cross-entropy loss is used as the loss function for client system 400 to measure the difference between the probability distribution of real data and the model output.

[0140] It should be noted that the number of clients 200 and edge servers 100 can be determined as follows:

[0141] The HAM10000 dataset is run through the ResNet model. Under 200 epochs, the number of clients 200 and edge servers 100 is changed respectively. The image recognition accuracy (Acc) and running time of the centralized (Normal), federated learning (FL), split learning (SL), federated split learning (SFLV1) and the image recognition method in this application are compared. Figure 5 It can be seen that:

[0142] like Figure 5 As shown in (a), the accuracy of the Nomal centralized application ResNet model can be stabilized at around 76% under 200 epochs. We use the centralized model as the standard. As the number of clients (5-100) increases, the following situations occur in the SL method (such as Figure 5 (c) shows that the convergence speed and accuracy are almost unaffected. Figure 5 (b) As the number of clients (5-100) increases, the convergence speed becomes slower and the accuracy becomes lower. Figure 5 (d) As the number of clients (5-100) increases, the convergence speed and accuracy are slightly affected. Figure 5 (e) Under 200 Epoch, when 5-100 clients 200 correspond to 1 edge server 100, the fewer the clients 200, the better the convergence speed and accuracy. When there are more than 20 clients 200, the overall convergence speed and accuracy will deteriorate. Figure 5 (f) When running two servers, the overall effect is the same as that of one edge server 100, but in the case of the same client 200, the two edge servers 100 have better effects. Figure 5 (b)~ Figure 5 As shown in (f), each curve corresponds to a performance curve of a combination of the number of clients 200 and servers 100.

[0143] For the image recognition method of the present application, the following simulation is performed with 10 clients 200 and 1, 2, 5, and 10 edge servers 100: Figure 6 As shown, when the number of clients 200 remains unchanged (10), the performance of the two edge servers 100 is optimal for the following reasons:

[0144] 1. The model we made is a distributed architecture (at least two edge servers 100 can reflect the distribution), so the effect of two edge servers 100 (S2) is better than one edge server 100 (S1).

[0145] 2. The fewer edge servers 100, the closer it is to a centralized model, and the better the convergence speed and accuracy.

[0146] For example, refer to Figure 7 As shown, when two edge servers are 100, they are 5 ( Figure 7 (a)), 10 ( Figure 7 (b)), 20 ( Figure 7 (c) Comparison of five methods under client 200. Figure 7 As shown, the five methods can quickly reach stability and converge. Taking centralized learning as the standard, the accuracy of federated learning FL and split learning SL is around 75, hovering below centralized learning. The accuracy of SFL is not as good as traditional split learning, and is stable at around 73. The image recognition method provided in this application improves accuracy and security by changing the model architecture, and adds an edge server 100 (the parallel image recognition method requires at least two edge servers 100). It can be seen that the test accuracy can be stabilized at 78, with a peak of 82. 5 clients 200 fluctuated greatly, 10 clients 200 fluctuated slightly, and 20 clients 200 were more stable. It can be seen that the more clients 200 there are, the better the overall effect.

[0147] Therefore, by Figures 5 to 7 It can be seen that:

[0148] 1. Impact of different numbers of clients 200 on Acc results: As the number of clients 200 increases, SL is not affected by the number of clients 200; the convergence speed of FL slows down and the accuracy also decreases; the convergence speed and accuracy of SFLV1 are slightly affected; the convergence speed and accuracy of the method of the present application exceed the centralized method when there are 20 clients 200.

[0149] 2. Impact of different numbers of servers on the Acc results of the UFSL method: When the number of clients 200 remains unchanged, the performance of the two edge servers 100 in the method of this application is optimal, which is closer to a centralized distributed architecture.

[0150] 3. Running time of five methods (C5_S2_E200): Normal—450.55 min; SL—407.75 min; FL—720.37 min; SFLV1—571.55 min; this application—421.84 min.

[0151] Taking Normal centralized approach as the benchmark, SL has the shortest runtime, and the split architecture can significantly reduce runtime. FL runs slowly due to waiting time and model architecture. SFLV1 combines the characteristics of both methods and lies in between. This application can fully combine the advantages of both approaches, significantly improving runtime and exceeding centralized approach.

[0152] 4. When the number of clients is the same (200), the differences in Acc among the five methods are as follows: The method proposed in this application improves accuracy and security by changing the model architecture, and the performance is the best. The test accuracy of S2_C10 can be stabilized at 78, with a peak of 82.

[0153] 5. Test the CV values ​​of Acc for FL, SL, and the three methods in this application. Since the CV coefficient value is less than 10, the effect is good and conforms to the normal distribution.

[0154] Therefore, the proposed method has an optimal split architecture of 1 / 2-3 / 4 / 5 / 6 / 7-8 models, and an optimal combination of 10 clients and 2 servers, which can ensure the highest Acc of the global model and a shorter runtime.

[0155] Reference Figure 8 As shown, an embodiment of the present application further provides an electronic device, including:

[0156] One or more processors 601;

[0157] The memory 602 stores one or more programs. When the one or more programs are executed by the one or more processors 601, the one or more processors 601 implement the method provided in any embodiment of the present application.

[0158] The memory 602 is a non-transient network system that can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 602 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory 602 remotely located relative to the processor 601, and these remote memories 602 may be connected to the processor 601 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the methods of the embodiments of this application.

[0160] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0161] In some embodiments, the electronic device further comprises:

[0162] Input / output interface, used to realize information input and output;

[0163] Communication interface, used to realize communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0164] A bus that transmits information between various components of the device (e.g., processor 601, memory 602, input / output interfaces, and communication interfaces);

[0165] The processor 601 , the memory 602 , the input / output interface and the communication interface can be connected to each other in communication within the device via a bus.

[0166] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method provided in any embodiment of the present application.

[0167] An embodiment of the present application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device performs a method as provided in any embodiment of the present application.

[0168] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0170] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division between physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0171] As used in this specification, the terms "component," "module," "system," and the like are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both an application running on a computing device and a computing device may be a component. One or more components may reside in a process or execution thread, and a component may be located on a single computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through a local or remote process based on signals having one or more data packets [e.g., data from two components interacting with another component on a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals].

[0172] The above description of some embodiments of the present application with reference to the accompanying drawings does not limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.

Claims

1. A method for image recognition, characterized in that: Applied to a distributed system, the distributed system is provided with a parameter server and a plurality of client systems communicatively connected to the parameter server, at least one of the client systems includes at least one edge server and at least one client communicatively connected to the edge server; the method comprises: Deploying a first sub-model and a second sub-model of a preset image model on each client, and deploying a third sub-model of the image model on the edge server, wherein the output of the first sub-model is connected to the third sub-model, and the output of the third sub-model is connected to the second sub-model; and the output of the second sub-model is the output of the image model; The client obtains first gradient data of the third sub-model in the current iteration, and calculates second gradient data corresponding to N training samples in the current iteration based on the first gradient data; The client performs gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data; The client sends the average gradient data to the edge server to update the model parameters of the edge server and perform the next iterative training until the image model meets a preset first convergence condition; When the image model satisfies a first convergence condition, sending the model parameters to the parameter server through the edge server to update the global model parameters; The client and the edge server retrain according to the updated global model parameters until a second convergence condition is met to obtain a trained image model; Image recognition is performed using the trained image model, and recognition data is output.

2. The image recognition method according to claim 1, characterized in that: The step of performing gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data includes: Obtaining a gradient norm bound, and performing gradient clipping on each of the N second gradient data according to the gradient norm bound to obtain one-to-one corresponding third gradient data; Get the calibration noise; respectively calculating the sum of N pieces of the third gradient data and the calibration noise to obtain fourth gradient data; The average gradient data is obtained according to the average value of the N fourth gradient data.

3. The image recognition method according to claim 2, characterized in that: The step of performing gradient clipping on the N pieces of the second gradient data according to the gradient norm bound to obtain one-to-one corresponding third gradient data includes: Calculating ratios of the norms of N pieces of the second gradient data to the gradient norm bound respectively; When the ratio is greater than or equal to 1, dividing the second gradient data by the ratio to obtain one-to-one corresponding third gradient data; When the ratio is less than or equal to 1, the second gradient data is used as the one-to-one corresponding third gradient data.

4. The image recognition method according to claim 1, wherein: After deploying the third sub-model of the image model on the edge server, the method further includes: Get calibration noise; The calibration noise is superimposed on the intermediate data output by the first sub-model, so that the intermediate data superimposed with the calibration noise is trained by the second sub-model.

5. The image recognition method according to claim 4, characterized in that: The obtaining of calibration noise includes: Get the preset input distance metric and output distance metric; The output distance metric is used as a norm subscript of the output change difference value of each training sample of the first sub-model to calculate a first norm; The input distance metric is used as a norm subscript of the input change difference value of each training sample of the first sub-model to calculate a second norm; Performing Laplace calculation according to the first norm and the second norm to obtain the calibration noise.

6. The image recognition method according to claim 1, characterized in that: The first sub-model, the second sub-model and the third sub-model are determined by the following steps: Obtaining network layer data of the image model; Performing a three-layer model combination splitting according to the network layer data to obtain a plurality of network layer combination sets, each network layer combination set corresponding to a three-layer model combination mode of the image model; Performing a preset number of iterative training on each of the network layer combination sets to obtain performance parameters; determining a target network layer combination set from the plurality of network layer combination sets according to the performance parameter; According to the target network layer combination set, the first sub-model, the second sub-model and the third sub-model are obtained.

7. The image recognition method according to claim 1, characterized in that: The updating of the global model parameters comprises the following steps: Calculate the average of the model parameters sent by multiple edge servers in communication with the parameter server to obtain an average model parameter; The global model parameters are replaced by the average model parameters.

8. A distributed system, characterized in that: include: a plurality of client systems, at least one of which comprises at least one edge server and at least one client in communication with the edge server; The client deploys a first sub-model and a second sub-model of a preset image model; the edge server deploys a third sub-model of the image model, wherein the output of the first sub-model is connected to the third sub-model, the output of the third sub-model is connected to the second sub-model; and the output of the second sub-model is the output of the image model; The client is used to: Obtaining first gradient data of the third sub-model in the current iteration, and calculating second gradient data corresponding to the N training samples in the current iteration based on the first gradient data; Perform gradient clipping and noise superposition processing on the N second gradient data to obtain average gradient data; The client sends the average gradient data to the edge server; The edge server is used to: Updating model parameters according to the average gradient data and performing next iterative training until the image model meets a preset first convergence condition; When the image model satisfies a first convergence condition, sending the model parameters to a parameter server through the edge server; The parameter server updates the global model parameters and causes the client and the edge server to retrain according to the updated global model parameters until a second convergence condition is met, thereby obtaining a trained image model; The client is also used to perform image recognition using the trained image model and output recognition data.

9. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images