A model processing method, a federated learning method and related equipment

Abstract

The embodiment of the present application discloses a model processing method, which can be applied to model training and pruning scenarios. The method can be executed by a client, and can also be executed by a component of the client (such as a processor, chip or chip system, etc.). The method includes: training a neural network model according to a first loss function and training data to obtain a first model; pruning the first model based on a second loss function and constraints to obtain a second model, and the constraints are used to constrain the accuracy of the second model to be no less than the accuracy of the first model. In the process of pruning the first model, considering the constraints based on the data loss function is equivalent to providing a direction for the pruning of the first model, so that the accuracy of the pruned second model is no less than the accuracy of the first model, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal artificial intelligence, and in particular to a model processing method, a federated learning method, and related equipment. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] At present, deep learning is widely used as one of the mainstream branches of machine learning. However, the large size and large number of parameters of deep neural network models, as well as the problems of computing, storage, power consumption, latency, etc., hinder the commercialization of deep learning models. To solve this problem, it is necessary to simplify the deep neural network. Among them, pruning technology is the most widely used. Pruning technology compresses the deep neural network by removing some parameters and modules. The classic pruning process includes three steps. First, the model is trained based on the local data set, then the trained model is pruned according to the preset rules, and finally the pruned model is fine-tuned using the local data set to avoid too much loss of model accuracy. The entire pruning process is cumbersome and inefficient. Summary of the invention

[0004] The embodiment of the present application provides a model processing method and related equipment, which can be used in combination with the federated learning method. In the process of pruning the first model, considering the constraint conditions based on the data loss function is equivalent to constraining the pruning direction of the first model, so that the accuracy of the pruned second model is not lower than the accuracy of the first model, reducing the subsequent steps of fine-tuning the model accuracy, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process.

[0005] In a first aspect, the present application provides a model processing method that can be applied to model training and pruning scenarios. The method can be executed by a model processing device (such as a client) or by a component of a client (such as a processor, a chip or a chip system, etc.). The method includes: obtaining training data including label values; taking the training data as input, training a neural network model according to a first loss function to obtain a first model, wherein the first model includes multiple substructures, and each of the multiple substructures includes at least two neurons; pruning the first model based on a second loss function and a constraint condition to obtain a second model, wherein the second loss function is used to indicate that at least one of the multiple substructures is to be pruned, and the constraint condition is used to constrain the accuracy of the second model to be not lower than the accuracy of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value.

[0006] Among them, the first loss function can also be understood as a data loss function, which is mainly used to evaluate the accuracy of the model in the process of using data to train the model. The second loss function can be understood as a sparse loss function, which is mainly used to sparse the model (or prune). The substructure can be a channel, feature map, network layer, subnetwork, or other predefined network structure composed of multiple neurons of a neural network model; when the neural network model is a convolutional neural network, the substructure can also be a convolution kernel. In short, a substructure can be regarded as a functional whole. Pruning a substructure during pruning means pruning all neurons included in the substructure.

[0007] In an embodiment of the present application, in the process of pruning the first model, the constraints based on the data loss function are considered, which is equivalent to constraining the pruning direction of the first model, so that the accuracy of the pruned second model is not lower than the accuracy of the first model, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, and pruning the substructure is more efficient than pruning neurons one by one, thereby improving the efficiency of the model pruning process while ensuring the accuracy of the pruned model, and the resulting model structure is also simpler.

[0008] Optionally, in a possible implementation of the first aspect, the above-mentioned constraint condition is specifically used to constrain the angle between the descent direction of the first loss function and the descent direction of the second loss function to be less than or equal to 90 degrees. The descent direction of the first loss function may be a gradient direction obtained by deriving the first loss function, and the descent direction of the second loss function may be a gradient direction obtained by deriving the second loss function.

[0009] In this possible implementation, by adjusting the angle between the descending direction of the first loss function and the descending direction of the second loss function to be less than or equal to 90 degrees, it can be ensured that the accuracy of the second model after pruning will not decrease compared with the accuracy of the first model before pruning, thereby reducing the subsequent steps of fine-tuning the model accuracy.

[0010] Optionally, in a possible implementation of the first aspect, the above-mentioned constraint condition is specifically used to constrain the value of the first loss function of the second model to be less than or equal to the value of the first loss function of the first model. In other words, the first model and the second model are used to predict the same data, and then the first loss function is used to measure the accuracy of the first model and the second model. The smaller the value of the first loss function, the higher the accuracy of the model.

[0011] In this possible implementation, the specific accuracy can be determined by using the value of the first loss function of the second model and the value of the first loss function of the first model. Of course, an evaluation method different from the first loss function can also be used to compare the accuracy of the first model and the second model, and this application does not limit the specific evaluation method.

[0012] Optionally, in a possible implementation manner of the first aspect, the second loss function includes a first sparse term, and the first sparse term is related to a weight of at least one substructure among the multiple substructures.

[0013] In this possible implementation, when pruning the first model, the first sparse term in the second loss function processes the substructure as a whole, so what is pruned during pruning are network structures such as channels, convolution kernels, feature maps, and network layers, rather than single neurons, which greatly improves the efficiency of pruning and makes the resulting model more refined and lightweight.

[0014] Optionally, in a possible implementation manner of the first aspect, the second loss function further includes a difference term, where the difference term indicates the difference between the first model and the second model.

[0015] In this possible implementation, by adding a difference term to the second loss function, the difference between the models before and after pruning can be constrained to a certain extent, ensuring the similarity of the models before and after pruning, and thus ensuring the accuracy of the pruned model.

[0016] Optionally, in a possible implementation of the first aspect, the above steps: pruning the first model based on the second loss function and constraints to obtain the second model, include: calculating the update coefficient based on the constraints, the update coefficient is used to adjust the direction of the first sparse term; using the update coefficient to update the first sparse term in the second loss function to obtain a third loss function, the third loss function includes a difference term and a second sparse term, the second sparse term is updated based on the update coefficient and the first sparse term; pruning the first model based on the third loss function to obtain the second model.

[0017] In this possible implementation, the pruning direction can be adjusted by introducing an update coefficient so that the pruned second model satisfies the constraint condition. The pruning in this case can also be understood as directional pruning.

[0018] Optionally, in a possible implementation manner of the first aspect, the third loss function includes:

[0019]

[0020] Among them, || ||2 is the L2 norm, V n is the parameter of the first model, W n is the parameter of the second model, λ is a hyperparameter used to adjust the weight of the first sparse term; s i is the update coefficient, by adjusting s i To satisfy the constraints, is the parameter of the i-th substructure in the second model.

[0021] In this possible implementation, the first sparse item is updated by updating the coefficient to obtain the second sparse item, and the updated second sparse item can perform direction pruning on the model to ensure that the accuracy of the model is not lost after pruning.

[0022] Optionally, in a possible implementation of the first aspect, the above step: pruning the first model based on the second loss function and the constraint condition to obtain the second model includes: randomly pruning the first model at least once based on the second loss function until the second model obtained after pruning the first model satisfies the constraint condition. Specifically, the first model may be randomly pruned based on the second loss function to obtain the second model; if the constraint condition is satisfied, the second model is output; if the constraint condition is not satisfied, the step of randomly pruning the first model based on the second loss function is repeated until the constraint condition is satisfied.

[0023] In this possible implementation, pruning can be performed by random pruning plus constraint judgment. The pruned model is output only when the constraint is met. This method does not require the use of data to fine-tune the pruned model and is more versatile.

[0024] Optionally, in a possible implementation of the first aspect, the above method is applied to a client in a federated learning system, and the data used to train the neural network model is local data of the client, such as data collected by a sensor of the client or data generated during the operation of a program application of the client, etc. The method further includes: receiving a neural network model sent by an upstream device; and sending a second model to the upstream device. The upstream device is a device such as a server that can communicate with the client.

[0025] In this possible implementation, the method can be applied to federated learning scenarios. On the one hand, by introducing constraints to prune the model substructure, it can help upstream devices screen out multiple models with no loss of accuracy and simplified structure for aggregation, thereby reducing the communication burden of the uplink (i.e., the communication link from the client to the upstream device).

[0026] Optionally, in a possible implementation of the first aspect, the training data includes: image data, audio data or text data, etc. It can be understood that the above three are just examples of training data. In actual applications, the specific form of training data varies according to the type of task processed by the neural network model, and is not limited here.

[0027] Optionally, in a possible implementation of the first aspect, the above-mentioned neural network model is used to classify and / or identify image data, etc. It is understandable that in practical applications, the neural network model can also be used for target detection, information recommendation, speech recognition, text recognition, question-answering tasks, human-computer games, etc., which are not specifically limited here.

[0028] In this possible implementation, the pruning method provided in the embodiment of the present application can be applied to the neural network model applied to any scenario (for example: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.), which is beneficial to improving the pruning efficiency of the neural network model. In addition to reducing the storage space occupied by the neural network, it can also ensure the accuracy of the neural network model.

[0029] In a possible implementation, the above model training and pruning steps can be performed once or multiple times, depending on the specific settings. In this case, a model that better meets user expectations can be obtained, and pruning can be completed without losing model accuracy, saving storage and communication costs.

[0030] The second aspect of the present application provides a federated learning method that can be applied to model pruning scenarios. The method can be executed by an upstream device (a cloud server or an edge server, etc.), and can also be executed by a component of an upstream device (such as a processor, a chip, or a chip system, etc.). The method can be understood as an operation of constraining pruning first and then aggregating. The method includes: sending a neural network model to multiple downstream devices, the neural network model includes multiple substructures, each substructure includes at least two neurons; receiving multiple first models from multiple downstream devices, the multiple first models are obtained by training the neural network model, wherein the loss function used in the training process can be referred to as the first loss function; based on the second loss function) and constraints, the multiple first models are pruned separately, wherein the second loss function is used to indicate that the substructures of the multiple first models are pruned, and the constraints are used to constrain the accuracy of each first model after pruning to be not less than the accuracy before pruning; aggregating the pruned multiple first models to obtain a second model.

[0031] Optionally, in a possible implementation of the second aspect, the above step: pruning the first model based on the second loss function and the constraint condition includes: randomly pruning the first model at least once based on the second loss function until the model obtained after pruning the first model satisfies the constraint condition. Specifically, the first model may be randomly pruned based on the second loss function to obtain the first model before and after pruning; if the constraint condition is satisfied, the model is output; if the constraint condition is not satisfied, the step of randomly pruning the first model based on the second loss function is repeated until the constraint condition is satisfied.

[0032] The third aspect of the present application provides a federated learning method that can be applied to model pruning scenarios. The method can be executed by a server (cloud server or edge server, etc.), and can also be executed by a component of the server (such as a processor, chip or chip system, etc.). The method can be understood as an operation of first aggregation and then constraint pruning. The method includes: sending a neural network model to multiple downstream devices, the neural network model includes multiple substructures, each substructure includes at least two neurons; receiving multiple first models from multiple downstream devices, the multiple first models are obtained by training the neural network model, wherein the loss function used in the training process can be referred to as the first loss function; aggregating the multiple first models to obtain a second model; pruning the second model based on the loss function (hereinafter referred to as the second loss function) and constraints, wherein the second loss function is used to indicate that the substructure of the second model is pruned, and the constraints are used to constrain the accuracy of the second model after pruning to be not lower than the accuracy before pruning.

[0033] Optionally, in a possible implementation of the third aspect, the above step: pruning the second model based on the second loss function and the constraint condition includes: randomly pruning the second model at least once based on the second loss function until the model obtained after pruning the second model meets the constraint condition. Specifically, the second model may be randomly pruned based on the second loss function to obtain a model; if the constraint condition is met, the pruned second model is output; if the constraint condition is not met, the step of randomly pruning the second model based on the second loss function is repeated until the constraint condition is met.

[0034] In the implementation provided in the second aspect / third aspect, after the server uses the method provided in the embodiment of the present application to prune the model, it is no longer necessary to use the training data to adjust the model to ensure the accuracy of the model. That is, the server can prune the model without using the training data of the client and ensure the accuracy of the model, thus avoiding the need to transmit the training data of the client to the upstream device during pruning, and protecting the data privacy of the client. In the implementation provided in the second aspect / third aspect above, the substructure can be a channel, feature map, network layer, subnetwork, or other predefined network structure composed of multiple neurons of the neural network model; when the neural network model is a convolutional neural network, the substructure can also be a convolution kernel. In short, a substructure can be regarded as a functional whole, and pruning the substructure during pruning means pruning all neurons included in the substructure. The method provided in the present application prunes each substructure as a whole, which is more efficient than pruning neurons one by one, and the resulting model structure is also more concise.

[0035] Optionally, in a possible implementation of the second aspect / the third aspect, the above-mentioned multiple first models are trained according to the first loss function, the loss function used for pruning is called the second loss function, and the constraint condition is specifically used to constrain the angle between the descent direction of the first loss function and the descent direction of the second loss function to be less than or equal to 90 degrees. The descent direction of the first loss function may be the gradient direction obtained by deriving the first loss function, and the descent direction of the second loss function may be the gradient direction obtained by deriving the second loss function.

[0036] In this possible implementation, by adjusting the angle between the descending direction of the first loss function and the descending direction of the second loss function to be less than or equal to 90 degrees, it can be ensured that the accuracy of the first model before and after pruning will not decrease, thereby reducing the subsequent steps of fine-tuning the model accuracy.

[0037] Optionally, in a possible implementation of the second aspect / third aspect, the above-mentioned constraint condition is specifically used to constrain the value of the first loss function of the pruned model to be less than or equal to the value of the first loss function of the model before pruning. In other words, the same data is predicted using the models before and after pruning, and then the accuracy of the models before and after pruning is measured using the first loss function. The smaller the value of the first loss function, the higher the accuracy of the model.

[0038] In this possible implementation, the specific accuracy can be determined by using the value of the first loss function of the model before and after pruning. Of course, an evaluation method different from the first loss function can also be used to compare the accuracy of the model before and after pruning, and this application does not limit the specific evaluation method.

[0039] Optionally, in a possible implementation manner of the second aspect / the third aspect, the above steps further include: sending the pruned model to the multiple downstream devices.

[0040] This possible implementation can be applied to scenarios where cloud servers or edge servers perform pruning and aggregation. After pruning and aggregation, the pruned model is sent to multiple downstream devices. The downstream devices use the pruned model for reasoning or retrain the pruned model. Pruning the model before sending it to the downstream devices can reduce the communication burden on the one hand, and reduce the storage space and processing power requirements of the downstream devices on the other.

[0041] Optionally, in a possible implementation manner of the second aspect / the third aspect, the above steps further include: sending the pruned model to an upstream device.

[0042] This possible implementation can be applied to the scenario where the edge server performs pruning and aggregation. After pruning and aggregation, the pruned model is sent to the upstream device, so that the upstream server can continue to perform operations such as aggregation and pruning on the model to integrate information from more client devices.

[0043] Optionally, in a possible implementation manner of the second aspect / the third aspect, the second loss function includes a first sparse term, and the first sparse term is related to the weight of at least one substructure among the multiple substructures.

[0044] In this possible implementation, when pruning the model, the first sparse term in the second loss function processes the substructure as a whole, so what is pruned during pruning are network structures such as channels, convolution kernels, feature maps, and network layers, rather than individual neurons, which greatly improves the efficiency of pruning and makes the resulting model more refined and lightweight.

[0045] Optionally, in a possible implementation manner of the second aspect / the third aspect, the second loss function further includes a difference term, and the difference term indicates the difference between the model before and after pruning.

[0046] In this possible implementation, by adding a difference term to the second loss function, the difference between the models before and after pruning can be constrained to a certain extent, thereby ensuring the similarity of the models before and after pruning.

[0047] Optionally, in a possible implementation of the second aspect / the third aspect, the above steps: pruning multiple first models separately based on the second loss function and constraints, including: calculating an update coefficient based on the constraints, the update coefficient being used to adjust the direction of the first sparse term; using the update coefficient to update the first sparse term in the second loss function to obtain a third loss function, the third loss function including a difference term and a second sparse term, the second sparse term being updated based on the update coefficient and the first sparse term; pruning the model based on the third loss function.

[0048] In this possible implementation, the pruning direction can be adjusted by introducing an update coefficient so that the first model after pruning satisfies the constraint condition. The pruning in this case can also be understood as directional pruning.

[0049] Optionally, in a possible implementation of the second aspect / the third aspect, the third loss function includes:

[0050]

[0051] Among them, || ||2 is the L2 norm, V n With W n are the parameters of the model before and after pruning, respectively, λ is a hyperparameter used to adjust the weight of the first sparse term; s i is the update coefficient, by adjusting s i To satisfy the constraints, is the parameter of the i-th substructure in the pruned model.

[0052] In this possible implementation, the first sparse item is updated by updating the coefficient to obtain the second sparse item, and the updated second sparse item can perform direction pruning on the model to ensure that the accuracy of the model is not lost after pruning.

[0053] Optionally, in a possible implementation of the second aspect / third aspect, the training data includes: image data, audio data, or text data, etc. It can be understood that the above three types are just examples of training data. In actual applications, the specific form of training data varies according to different inputs of the neural network model, and is not limited here.

[0054] Optionally, in a possible implementation of the second aspect / third aspect, the above-mentioned neural network model is used to classify and / or identify image data, etc. It can be understood that in practical applications, the neural network model can also be used for prediction, encoding, decoding, etc., which is not specifically limited here.

[0055] In one possible implementation, the above steps of receiving, pruning, aggregating, and sending may be performed once or multiple times, depending on the specific settings. In this case, a model that better meets user expectations can be obtained, and pruning can be completed without losing model accuracy, thereby saving storage and communication costs.

[0056] The fourth aspect of the present application provides a model processing device that can be applied to model training and pruning scenarios. The model processing device can be a client, and the model processing device includes: an acquisition unit for acquiring training data including label values; a training unit for taking the training data as input, training a neural network model according to a first loss function, and obtaining a first model, wherein the first model includes multiple substructures, and each substructure includes at least two neurons; a pruning unit for pruning the first model based on a second loss function and a constraint condition to obtain a second model, wherein the second loss function is used to indicate that at least one of the multiple substructures is pruned, and the constraint condition is used to constrain the accuracy of the second model to be not lower than the accuracy of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value. Among them, the first loss function can also be understood as a data loss function, which is mainly used to evaluate the accuracy of the model in the process of using the data to train the model. The second loss function can be understood as a sparse loss function, which is mainly used to sparse (or prune) the model.

[0057] Optionally, the various units of the model processing device provided in the fourth aspect may be configured to implement the method in any possible implementation manner of the first aspect.

[0058] In a fifth aspect, the present application provides an upstream device that can be applied to model training and pruning scenarios, federated learning scenarios, etc. The upstream device can be a cloud server or an edge server in a federated learning scenario, and the upstream device includes: a sending unit for sending a neural network model to multiple downstream devices, the neural network model includes multiple substructures, each substructure includes at least two neurons; a receiving unit for receiving multiple first models from multiple downstream devices, the multiple first models are obtained by training the neural network model, wherein the loss function used in the training process can be referred to as the first loss function; a pruning unit for pruning the multiple first models based on the loss function (hereinafter referred to as the second loss function) and constraints, wherein the second loss function is used to indicate pruning the substructures of the multiple first models, and the constraints are used to constrain the accuracy of each first model after pruning to be not lower than the accuracy before pruning; an aggregation unit for aggregating the multiple first models after pruning to obtain a second model.

[0059] Optionally, each unit of the upstream device provided in the fifth aspect may be configured to implement the method in any possible implementation manner of the second aspect.

[0060] The sixth aspect of the present application provides an upstream device that can be applied to model training and pruning scenarios, federated learning scenarios, etc. The upstream device can be a cloud server or an edge server in a federated learning scenario, and the upstream device includes: a sending unit, used to send a neural network model to multiple downstream devices, the neural network model includes multiple substructures, each substructure includes at least two neurons; a receiving unit, used to receive multiple first models from multiple downstream devices, the multiple first models are obtained by training the neural network model, wherein the loss function used in the training process can be referred to as a first loss function; an aggregation unit, used to aggregate the multiple first models to obtain a second model; a pruning unit, used to prune the second model based on a loss function (hereinafter referred to as a second loss function) and constraints, wherein the second loss function is used to indicate pruning the substructure of the second model, and the constraints are used to constrain the accuracy of the second model after pruning to be not lower than the accuracy before pruning.

[0061] Optionally, each unit of the upstream device provided in the sixth aspect may be configured to implement the method in any possible implementation manner of the third aspect.

[0062] In the seventh aspect of the present application, there is provided an electronic device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the programs or instructions are executed by the processor, the electronic device implements the method in any possible implementation manner of the above-mentioned first aspect, second aspect, and third aspect.

[0063] In an eighth aspect, the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction is executed on a computer, the computer executes a method in any possible implementation of the first, second, or third aspects.

[0064] A ninth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect, the second aspect, or the third aspect.

[0065] The technical effects brought about by any possible implementation method of the fourth, fifth, sixth, seventh, eighth and ninth aspects mentioned above can refer to the technical effects brought about by the corresponding implementation methods in the first, second and third aspects above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0067] Figure 2 A schematic diagram of the architecture of a federated learning system provided for this application;

[0068] Figure 3 A schematic diagram of the architecture of another federated learning system provided for this application;

[0069] Figure 4 A schematic diagram of the architecture of another federated learning system provided for this application;

[0070] Figure 5 A schematic diagram of the architecture of another federated learning system provided for this application;

[0071] Figure 6 A flowchart of the federated learning method provided for this application;

[0072] Fig. 7A , Figure 8-Figure 10 Several schematic diagrams of pruning directions during the pruning process provided for this application;

[0073] Figure 7B A schematic diagram of the structure of the model before and after pruning provided in this application;

[0074] Fig.11 Another flowchart of the federated learning method provided for this application;

[0075] Fig.12 A flow chart of the model processing method provided in this application;

[0076] Fig.13A structural schematic diagram of the model processing device provided in this application;

[0077] Fig.14 A schematic diagram of the structure of the upstream equipment provided for this application;

[0078] Fig.15 Another structural schematic diagram of the upstream equipment provided for this application;

[0079] Fig.16 Another structural schematic diagram of the model processing device provided by this application;

[0080] Fig.17 Another structural schematic diagram of the upstream equipment provided for this application. DETAILED DESCRIPTION

[0081] The embodiment of the present application provides a model processing method, a federated learning method and related equipment. In the process of pruning the first model, considering the constraint conditions based on the data loss function is equivalent to providing a direction for the pruning of the first model, so that the accuracy of the pruned second model is not lower than the accuracy of the first model, reducing the subsequent steps of fine-tuning the model accuracy, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process.

[0082] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0084] (1) Infrastructure

[0085] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0086] (2) Data

[0087] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0088] (3) Data processing

[0089] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0090] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0091] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0092] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0093] (4) General ability

[0094] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0095] (5) Smart products and industry applications

[0096] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0097] The embodiments of the present application can be applied to the client or the cloud, and can also be applied to the training of machine learning models used in various application federated learning scenarios. The trained machine learning model can be applied to the above-mentioned various application fields to achieve classification, regression or other functions. The processing objects of the trained machine learning model can be image samples, discrete data samples, text samples or voice samples, etc., which are not exhaustively listed here. The machine learning model can be specifically expressed as a neural network, a linear model or other types of machine learning models, etc. Correspondingly, the multiple modules that constitute the machine learning model can be specifically expressed as a neural network module, a current model module or modules that constitute other types of machine learning models, etc., which are not exhaustively listed here. In the subsequent embodiments, only the machine learning model expressed as a neural network is used as an example for explanation. When the machine learning model is expressed as other types other than a neural network, it can be understood by analogy, and it will not be repeated in the embodiments of the present application.

[0098] The embodiments of the present application can be applied to the client, the cloud, or federated learning, etc., mainly to train and prune neural networks, and therefore involve a large number of neural network-related applications. In order to better understand the solutions of the embodiments of the present application, the following first introduces the relevant terms and concepts of neural networks that may be involved in the embodiments of the present application.

[0099] 1. Neural Networks

[0100] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be shown as formula (1-1):

[0101]

[0102] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0103] 2. Deep Neural Networks

[0104] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are intermediate layers, or hidden layers. In a neural network that has not been pruned or compressed, the layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0105] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0106] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0107] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0108] 3. Convolutional Neural Networks

[0109] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0110] 4. Recurrent neural network (RNN)

[0111] In the traditional neural network model, the layers are fully connected, and the devices between each layer are disconnected. However, this ordinary neural network cannot solve many problems. For example, predicting the next word in a sentence, because the previous and next words in a sentence are not independent, the previous word is generally needed. A recurrent neural network refers to a sequence in which the current output is also related to the previous output. The specific manifestation is that the network will remember the previous information, save it in the internal state of the network, and apply it to the calculation of the current output.

[0112] 5. Loss Function

[0113] In the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as squared error, cross entropy, logarithm, exponential, etc. For example, the mean square error can be used as the loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0114] 6. Back Propagation Algorithm

[0115] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0116] In the present application, when the client performs model training, the global model can be trained through a loss function or through a BP algorithm to obtain a trained global model.

[0117] 7. Federated learning (FL)

[0118] A distributed machine learning algorithm that uses multiple clients, such as mobile devices or edge servers, and servers to collaboratively complete model training and algorithm updates without leaving the domain to obtain a trained global model. It can be understood that in the process of machine learning, each participant can use the data of other parties for joint modeling. All parties do not need to share data resources, that is, data joint training is performed without leaving the local area to establish a shared machine learning model.

[0119] First, the embodiments of the present application can be applied to a model processing device (eg, a client, a cloud) or a federated learning system. The following first introduces the federated learning system provided by the present application.

[0120] See also Figure 2 , a schematic diagram of the architecture of a federated learning system provided in this application. The system (or cluster for short) may include multiple servers, which can establish connections with each other, that is, the servers can also communicate with each other. Each server can communicate with one or more clients, and the clients can be deployed in various devices, such as mobile terminals or servers, etc. Figure 2 The client 1, client 2, ..., client N-1 and client N etc. are shown in FIG.

[0121] Specifically, the servers or the servers and the clients can interact through any communication mechanism / communication standard communication network, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes, but is not limited to, a fifth generation mobile communication technology (5th-Generation, 5G) system, a long term evolution (long term evolution, LTE) system, a global system for mobile communication (global system for mobilecommunication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wirelessfidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near fieldcommunication, NFC) in any one or more combinations. The wired network may include a fiber optic communication network or a network composed of coaxial cables, etc.

[0122] Generally, the client can be deployed in various servers or terminals. The client mentioned below may also refer to a server or terminal on which the client software program is deployed. The terminal may include a mobile terminal or a fixedly installed terminal, etc. For example, the terminal may specifically include a mobile phone, a tablet, a personal computer (PC), a smart bracelet, a speaker, a television, a smart watch or other terminals, etc.

[0123] When conducting federated learning, each server can send the model to be trained to the client that has established a connection with it. The client can train the model using the training samples stored locally and feed back the parameters and other data of the trained model to the server. After receiving the trained model or models fed back by one or more clients, the server can prune the received model or models and aggregate the data of the pruned model or models to obtain aggregated data, which is equivalent to the aggregated model. After the stop condition is met, the final model can be output to complete the federated learning.

[0124] Usually, in order to solve the problem of long transmission delay caused by the long distance between the client and the server, an intermediate layer server (referred to as an edge server in this application) is generally introduced between the server and the client to form a multi-layer architecture, namely, a client-edge server-cloud server architecture, thereby reducing the transmission delay between the client and the federated learning system through the edge server.

[0125] Specifically, the federated learning system to which the federated learning method provided in the present application can be applied may include a variety of topological relationships. For example, the federated learning system may include a two-layer or more-layer architecture. Some possible architectures are exemplarily introduced below.

[0126] 1. Two-tier architecture

[0127] like Figure 3 As shown, a structural diagram of a federated learning system provided by this application.

[0128] The federated learning system includes a two-layer architecture formed by a server and a client. The server can directly establish a connection with one or more clients.

[0129] During the federated learning process, the server sends the global model to one or more clients that have established connections with it.

[0130] In one possible implementation, the client uses locally stored training samples to train the received global model and feeds the trained global model back to the server. The server performs pruning based on the received trained global model and updates the locally stored global model to obtain the final global model.

[0131] In another possible implementation method, the client uses locally stored training samples to train and prune the received global model, and feeds the pruned global model back to the server. The server updates the locally stored global model based on the received trained and pruned global model to obtain the final global model.

[0132] Second and third-tier architecture

[0133] like Figure 4 As shown, a structural diagram of a federated learning system provided by this application.

[0134] Among them, the federated learning system includes one or more cloud servers, one or more edge servers and one or more clients, forming a three-layer architecture of cloud server-edge server-client.

[0135] In the system, one or more edge servers are connected to the cloud server, and one or more clients are connected to the edge servers.

[0136] During the federated learning process, the cloud server sends the locally saved global model to the edge server, and then the edge server sends the global model to the client connected to it.

[0137] In one possible implementation, the client uses locally stored training samples to train the received global model and feeds the trained global model back to the edge server. The edge server prunes the received trained global model and uses the pruned global model to update the locally stored global model. The edge server then feeds back the updated global model to the cloud server to complete federated learning.

[0138] In another possible implementation method, the client uses locally stored training samples to train and prune the received global model, and feeds the trained and pruned global model back to the edge server. The edge server updates the locally stored global model based on the received trained global model, and feeds the updated global model back to the cloud server to complete federated learning.

[0139] In one possible implementation method, the client uses locally stored training samples to train the received global model and feeds back the trained global model to the edge server. The edge server updates the locally stored global model based on the received trained global model and feeds back the updated global model to the cloud server. The cloud server then prunes the received global model to obtain the pruned global model, thus completing federated learning.

[0140] The model pruning process can be performed on the client, on the edge server, or on the cloud server. In addition to the above example of performing model pruning only on the client, edge server, or cloud server, pruning can also be performed in multiple links during federated learning. For example, when the client trains the model, pruning is performed before sending it to the edge server. When the edge server aggregates the model, pruning is also performed before sending it to the cloud server for processing. The specific details are not limited here.

[0141] 3. Architecture above three layers

[0142] like Figure 5 As shown, this is a schematic diagram of the structure of another federated learning system provided by this application.

[0143] The federated learning system includes an architecture of more than three layers, one of which includes one or more cloud servers, and multiple edge servers form an architecture of two or more layers, such as one or more upstream edge servers forming a layer of architecture, each upstream edge server is connected to one or more downstream edge servers. Each edge server in the last layer of architecture formed by the edge server is connected to one or more clients, so that the client forms a layer of architecture.

[0144] In the process of federated learning, the cloud server at the most upstream sends the latest global model stored locally to the edge server at the next layer, and then the edge server sends the global model to the next layer layer by layer until it is sent to the client. After receiving the global model sent by the edge server, the client uses the locally stored training samples to train and prune the received global model, and feeds back the trained and pruned global model to the edge server at the previous layer. Then, the edge server at the previous layer updates the locally stored global model based on the received trained global model, and then uploads the updated global model to the edge server at the next layer, and so on, until the second-layer edge server uploads the updated global model to the cloud server, and the cloud server updates the local global model based on the received global model to obtain the final global model and complete federated learning. It can be understood that here only the client training and pruning the model is used as an example for explanation. Similar to the above three-layer architecture, the pruning process can be in any layer in the federated learning system, and is not limited here.

[0145] It should be noted that in this application, for each device in the federated learning architecture, the direction of data transmission to the cloud server is called upstream, and the direction of data transmission to the client is called downstream. Figure 3 As shown in , the server is the upstream device of the client, and the client is the downstream device of the server, such as Figure 4 As shown, the cloud server can be called an upstream device of the edge server, the client can be called a downstream device of the edge server, and so on.

[0146] In addition, a brief introduction is given to the scenarios in which the models in the embodiments of the present application (for example, neural network models, first models, second models, third models, etc.) are applied. The model can be applied to the aforementioned smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, and any other scenarios that require neural network models to classify, identify, predict, recommend, translate, encode, decode, and other scenarios for input data such as text, images, or voice.

[0147] The above introduces the application scenarios of the federated learning system and model provided in this application. The following introduces the detailed steps executed by each device in the federated learning system.

[0148] In the embodiment of the present application, in the federated learning system, depending on whether the pruning step is performed by an upstream device (cloud server or edge server) or by a client, it can be divided into two cases, which are described below respectively:

[0149] First, the upstream device performs the pruning step.

[0150] See also Figure 6, an embodiment of the federated learning method provided in an embodiment of the present application includes steps 601 to 608.

[0151] Step 601: The upstream device sends a neural network model to the client. Correspondingly, the client receives the neural network model sent by the upstream device.

[0152] The upstream device in the embodiment of the present application may be the aforementioned Figure 2-Figure 5 For example, the upstream device may be a server in the federated learning system. Figure 2 Any one of the multiple servers shown in , or the aforementioned Figure 3 Any server in the two-tier architecture shown in Figure 4 Any of the cloud servers or edge servers shown in , or Figure 5 Any one of the cloud servers or edge servers shown in the description. The number of the client can be one or more. If the upstream device establishes a connection with multiple clients, the upstream device can send a neural network model to each client.

[0153] The above-mentioned neural network model can be a model stored locally by the upstream device, such as a global model stored locally by a cloud server, or the upstream device can receive a model sent by other servers and save the received model locally or update the locally stored model. Specifically, the upstream device can send structural parameters of the neural network model (such as the width, depth or convolution kernel size of the neural network, etc.) or initial weight parameters to the client. Optionally, the upstream device can also send training configuration parameters to the client, such as learning rate, number of epochs or categories in the security algorithm, so that the client that finally performs training can use the training configuration parameters to train the neural network model.

[0154] For example, when the upstream device is a cloud server, the neural network model may be a global model stored on the cloud server. For ease of distinction, the global model stored on the cloud server will be referred to as a cloud-side model below.

[0155] For another example, when the upstream device is an edge server, the neural network model can be a local model stored on the edge server, or called an edge server model. After the edge server receives the model sent by the upper-layer edge server or cloud server, the received model is used as the edge server model or the existing edge server model is updated to obtain a new edge server model, and the new edge server model (i.e., neural network model) is sent to the client. It should also be noted that when the upstream device is an edge server, the upstream device can directly send the edge server model (or neural network model) to the client.

[0156] In the embodiments of the present application, the neural network models mentioned, such as the first model, the second model or the third model, etc., can specifically include convolutional neural networks (CNN), deep convolutional neural networks (DCNN), recurrent neural networks (RNN) and other neural networks. The model to be learned can be determined according to the actual application scenario, and the present application does not limit this.

[0157] Optionally, the upstream device can actively send the neural network model to the client connected to it, or it can send the neural network model to the client at the request of the client. For example, if the upstream device is an edge server, the client can send a request message to the edge server to request to participate in federated learning. After receiving the request message, if the edge server confirms that the client is allowed to participate in federated learning, it can send the neural network model to the client. For another example, if the upstream device is a cloud server, the client can send a request message to the cloud server to request to participate in federated learning. After receiving the request message and confirming that the client is allowed to participate in federated learning, the cloud server can send the locally stored cloud-side model to the edge server. The edge server updates the local network model based on the received model to obtain the neural network model, and sends the neural network model to the client.

[0158] Step 602: The client uses the training data as input and trains the neural network model according to the first loss function to obtain a first model.

[0159] After receiving the neural network model sent by the upstream device, the client can update the locally stored end-side model based on the neural network model, for example, by replacing, weighted fusion, etc. to obtain a new neural network model. The new neural network model is trained using the training data with label values ​​and the first loss function to obtain the first model.

[0160] The training data in the embodiments of the present application may be of various types or forms, specifically related to the scenarios in which the model is applied. For example: when the role of the model is audio recognition, the specific form of the training data may be audio data, etc. For another example: when the role of the model is image classification, the specific form of the training data may be image data, etc. For another example: when the role of the model is to predict speech, the specific form of the training data may be text data, etc. It can be understood that the above-mentioned cases are just examples, and are not necessarily a one-to-one correspondence. For example, for audio recognition, the specific form of the training data may also be image data or text data, etc. (for example: if applied to the picture-playing voice scene in the field of education, the role of the model is to recognize the voice corresponding to the image, then the specific form of the training data may be image data). In practical applications, there are other scenarios, for example: when the role of the model is a movie recommendation scene, the training data may be a word vector corresponding to the movie, etc. In some application scenarios, the above-mentioned training data may also include data of different modalities at the same time. For example, in an autonomous driving scenario, the training data may include image / video data collected by a camera, and may also include voice / text data indicated by a user, etc. The specific form or type of the training data in the embodiments of the present application is not limited.

[0161] The client can use the locally stored training samples to train the neural network model (or the new neural network model) to obtain the first model. For example, the client can be deployed in a mobile terminal, which can collect a large amount of data during operation. The client can use the collected data as training samples to perform personalized training on the neural network model to obtain a personalized model of the client.

[0162] Among them, the process of training the neural network model by the client (taking one of the first clients as an example) can specifically include: taking the training data as the input of the neural network model, training the neural network model with the goal of reducing the value of the first loss function to obtain the first model. The first loss function is used to indicate the difference between the output value and the label value of the neural network model. Further, the neural network is trained using the training data and the optimization algorithm to obtain the first model. The first loss function can be understood as a data loss function (referred to as training loss).

[0163] The first loss function in the embodiment of the present application can be a mean square error loss, or a cross entropy loss, etc., which can be used to measure the difference between the output value of the neural network model and the label value (or true value).

[0164] The above-mentioned optimization algorithm can be a gradient descent method, a Newton method, an adaptive moment estimation method, or other optimization algorithms that can be used in machine learning. The specifics are not limited here, and the following description is taken as an example of the gradient algorithm.

[0165] Optionally, a specific form of the gradient algorithm is as follows:

[0166] Formula 1:

[0167] Among them, v i and They represent the neural network model parameters before and after the update in the nth round of training of federated learning, γ is the learning rate of the gradient descent optimization algorithm or the step size of each update. f is the first loss function (the specific form can be the above-mentioned mean square error loss, cross entropy loss, etc.). is the gradient of f, for example, obtained by taking the derivative of f.

[0168] Optionally, the above formula is only an example of a gradient algorithm. In practical applications, the gradient algorithm may also be other types of formulas, which are not specifically limited here.

[0169] In addition, in the above training process, all or part (e.g., slices) of the training data can be used. Generally, a slice method is used, and each iteration uses a slice (batch) of data. To calculate the loss function and update the gradient value:

[0170] Formula 2:

[0171] in, Represents a data set of a slice in the data set stored by the i-th client (i.e. the training data mentioned above) A set of training data in represents the input data for neural network model training, Represents input data The corresponding true label (or label value).

[0172] Optionally, the above formula 2 is only an example of calculating the gradient. In practical applications, the gradient can also be calculated by other types of formulas, which are not specifically limited here.

[0173] The first model in the embodiment of the present application can be the neural network model in the above-mentioned training process, or it can be the first model after the value of the first loss function is less than the first threshold. In other words, the first model can be the neural network model in the above-mentioned training process, or it can be a model obtained after the training is completed based on the local data set of the client. The specific details are not limited here.

[0174] Step 603: The client sends the first model to the upstream device. Correspondingly, the upstream device receives the first model sent by the client.

[0175] Optionally, the client sends the first model or information of the first model, such as weight parameters, gradient parameters, etc. to the upstream device. Correspondingly, the upstream device receives the first model sent by the client.

[0176] Step 604: The upstream device prunes the first model based on the second loss function and the constraint condition to obtain a second model.

[0177] After the upstream device receives the first model sent by the client, it can prune the first model based on the second loss function and the constraint condition to obtain the second model.

[0178] Optionally, after receiving the first model sent by the client, the substructure of the first model can be determined first, and then the first model can be pruned on the substructure. The substructure includes at least two neurons, and the substructure can be set according to actual needs. The substructure can be a channel, feature map, network layer, subnetwork, or other predefined network structure composed of multiple neurons of the neural network model; when the neural network model is a convolutional neural network, the substructure can also be a convolution kernel. In short, a substructure can be regarded as a functional whole. Pruning the substructure during pruning means pruning all neurons included in the substructure. By pruning the model substructure, the model can be compressed from the model structure, which is convenient for the implementation of underlying hardware acceleration.

[0179] The above-mentioned second loss function can be understood as a sparse loss function (abbreviated as sparse loss), which includes a difference term and a first sparse term, wherein the difference term is used to represent the difference in parameters between the first model and the second model. The first sparse term is used to prune at least one of the multiple substructures of the first model. The constraint condition is used to constrain the accuracy of the second model to be no less than that of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value.

[0180] Optionally, a specific form of the second loss function is as follows:

[0181] Formula 3:

[0182] Where n is the number of iterations, n is a positive integer, is the difference term, is the first sparse item. || ||2 is the L2 norm, V n is the parameter of the first model, W nis a parameter of the second model, λ is a hyperparameter used to adjust the weight of the first sparse item, and the hyperparameter in the embodiment of the present application can take any non-negative real number. For example, when λ=0, it means that the weight of the sparse item is 0, that is, the substructure is not required to be sparse during the training process, which is usually applicable to the scenario where the first model is small, the transmission communication cost is small, and the substructure of the first model does not need to be sparse. is the i-th substructure in the second model.

[0183] It can be understood that the above second loss function is only an example. In practical applications, there may be other forms of second loss functions. For example, the L2 norm in the first sparse term can be replaced by the L0 norm, the L1 norm, the approximation of the L0 norm, the approximation of the L1 norm, the L0 and L p Mixed norm, L1 and L p Mixed norms and the like can be used to guide the function of variable sparsity. For another example, the difference term can be replaced by any other function that measures the similarity or distance between two variables, such as Euclidean distance, Mahalanobis distance, mutual information, cosine similarity, inner product or norm. The selection of the difference term and the first sparse term can be adapted according to the actual application scenario, and is not limited here.

[0184] Optionally, if there are multiple clients, that is, the upstream device receives multiple first models, the upstream device can use the same sparse loss function and constraints to prune the multiple first models separately to obtain multiple second models, and the multiple second models correspond one to one to the multiple first models; of course, the upstream device can also group the multiple first models according to the type of substructure, and the sparse loss function corresponding to each group can be the same or different, and then combine the multiple first models of the same group for pruning; or, the upstream device can also combine all the received first models for pruning together. The specific pruning method of multiple first models is not limited here. The following describes the specific pruning process using a first model as an example.

[0185] The pruning direction of the first model in the embodiment of the present application can be understood as the descent direction of Wn obtained by calculating the second loss function, and the training data direction of the first model can be understood as the descent direction of Vn obtained by calculating the first loss function. For example, for gradient descent, the pruning direction of the first model can be understood as the gradient direction of Wn obtained by deriving the second loss function, and the training data direction of the first model can be understood as the gradient direction of Vn obtained by deriving the first loss function.

[0186] In the embodiment of the present application, there are multiple ways to prune the first model based on the second loss function and the constraint conditions, which are described by way of example below:

[0187] The first one is to introduce the update coefficient s iThe first model is pruned in this way.

[0188] In one possible implementation, an update coefficient is calculated based on a constraint condition, and the update coefficient is used to adjust the direction of the first sparse term. The first sparse term in the second loss function is updated using the update coefficient to obtain a third loss function, and the third loss function includes a difference term and a second sparse term, and the second sparse term is updated based on the update coefficient and the first sparse term. After obtaining the third loss function, the first model is pruned based on the third loss function to obtain the second model. Specifically, a subspace can be determined according to the constraint condition, and the accuracy of the second model in the subspace is the same as the accuracy of the first model.

[0189] Optionally, a form of the third loss function is as follows:

[0190] Formula 4:

[0191] Among them, s i is the update coefficient, by adjusting s i To satisfy the constraint condition. The rest of the description can refer to the above description of the second loss function, which will not be repeated here.

[0192] Optionally, the above formula 4 is only an example of the third loss function, and in practical applications, it can be set according to the description of the above second loss function, and is not specifically limited here.

[0193] In order to understand more intuitively i , described below with reference to the accompanying drawings.

[0194] For example, see Fig. 7A , assuming that the first model parameter V n It includes three substructures, namely V1, V2 and V3. Fig. 7A The pruning direction is shown in the figure. First, V n Grouping is performed, assuming that it is divided into two groups (or understood as two substructures): a and b. Where a = {1, 2}, b = {3}. In this way, for the first model parameter V n Pruning can be understood as a and / or V b Carry out pruning. Fig. 7A To V a The pruning is described as an example, that is, the first model is pruned until V a becomes 0. Among them, the descending direction of the second loss function is Fig. 7A Pruning direction before correction.

[0195] The above example is expressed in mathematical expressions as follows:

[0196] Vn :=(V1,V2,V3); a=(1,2),b={3};

[0197] V a =(V1,V2);V b =(V3);

[0198]

[0199]

[0200]

[0201] Among them, E() can be understood as the group normalization operator. In addition, the Hessian matrix There are many eigenvalues ​​close to 0, and perturbing the model parameters in the directions corresponding to these eigenvalues ​​will hardly change the accuracy of the model, such as Fig. 7A The P0 shown here represents the subspace generated by these directions ( Fig. 7A In this example, P0 is a plane. The accuracy of the first model in the plane is the same as that of the second model. Π0 represents the projection operator onto the subspace P0, s a is the s corresponding to group a i , the calculation method can refer to the following formula:

[0202] Formula 5:

[0203] It can be understood that the above calculation s i The formula is just an example. In practical applications, there may be other types of formulas, which are not limited here.

[0204] For example, the first model includes an input layer, hidden layers 1-3, and an output layer. The comparison diagram of the model before pruning (i.e., the first model) and the model after pruning (i.e., the second model) is shown. Figure 7B , where the substructure can be a channel, feature map, network layer, subnetwork, or other predefined network structure composed of multiple neurons of a neural network model; when the neural network model is a convolutional neural network, the substructure can also be a convolution kernel. In short, a substructure can be regarded as a functional whole. When pruning a substructure, it means pruning all neurons included in the substructure. Figure 7B The description is only based on an example that a substructure includes two neurons. It is understandable that a substructure may include more or fewer neurons. Figure 7B It can be seen that the pruned model has two fewer substructures than the model before pruning. Figure 7BJust to more intuitively describe the changes of the model before and after pruning, the number of pruned substructures can be one or more, and is not limited here.

[0205] Another possible implementation method is to use s i Adjust V b V a Prune the branches so that the predicted training data direction V′ n The direction obtained after pruning is the corrected pruning direction, and the first model is pruned based on the direction, which has little effect on the accuracy of the first model. For example, see Figure 8 .

[0206] The second method is to correct the pruning direction of the first model based on the constraint condition to obtain a corrected pruning direction, and prune the first model based on the corrected pruning direction.

[0207] Gradient Direction Gradient Direction In the embodiment of the present application, there are multiple ways to correct the pruning direction of the first model based on the constraint conditions, which are described below:

[0208] 1. Determine the pruning direction of the first model based on the constraints.

[0209] In addition to introducing the update coefficient s in the first method above i In addition, the better second model (i.e. the pruned model) can be determined by the following method:

[0210] Optionally, assuming that pruning is performed while training data, another form of the gradient algorithm in step 602 can be as follows:

[0211] Formula 6:

[0212] Formula 7:

[0213] Where n is the number of iterations, Z n+1 is the nth group of training data, and Z n+1 ∈Z, γ is the learning rate of the gradient descent optimization algorithm or the step size of each update. f is the first loss function (the specific form can be the above-mentioned mean square error loss, cross entropy loss, etc.). is the gradient of f, for example, obtained by taking the derivative of f. The explanation of the remaining parameters can refer to the explanation of the gradient algorithm in the above step 602, which will not be repeated here. g(n,γ) is the function for adjusting the pruning direction, c and μ are two hyperparameters for controlling the intensity of the sparse penalty, μ∈(0.5,1], i represents the i-th substructure,

[0214] In other words, update V using the training data Z nGet the first model V n+1 , and based on V n+1 Update W n+1 , and then use W n+1 Replace V in the above gradient algorithm n , thereby continuously pruning the first model until a second model that meets actual needs is obtained (for example, the number of iterations reaches a threshold or the precision / accuracy of the second model reaches a threshold, etc.).

[0215] Optionally, the above formula 6 and formula 7 are only examples of solving the better / optimal second model. In practical applications, there may be other methods, which are not limited here. For example, the above formula 7 can be replaced by the following formula 8:

[0216] Formula 8:

[0217] in,() + It means that only the numbers greater than 0 are taken, and the numbers less than 0 are set to zero.

[0218] 2. Determine a subspace where the accuracy of the first model is consistent with the accuracy of the second model, and correct the pruning direction of the first model based on the subspace.

[0219] Based on the constraint condition, a subspace (such as P0 described above, which will not be repeated here) in which the accuracy of the first model is the same (or consistent) as the accuracy of the second model is determined, and the pruning direction of the first model is corrected based on the subspace.

[0220] In the embodiment of the present application, there are multiple methods for correcting the pruning direction of the first model based on the subspace, which are described below respectively:

[0221] 2.1. Project the pruning direction of the first model into the subspace to obtain the corrected pruning direction.

[0222] Optionally, the pruning direction of the first model may be projected into the subspace to obtain a corrected pruning direction, and the first model may be pruned according to the corrected pruning direction, thereby ensuring that the accuracy of the first model before pruning is the same as the accuracy of the second model after pruning.

[0223] For example, Fig. 7A As shown in , after determining the subspace P0, the pruning direction before correction is projected to the subspace P0, so as to obtain Fig. 7A The corrected pruning directions are shown.

[0224] 2.2. The pruning direction of the first model is mirrored based on the subspace to obtain the corrected pruning direction.

[0225] Optionally, after determining the subspace, a corrected pruning direction can be obtained by mirroring the subspace, and the first model is pruned according to the corrected pruning direction, which can ensure that the accuracy of the first model before pruning is close to the accuracy of the second model after pruning. In other words, the value of the first loss function of the second model is smaller than the value of the first loss function of the first model.

[0226] For example, Fig. 9 As shown in, after determining the subspace P0, the pruning direction before correction is mirrored based on the subspace P0, so as to obtain Fig. 9 The mirrored pruning direction (i.e., the corrected pruning direction) is shown.

[0227] 3. If the angle between the pruning direction of the first model and the training data direction of the first model is an obtuse angle, adjust it to an acute angle.

[0228] Optionally, before pruning the first model, the angle between the training data direction of the first model and the pruning direction of the first model can be determined. If the angle is an obtuse angle (i.e., the pruning direction is opposite to the data training direction, which is also the reason why the model needs to be fine-tuned after pruning in the prior art), the pruning direction of the first model is adjusted to satisfy the angle between the corrected pruning direction and the training data direction of the first model is an acute angle or a right angle. Then, the first model is pruned according to the corrected pruning direction to ensure that the accuracy of the first model before pruning is similar to the accuracy of the second model after pruning. In other words, the value of the first loss function of the second model is less than or equal to the value of the first loss function of the first model.

[0229] For example, Fig.10 As shown, the pruning direction before correction may be at an obtuse angle to the data training direction of the first model, that is, the two directions are inconsistent. In order to ensure that the pruning direction is consistent with the data training direction in a large direction and reduce the steps of subsequent fine-tuning of the model, the pruning direction is adjusted to a direction range in which the angle between the pruning direction and the data training direction of the first model is an acute angle or a right angle.

[0230] It is understandable that there are many ways to prune the first model based on the second loss function and constraints. The above methods are just examples. In actual applications, there are other ways to prune the first model based on the second loss function and constraints. For example, the first model can be repeatedly randomly optimized until the pruned second model meets the constraints, etc. The specifics are not limited here.

[0231] Step 605: The upstream device aggregates the multiple second models to obtain a third model. This step is optional.

[0232] After the upstream device obtains multiple second models based on step 605, it can aggregate the multiple second models to obtain a third model, and use the third model as a global model.

[0233] Optionally, the third model is used as the neural network model in step 601 to repeatedly execute steps 601 to 605. In other words, steps 601 to 605 are counted as one iteration, and steps 601 to 605 in the embodiment of the present application can be executed multiple times. Furthermore, if steps 601 to 605 are executed in a loop, a stop condition of steps 601 to 605 (which can also be understood as a stop condition for pruning updates) can be set, and the stop condition can be the number of loops, the sparsity of the model after pruning reaches a certain threshold, etc., which is not limited here.

[0234] In the embodiment of the present application, the above-mentioned aggregation method can be to calculate the weighted average of multiple second models, or to calculate the average of multiple second models, etc., which is not limited here.

[0235] Optionally, if the data of each client's model is identically distributed, the accuracy of the aggregated third model is higher than that of the first model; if the data of each client's model is asynchronous, the accuracy of the aggregated third model may be lower than that of the first model.

[0236] Optionally, after the upstream device aggregates multiple second models to obtain a third model, it can also prune the third model through a loss function and a constraint condition to obtain a fourth model. If the upstream device is a cloud server, the upstream device can send the fourth model to an edge server or a client. If the upstream device is an edge server, the edge server can send the fourth model to the cloud service so that the upper server can process the fourth model.

[0237] Step 606, whether the first preset condition is met, if yes, the training ends. If no, the above steps are repeated with the third model as the neural network model. This step is optional.

[0238] Optionally, after the upstream device obtains the third model, it can determine whether the first preset condition is met. If so, the training process of the model ends. After the training process ends, steps 607 and 608 can be executed, or the third model can be sent to the upstream edge server or cloud server. In the embodiment of the present application, the steps performed after the training is completed are not limited.

[0239] If not (not satisfied), the third model is used as the neural network model in step 601 and the execution is repeated Figure 6The steps from step 601 to step 606 shown (or understood as one iteration). That is, the upstream device sends the third model to the client, and the client uses the local data set to train the third model to obtain the fourth model, and sends the fourth model to the upstream device. The upstream device receives multiple fourth models sent by multiple clients, and the upstream device prunes the multiple fourth models based on the sparse loss function and the constraint conditions to obtain multiple fifth models. Aggregate the multiple fifth models to obtain the sixth model, and then determine whether the first preset condition is met. If so, the training ends. If not, the sixth model is used as the neural network model of the first iteration or the third model of the second iteration and repeated. Figure 6 The steps shown are repeated until the first preset condition is met.

[0240] Optionally, if the first preset condition is not met, during the iteration process, the upstream device may also send a first indication message to the client to indicate that the training is not completed, so that the client determines whether to continue training the model according to the first indication message.

[0241] Among them, the first preset condition may be that the third model converges, the number of cycles from step 601 to step 605 reaches a threshold, the global model accuracy reaches a threshold, etc., and the specific conditions are not limited here.

[0242] Step 607: The upstream device sends the third model and the second indication information to the client. This step is optional.

[0243] Optionally, if the first preset condition is met, the upstream device sends the third model and second indication information to the client, and the second indication information is used to indicate that the training process of the third model is completed.

[0244] Step 608: The client uses the third model to perform reasoning according to the second indication information. This step is optional.

[0245] Optionally, after receiving the third model and the second indication information, the client can learn, based on the second indication information, that the training process of the third model has ended, and use the third model for reasoning.

[0246] In this embodiment, in the federated learning scenario, the client trains the neural network model through local data to obtain the first model, and sends the first model to the upstream device. The upstream device then prunes the first model according to the constraints. On the one hand, the upstream device considers the constraints during the pruning process, so that the accuracy of the pruned second model is higher than or equal to the first model, which can also be understood as pruning that does not increase the training loss, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process. On the other hand, by pruning the model substructure, the model can be compressed from the model structure, which is convenient for the implementation of underlying hardware acceleration. And the model size is reduced, reducing the storage and computing overhead of the client.

[0247] Second, the client performs the pruning step.

[0248] See also Fig.11 , another embodiment of the federated learning method provided in an embodiment of the present application, the embodiment includes steps 1101 to 1108.

[0249] Step 1101: The upstream device sends a neural network model to the client. Correspondingly, the client receives the neural network model sent by the upstream device.

[0250] Step 1102: The client uses the training data as input and trains the neural network model according to the first loss function to obtain a first model.

[0251] Steps 1101 and 1102 in this embodiment are similar to those in the above Figure 6 Step 601 in the illustrated embodiment is similar to step 602 and will not be described in detail herein.

[0252] Step 1103: The client prunes the first model based on the second loss function and the constraint condition to obtain a second model.

[0253] Step 1103 performed by the client in this embodiment is the same as the aforementioned Figure 6 The step 604 performed by the upstream device in the illustrated embodiment is similar and will not be described in detail here.

[0254] Optionally, step 1102 and step 1103 can be regarded as an iterative process. After the client obtains the second model, it can determine whether the second preset condition is met. If so, step 1104 is executed. If not (not met), the third model is used as the neural network model in step 1102 to repeat steps 1102 and 1103 (or understood as one iteration). That is, the training data is used as input, and the third model is trained according to the first loss function to obtain the trained third model. And the trained third model is pruned based on the constraints to obtain the seventh model, and then it is determined whether the first preset condition is met. If so, step 1104 is executed. If not, the seventh model is used as the neural network model of the first iteration or the third model of the second iteration to repeat steps 1102 and 1103 until the second preset condition is met.

[0255] Among them, the second preset condition can be that the model converges, the number of cycles of step 1102 and step 1103 reaches a threshold, the model accuracy reaches a threshold, etc., and the specific conditions are not limited here.

[0256] Optionally, step 1103 may also be regarded as an iterative process. After the client obtains the second model, it may determine whether the third preset condition is met. If so, step 1104 is executed. If not, the third model is used as the neural network model in step 1102 and step 1102 is repeatedly executed until the third preset condition is met.

[0257] Among them, the third preset condition can be model convergence, the number of cycles of step 1102 reaches a threshold, the model accuracy reaches a threshold, etc., and the specific conditions are not limited here.

[0258] Step 1104: The client sends the second model to the upstream device. Correspondingly, the upstream device receives the second model sent by the client.

[0259] Optionally, the client sends the second model or information of the second model, such as weight parameters, gradient parameters, etc. to the upstream device. Correspondingly, the upstream device receives the second model sent by the client.

[0260] Step 1105: The upstream device aggregates the multiple second models to obtain a third model.

[0261] Step 1105 in this embodiment is similar to the above Figure 6 Step 605 in the illustrated embodiment is similar and will not be described in detail here.

[0262] Step 1106, whether the first preset condition is met, if yes, the training ends. If no, the above steps are repeated with the third model as the neural network model. This step is optional.

[0263] Optionally, after the upstream device obtains the third model, it can determine whether the first preset condition is met. If so, the training ends. After the training process ends, steps 1107 and 1108 can be executed, or the third model can be sent to the upstream edge server or cloud server. In the embodiment of the present application, the steps performed after the training ends are not limited.

[0264] If not (not satisfied), the third model is used as the neural network model in step 601 and the execution is repeated Figure 6 The steps shown (or understood as one iteration). That is, the upstream device sends the third model to the client, and the client uses the local data set to train the third model to obtain the fourth model, and prunes multiple fourth models based on the sparse loss function and constraints to obtain multiple fifth models. Send the fifth model to the upstream device. The upstream device receives multiple fifth models sent by multiple clients, aggregates multiple fifth models to obtain the sixth model, and then determines whether the first preset condition is met. If it is met, the training ends. If not, the sixth model is used as the neural network model of the first iteration or the third model of the second iteration and is repeatedly executed. Figure 6 The steps shown are repeated until the first preset condition is met.

[0265] Optionally, if the first preset condition is not met, during the iteration process, the upstream device may also send a first indication message to the client to indicate that the training is not completed, so that the client determines whether to continue training the model according to the first indication message.

[0266] Among them, the first preset condition may be that the third model converges, the number of cycles from step 1101 to step 1105 reaches a threshold, the global model accuracy reaches a threshold, etc., and the specific conditions are not limited here.

[0267] Step 1107: The upstream device sends second indication information to the client. This step is optional.

[0268] Step 1108: The client uses the third model to perform reasoning according to the second indication information. This step is optional.

[0269] Steps 1107 and 1108 in this embodiment are similar to those in the above Figure 6 Step 607 and step 608 in the illustrated embodiment are similar and will not be described in detail here.

[0270] This embodiment is similar to the aforementioned Figure 6 The differences in the embodiments shown are mainly that Figure 6 The pruning step in the illustrated embodiment is performed by an upstream device, and the pruning step in this embodiment is performed by a client.

[0271] In this embodiment, in the federated learning scenario, the client trains the neural network model through local data to obtain the first model, then prunes the first model according to the constraints to obtain the second model, and sends the second model to the upstream device, which aggregates the second model to obtain the global model. On the one hand, the upstream device considers the constraints during the pruning process, so that the accuracy of the pruned second model is higher than or equal to the first model, which can also be understood as pruning that does not increase the training loss, reducing the subsequent steps of fine-tuning the model accuracy, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process. On the other hand, by pruning the model substructure, the model can be compressed from the model structure, which is convenient for the implementation of underlying hardware acceleration. And the model size is reduced, reducing the storage and computing overhead of the client.

[0272] The above describes the application of the method in the embodiment of the present application to the federated learning scenario. The following describes a model processing method also provided in the embodiment of the present application. Fig.12 An embodiment of the model processing method provided in an embodiment of the present application can be executed by a model processing device (such as a client) or by a component of the model processing device (such as a processor, a chip, or a chip system, etc.), wherein the model processing device can be a cloud server or a client. The embodiment includes steps 1201 to 1203.

[0273] Step 1201, obtaining training data including label values.

[0274] In the embodiment of the present application, the training data based on the label value can be stored in other devices such as a server, and the model processing device obtains the training data through other devices such as a server. It can also be collected by the model processing device during operation, which is not limited here.

[0275] Step 1202, taking the training data as input, training the neural network model according to the first loss function to obtain a first model.

[0276] Step 1202 performed by the model processing device in this embodiment is the same as the aforementioned Figure 6 Step 602 performed by the client in the illustrated embodiment is similar and will not be described in detail here.

[0277] Step 1203: Prune the first model based on the second loss function and the constraint condition to obtain the second model.

[0278] Step 1203 performed by the model processing device in this embodiment is the same as the aforementioned Figure 6 The step 604 performed by the upstream device in the illustrated embodiment is similar and will not be described in detail here.

[0279] In this embodiment, the model processing device trains the neural network model through local data to obtain a first model, and then prunes the first model according to the constraint conditions to obtain a second model. The model processing device considers the constraint conditions during the pruning process, so that the accuracy of the pruned second model is higher than or equal to the first model, which can also be understood as pruning that does not increase the training loss and reduces the subsequent model fine-tuning process.

[0280] The above describes the model processing method and federated learning method in the embodiment of the present application. The following describes the model processing device and upstream device in the embodiment of the present application. Fig.13 , an embodiment of the model processing device in the embodiment of the present application includes:

[0281] An acquisition unit 1301 is used to acquire training data including label values;

[0282] A training unit 1302 is used to train a neural network model according to a first loss function using training data as input to obtain a first model, wherein the first model includes a plurality of substructures, and each of the plurality of substructures includes at least two neurons;

[0283] The pruning unit 1303 is used to prune the first model based on the second loss function and the constraint condition to obtain the second model, wherein the second loss function is used to indicate that at least one substructure among the multiple substructures is pruned, and the constraint condition is used to constrain the accuracy of the second model to be not lower than the accuracy of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value.

[0284] Optionally, the model processing device further includes:

[0285] A receiving unit 1304, configured to receive a neural network model sent by an upstream device;

[0286] The sending unit 1305 is configured to send the second model to an upstream device.

[0287] In this embodiment, the operations performed by each unit in the model processing device are the same as those described above. Figures 2 to 5 or Fig.11 Client or Fig.12 The steps and related descriptions performed by the model processing device in the illustrated embodiment are similar and will not be repeated here.

[0288] In this embodiment, the pruning unit 1303 considers the constraints based on the data loss function during the process of pruning the first model, which is equivalent to providing a direction for the pruning of the first model, so that the accuracy of the pruned second model is not lower than the accuracy of the first model, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, thereby improving the efficiency of the model pruning process while ensuring the accuracy of the pruned model.

[0289] See also Fig.14 , an embodiment of an upstream device in an embodiment of the present application, the upstream device may be the aforementioned cloud server or the aforementioned edge server, and the upstream device includes:

[0290] A sending unit 1401 is used to send a neural network model to multiple downstream devices, where the neural network model includes multiple substructures, and each of the multiple substructures includes at least two neurons;

[0291] A receiving unit 1402 is used to receive a plurality of first models from a plurality of downstream devices, where the plurality of first models are obtained by training a neural network model;

[0292] A pruning unit 1403 is used to prune the multiple first models respectively based on a loss function and a constraint condition, wherein the loss function is used to indicate that the substructures of the multiple first models are pruned, and the constraint condition is used to constrain the accuracy of each first model after pruning to be not less than the accuracy before pruning;

[0293] The aggregation unit 1404 is used to aggregate the plurality of pruned first models to obtain a second model.

[0294] In this embodiment, the operations performed by each unit in the upstream device are the same as those described above. Figures 2 to 11 The steps and related descriptions performed by the cloud server or edge server in the illustrated embodiment are similar and will not be repeated here.

[0295] In this embodiment, in the federated learning scenario, the client trains the neural network model through local data to obtain the first model, and sends the first model to the upstream device. The pruning unit 1403 prunes the first model according to the constraints. On the one hand, the pruning unit 1403 considers the constraints in the pruning process, so that the accuracy of the first model before and after pruning is similar, which can also be understood as pruning that does not increase the training loss, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process. On the other hand, the pruning unit 1403 can compress the model from the model structure by pruning the model substructure, which is convenient for the implementation of underlying hardware acceleration. And it reduces the model size and reduces the storage and computing overhead of the client.

[0296] See also Fig.15 Another embodiment of the upstream device in the embodiment of the present application, the upstream device may be the aforementioned cloud server or the aforementioned edge server, and the upstream device includes:

[0297] A sending unit 1501 is used to send a neural network model to multiple downstream devices, where the neural network model includes multiple substructures, and each of the multiple substructures includes at least two neurons;

[0298] A receiving unit 1502 is configured to receive a plurality of first models from the plurality of downstream devices, where the plurality of first models are obtained by training the neural network model;

[0299] An aggregation unit 1503, configured to aggregate the multiple first models to obtain a second model;

[0300] The pruning unit 1504 is used to prune the second model based on a loss function and a constraint condition, wherein the loss function is used to indicate that the substructure of the second model is to be pruned, and the constraint condition is used to constrain the accuracy of the second model after pruning to be no less than the accuracy before pruning.

[0301] In this embodiment, the operations performed by each unit in the upstream device are the same as those described above. Figures 2 to 11 The steps and related descriptions performed by the cloud server or edge server in the illustrated embodiment are similar and will not be repeated here.

[0302] In this embodiment, in the federated learning scenario, the client trains the neural network model through local data to obtain the first model, and sends the first model to the upstream device. The pruning unit 1504 prunes the second model according to the constraints. On the one hand, the pruning unit 1504 considers the constraints in the pruning process, so that the accuracy of the second model before and after pruning is similar, which can also be understood as pruning that does not increase the training loss, reducing the subsequent steps of adjusting the model accuracy through fine-tuning, thereby ensuring the accuracy of the pruned model while improving the efficiency of the model pruning process. On the other hand, the pruning unit 1504 can compress the model from the model structure by pruning the model substructure, which is convenient for the implementation of underlying hardware acceleration. And it reduces the model size and reduces the storage and computing overhead of the client.

[0303] The present application also provides a model processing device, such as Fig.16 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application (i.e., the aforementioned Figures 2 to 11 Client or Fig.12 The steps performed by the model processing device in the illustrated embodiment are similar to the related description). The model processing device may be any terminal device including a mobile phone, a tablet computer, etc., and the model processing device is a client, and the client is a mobile phone as an example:

[0304] Fig.16The block diagram shows a partial structure of a model processing device - a mobile phone provided in an embodiment of the present application. Fig.16 The mobile phone includes: a radio frequency (RF) circuit 1610, a memory 1620, an input unit 1630, a display unit 1640, a sensor 1650, an audio circuit 1660, a wireless fidelity (WiFi) module 1670, a processor 1680, and a power supply 1690. Those skilled in the art will appreciate that Fig.16 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0305] Combine the following Fig.16 A detailed introduction to the various components of the mobile phone:

[0306] The RF circuit 1610 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1680 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1610 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1610 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0307] The memory 1620 can be used to store software programs and modules. The processor 1680 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1620. The memory 1620 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1620 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0308] The input unit 1630 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1630 may include a touch panel 1631 and other input devices 1632. The touch panel 1631, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1631 or near the touch panel 1631 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1631 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1680, and can receive and execute commands sent by the processor 1680. In addition, the touch panel 1631 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1631, the input unit 1630 may also include other input devices 1632. Specifically, other input devices 1632 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0309] The display unit 1640 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1640 may include a display panel 1641. Optionally, the display panel 1641 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1631 may cover the display panel 1641. When the touch panel 1631 detects a touch operation on or near it, it is transmitted to the processor 1680 to determine the type of touch event. Subsequently, the processor 1680 provides a corresponding visual output on the display panel 1641 according to the type of touch event. Although in Fig.16 In the embodiment, the touch panel 1631 and the display panel 1641 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1631 and the display panel 1641 can be integrated to realize the input and output functions of the mobile phone.

[0310] The mobile phone may also include at least one sensor 1650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1641 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1641 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; As for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared, IMU, SLAM sensors, etc., they will not be repeated here.

[0311] The audio circuit 1660, the speaker 1661, and the microphone 1662 can provide an audio interface between the user and the mobile phone. The audio circuit 1660 can transmit the received audio data to the speaker 1661 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1662 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1660 and converted into audio data, and then the audio data is output to the processor 1680 for processing, and then sent to another mobile phone through the RF circuit 1610, or the audio data is output to the memory 1620 for further processing.

[0312] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1670. It provides users with wireless broadband Internet access. Fig.16 A WiFi module 1670 is shown, but it is understandable that it is not an essential component of the mobile phone.

[0313] The processor 1680 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1620, and calling data stored in the memory 1620, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1680 may include one or more processing units; preferably, the processor 1680 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1680.

[0314] The mobile phone also includes a power supply 1690 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1680 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption management.

[0315] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0316] In the embodiment of the present application, the processor 1680 included in the mobile phone can execute the aforementioned Figures 2 to 5 , Fig.11 Client or Fig.12 The functions of the model processing device in the illustrated embodiment will not be described in detail here.

[0317] See also Fig.17 , a schematic diagram of the structure of another upstream device provided by the present application. The upstream device may include a processor 1701, a memory 1702 and a communication interface 1703. The processor 1701, the memory 1702 and the communication interface 1703 are interconnected via a line. The memory 1702 stores program instructions and data.

[0318] The memory 1702 stores the aforementioned Figures 2 to 6 or Fig.11 The program instructions and data corresponding to the steps executed by the cloud server or edge server in the illustrated embodiment.

[0319] Processor 1701, used to execute the above Figures 2 to 6 or Fig.11Steps performed by a cloud server or edge server in the illustrated embodiment.

[0320] The communication interface 1703 can be used to receive and send data and to perform the aforementioned Figures 2 to 6 or Fig.11 The cloud server or edge server in the illustrated embodiment is associated with the steps of acquiring, sending, and receiving.

[0321] In one implementation, the upstream device may include a Fig.17 More or fewer components are merely illustrative in this application and are not limiting.

[0322] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0323] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0324] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated units may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.

[0325] When the integrated unit is implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0326] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

Claims

1. A model processing method, characterized in that: The method comprises: Acquire training data including label values, wherein the training data includes: image data, audio data or text data; Taking the training data as input, training a neural network model according to a first loss function to obtain a first model, wherein the first model includes a plurality of substructures, and each of the plurality of substructures includes at least two neurons; The first model is pruned based on a second loss function and a constraint condition to obtain a second model, wherein the second loss function is used to indicate that at least one of the multiple substructures is pruned, and the constraint condition is used to constrain the accuracy of the second model to be no less than the accuracy of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value.

2. The method according to claim 1, characterized in that The constraint condition is specifically used to constrain the angle between the descending direction of the first loss function and the descending direction of the second loss function to be less than or equal to 90 degrees.

3. The method according to claim 1, characterized in that The constraint condition is specifically used to constrain the value of the first loss function of the second model to be less than or equal to the value of the first loss function of the first model.

4. The method according to any one of claims 1 to 3, characterized in that The second loss function includes a first sparse term associated with a weight of at least one substructure of the plurality of substructures.

5. The method according to any one of claims 1 to 3, characterized in that The pruning of the first model based on the second loss function and the constraint condition to obtain the second model includes: The first model is randomly pruned at least once based on the second loss function until the second model obtained after pruning the first model satisfies the constraint condition.

6. The method according to any one of claims 1 to 3, characterized in that The method is applied to a client, the training data is local data of the client, and the method further includes: Receiving the neural network model sent by an upstream device; The second model is sent to the upstream device.

7. The method according to any one of claims 1 to 3, characterized in that The neural network model is used to classify and / or recognize image data.

8. A federated learning method, characterized in that: The method comprises: Sending a neural network model to a plurality of downstream devices, the neural network model comprising a plurality of substructures, each of the plurality of substructures comprising at least two neurons; Receiving a plurality of first models from the plurality of downstream devices, the plurality of first models being obtained by training the neural network model based on training data, the training data comprising: image data, audio data or text data; Pruning the multiple first models respectively based on a loss function and a constraint condition, wherein the loss function is used to indicate that the substructures of the multiple first models are pruned, and the constraint condition is used to constrain the accuracy of each first model after pruning to be no less than the accuracy before pruning; The pruned multiple first models are aggregated to obtain a second model.

9. A federated learning method, characterized in that: The method comprises: Sending a neural network model to a plurality of downstream devices, the neural network model comprising a plurality of substructures, each of the plurality of substructures comprising at least two neurons; Receiving a plurality of first models from the plurality of downstream devices, the plurality of first models being obtained by training the neural network model based on training data, the training data comprising: image data, audio data or text data; Aggregating the multiple first models to obtain a second model; The second model is pruned based on a loss function and constraints, wherein the loss function is used to indicate that the substructure of the second model is pruned, and the constraints are used to constrain the accuracy of the second model after pruning to be no less than the accuracy before pruning.

10. A model processing device, characterized in that: The device comprises: An acquisition unit, used to acquire training data including a label value, wherein the training data includes: image data, audio data or text data; A training unit, configured to take the training data as input, train a neural network model according to a first loss function, and obtain a first model, wherein the first model includes a plurality of substructures, and each of the plurality of substructures includes at least two neurons; A pruning unit, used for pruning the first model based on a second loss function and a constraint condition to obtain a second model, wherein the second loss function is used to indicate that at least one of the multiple substructures is pruned, and the constraint condition is used to constrain the accuracy of the second model to be no less than the accuracy of the first model, and the accuracy indicates the degree of difference between the output value of the model and the label value.

11. The device according to claim 10, characterized in that The constraint condition is specifically used to constrain the angle between the descending direction of the first loss function and the descending direction of the second loss function to be less than or equal to 90 degrees.

12. The device according to claim 10, characterized in that The constraint condition is specifically used to constrain the value of the first loss function of the second model to be less than or equal to the value of the first loss function of the first model.

13. The device according to any one of claims 10 to 12, characterized in that The second loss function includes a first sparse term associated with a weight of at least one substructure of the plurality of substructures.

14. The device according to any one of claims 10 to 12, characterized in that The pruning unit is specifically used to randomly prune the first model at least once based on the second loss function until the second model obtained after pruning the first model meets the constraint condition.

15. The device according to any one of claims 10 to 12, characterized in that The model processing device is a client, the training data is local data of the client, and the model processing device further includes: A receiving unit, configured to receive the neural network model sent by an upstream device; A sending unit is used to send the second model to the upstream device.

16. The device according to any one of claims 10 to 12, characterized in that The neural network model is used to classify and / or recognize image data.

17. An upstream device, characterized in that: The upstream device is applied to the federated learning method, and the upstream device includes: A sending unit, configured to send a neural network model to a plurality of downstream devices, wherein the neural network model includes a plurality of substructures, and each of the plurality of substructures includes at least two neurons; A receiving unit, configured to receive a plurality of first models from the plurality of downstream devices, wherein the plurality of first models are obtained by training the neural network model based on training data, wherein the training data includes: image data, audio data or text data; A pruning unit, configured to prune the plurality of first models respectively based on a loss function and a constraint condition, wherein the loss function is used to indicate that the substructures of the plurality of first models are to be pruned, and the constraint condition is used to constrain the accuracy of each of the first models after pruning to be no less than the accuracy before pruning; The aggregation unit is used to aggregate the plurality of pruned first models to obtain a second model.

18. An upstream device, characterized in that: The upstream device is applied to the federated learning method, and the upstream device includes: A sending unit, configured to send a neural network model to a plurality of downstream devices, wherein the neural network model includes a plurality of substructures, and each of the plurality of substructures includes at least two neurons; A receiving unit, configured to receive a plurality of first models from the plurality of downstream devices, wherein the plurality of first models are obtained by training the neural network model based on training data, wherein the training data includes: image data, audio data or text data; an aggregation unit, configured to aggregate the plurality of first models to obtain a second model; A pruning unit is used to prune the second model based on a loss function and a constraint condition, wherein the loss function is used to indicate that the substructure of the second model is to be pruned, and the constraint condition is used to constrain the accuracy of the second model after pruning to be no less than the accuracy before pruning.

19. An electronic device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 7.

20. An electronic device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the electronic device executes the method as claimed in claim 8 or 9.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

22. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 9.

Images