A model training method and device
By compressing the model to be trained into a sub-model and training on the terminal device, the problem of insufficient processing capabilities of the terminal device is solved, the efficiency and adaptability of federated learning are improved, and resource optimization is achieved.
Patent Information
- Application Number
- CN202010762709.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-31
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-07-31
AI Technical Summary
The existing federated learning technology requires that the model of central nodes and distributed nodes is isomorphic, but terminal devices such as mobile phones cannot calculate large models of the same scale due to data processing capabilities, resulting in inefficient model training.
By compressing the model to be trained into a sub-model, sending a sub-model to the terminal device for training, and receiving gradient updates, the server trains based on these gradient update models, and using knowledge distillation and other technologies to optimize the model to avoid the terminal device and server training at the same time. Scale model.
It reduces the storage and computing resources required by terminal devices and servers, improves the efficiency and adaptability of model training, and is suitable for devices with different data processing capabilities.
Smart Images

Figure CN112070207B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a model training method and device. Background Art
[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0003] A federated learning system trains a machine learning model based on data generated from a large number of users interacting with their devices (e.g., smartphones, etc.) without taking the data off the devices. For example, each loop selects a subset of online devices, and the current version of the machine learning model is sent to those selected devices. Each of those selected devices is tasked with computing updates to the model using their own locally generated and locally stored data. The model updates are then sent back to the server, averaged, and applied to the server's model to produce a new version of the model for the next iteration of users (e.g., the next subset of devices).
[0004] Federated learning is divided into two steps: model download and model upload. The central node sends the model to the terminal device through the network; each terminal device uses local data to calculate the gradient of the model; each distributed node encrypts the gradient and uploads it to the central node; the central node summarizes the gradients of each terminal distributed node and uses the parameter averaging algorithm to update the parameters of the central node model.
[0005] Existing federated learning technology requires that the models of the central node and each distributed node be isomorphic. However, in some scenarios, terminal devices such as mobile phones are unable to calculate large models of the same scale as the central node due to limited data processing capabilities. Summary of the invention
[0006] In a first aspect, the present application provides a model training method, the method comprising:
[0007] Sending a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained;
[0008] receiving a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model;
[0009] Based at least on the first gradient, model training is performed on the model to be trained to obtain an updated model to be trained.
[0010] The embodiment of the present application provides a model training method, including: sending a first sub-model to a first device; wherein the first sub-model is obtained by compressing a model to be trained; receiving a first gradient sent by the first device, wherein the first gradient is obtained when the first device trains the first sub-model; and performing model training on the model to be trained based at least on the first gradient to obtain an updated model to be trained. In the above manner, the server compresses the model to be trained and sends it to the terminal device, so that the terminal device does not need to train a large model of the same size as the server.
[0011] In one possible implementation, the storage resources required to store the first sub-model are smaller than the storage resources required to store the model to be trained.
[0012] In one possible implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
[0013] In a possible implementation, the computing resources include at least one of the following: memory resources, video memory resources, or CPU resources.
[0014] In a possible implementation, the first sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization.
[0015] In a possible implementation, performing model training on the to-be-trained model based at least on the first gradient includes:
[0016] Based on the first gradient, training the first sub-model to obtain an updated first sub-model;
[0017] At least the updated first sub-model is used as the teacher model, the model to be trained is used as the student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0018] In a possible implementation, the method further includes:
[0019] Acquire a data processing capability of the first device, where the data processing capability includes at least one of the following: available storage resources or available computing resources;
[0020] Based on a first mapping relationship, a first sub-model corresponding to the first device is obtained; wherein the first mapping relationship includes a correspondence between data processing capabilities and sub-models, and in the first mapping relationship, the data processing capabilities of the first device correspond to the first sub-model.
[0021] In a possible implementation, the method further includes:
[0022] Sending a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model;
[0023] receiving a second gradient sent by the second device, where the second gradient is obtained when the second device trains the second sub-model;
[0024] The performing model training on the model to be trained based at least on the first gradient includes:
[0025] Based on the first gradient and the second gradient, model training is performed on the model to be trained to obtain an updated model to be trained.
[0026] In a possible implementation, the first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources.
[0027] In a possible implementation, sizes of storage resources required to store the first sub-model and the second sub-model are different.
[0028] In a possible implementation, the sizes of computing resources required to run the first sub-model and the second sub-model are different.
[0029] In a possible implementation, the second sub-model is obtained by performing at least one of the following processing on the model to be trained: pruning, knowledge distillation or quantization.
[0030] In a possible implementation, performing model training on the to-be-trained model based on the first gradient and the second gradient includes:
[0031] Based on the first gradient, training the first sub-model to obtain an updated first sub-model;
[0032] Based on the second gradient, training the second sub-model to obtain an updated second sub-model;
[0033] The updated first sub-model and the updated second sub-model are used as teacher models, the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0034] In a second aspect, the present application provides a model training device, the device comprising:
[0035] A sending module, configured to send a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained;
[0036] A receiving module, configured to receive a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model;
[0037] A model training module is used to perform model training on the model to be trained based on at least the first gradient to obtain an updated model to be trained.
[0038] In one possible implementation, the storage resources required to store the first sub-model are smaller than the storage resources required to store the model to be trained.
[0039] In one possible implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
[0040] In a possible implementation, the computing resources include at least one of the following: memory resources, video memory resources, or CPU resources.
[0041] In a possible implementation, the first sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization.
[0042] In one possible implementation, the model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; at least the updated first sub-model is used as a teacher model, and the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0043] In a possible implementation, the device further includes:
[0044] An acquisition module is used to acquire the data processing capability of the first device, wherein the data processing capability includes at least one of the following: available storage resources or available computing resources; based on a first mapping relationship, acquiring a first sub-model corresponding to the first device; wherein the first mapping relationship includes a correspondence between the data processing capability and the sub-model, and in the first mapping relationship, the data processing capability of the first device corresponds to the first sub-model.
[0045] In a possible implementation, the sending module is used to send a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model;
[0046] The receiving module is used to receive a second gradient sent by the second device, where the second gradient is obtained when the second device trains the second sub-model;
[0047] The model training module is used to perform model training on the model to be trained based on the first gradient and the second gradient to obtain an updated model to be trained.
[0048] In a possible implementation, the first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources.
[0049] In a possible implementation, sizes of storage resources required to store the first sub-model and the second sub-model are different.
[0050] In a possible implementation, the sizes of computing resources required to run the first sub-model and the second sub-model are different.
[0051] In a possible implementation, the second sub-model is obtained by performing at least one of the following processing on the model to be trained: pruning, knowledge distillation or quantization.
[0052] In a possible implementation, the model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; train the second sub-model based on the second gradient to obtain an updated second sub-model; use the updated first sub-model and the updated second sub-model as teacher models, the model to be trained as a student model, and perform knowledge distillation on the model to be trained to obtain an updated model to be trained.
[0053] In a third aspect, the present application provides a system, including a first device and a server; wherein:
[0054] The server is used to send a first sub-model to the first device; wherein the first sub-model is obtained by compressing the model to be trained;
[0055] The first device is used to train the first sub-model to obtain a first gradient; and send the first gradient to the server;
[0056] The server is used to perform model training on the model to be trained based on at least the first gradient to obtain an updated model to be trained.
[0057] In one possible implementation, the storage resources required to store the first sub-model are smaller than the storage resources required to store the model to be trained.
[0058] In one possible implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
[0059] In a possible implementation, the computing resources include at least one of the following: memory resources, video memory resources, or CPU resources.
[0060] In a possible implementation, the first sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization.
[0061] In a possible implementation, the server is used to train the first sub-model based on the first gradient to obtain an updated first sub-model;
[0062] At least the updated first sub-model is used as the teacher model, the model to be trained is used as the student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0063] In a possible implementation, the server is used to obtain the data processing capability of the first device, where the data processing capability includes at least one of the following: available storage resources or available computing resources;
[0064] Based on a first mapping relationship, a first sub-model corresponding to the first device is obtained; wherein the first mapping relationship includes a correspondence between data processing capabilities and sub-models, and in the first mapping relationship, the data processing capabilities of the first device correspond to the first sub-model.
[0065] In a possible implementation, the system further includes: a second device, and the server is configured to send a second sub-model to the second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model;
[0066] The second device is used to train the second sub-model to obtain a second gradient; and send the second gradient to the server;
[0067] The server, configured to perform model training on the model to be trained based at least on the first gradient, comprises:
[0068] Based on the first gradient and the second gradient, model training is performed on the model to be trained to obtain an updated model to be trained.
[0069] In a possible implementation, the first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: different usable storage resources or different usable computing resources.
[0070] In a possible implementation, sizes of storage resources required to store the first sub-model and the second sub-model are different.
[0071] In a possible implementation, the sizes of computing resources required to run the first sub-model and the second sub-model are different.
[0072] In a possible implementation, the second sub-model is obtained by performing at least one of the following processing on the model to be trained: pruning, knowledge distillation or quantization.
[0073] In a possible implementation, the server is used to train the first sub-model based on the first gradient to obtain an updated first sub-model;
[0074] Based on the second gradient, training the second sub-model to obtain an updated second sub-model;
[0075] The updated first sub-model and the updated second sub-model are used as teacher models, the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0076] In a fourth aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the above-mentioned first aspect and any optional method of the first aspect.
[0077] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect or any optional method thereof.
[0078] In a sixth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.
[0079] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting an execution device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or training device. The chip system can be composed of chips, or it can include chips and other discrete devices.
[0080] The embodiment of the present application provides a model training method, including: sending a first sub-model to a first device; wherein the first sub-model is obtained by compressing a model to be trained; receiving a first gradient sent by the first device, wherein the first gradient is obtained when the first device trains the first sub-model; and performing model training on the model to be trained based at least on the first gradient to obtain an updated model to be trained. In the above manner, the server compresses the model to be trained and sends it to the terminal device, so that the terminal device does not need to train a large model of the same size as the server. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] Figure 1 A structural diagram of the main framework of artificial intelligence;
[0082] Figure 2 A schematic diagram of a computing system for performing model training in an embodiment of the present application;
[0083] Figure 3 It is a schematic diagram of a system architecture provided by an embodiment of the present application;
[0084] Figure 4 A schematic diagram of the architecture of a model training method provided in an embodiment of the present application;
[0085] Figure 5 A schematic diagram of a model training method provided in an embodiment of the present application;
[0086] Figure 6 and Figure 7 An example of a model training method provided in an embodiment of the present application;
[0087] Figure 8 A schematic diagram of a data processing device provided in an embodiment of the present application;
[0088] Fig. 9 A schematic diagram of a system provided in an embodiment of the present application;
[0089] Fig.10 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;
[0090] Fig.11It is a structural schematic diagram of a training device provided in an embodiment of the present application;
[0091] Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0092] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.
[0093] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0094] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0095] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.
[0096] (1) Infrastructure
[0097] The infrastructure provides computing power support for the AI system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
[0098] (2) Data
[0099] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.
[0100] (3) Data processing
[0101] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
[0102] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0103] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0104] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
[0105] (4) General ability
[0106] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0107] (5) Smart products and industry applications
[0108] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.
[0109] Figure 2 Schematic diagram of a computing system for performing model training in an embodiment of the present application. The computing system includes a terminal device 102 (hereinafter referred to as a first device and a second device) and a server 130 (also referred to as a central node) coupled via network communication. The terminal device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or a tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.
[0110] The terminal device 102 may include a processor 112 and a memory 114. The processor 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a controller, a microcontroller, etc.). The memory 114 may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM). The memory 114 may store data 116 and instructions 118 executed by the processor 112 to enable the terminal device 102 to perform operations.
[0111] In some embodiments, the memory 114 can store one or more models 120. For example, the model 120 can be or can additionally include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network can include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.
[0112] In some implementations, one or more models 120 may be received from server 130 via network 180 , stored in memory 114 , and then used or otherwise implemented by one or more processors 112 .
[0113] The terminal device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other devices by which a user may provide user input.
[0114] The terminal device 102 may further include a communication interface 123, through which the terminal device 102 may be communicatively connected to the server 130. The server 130 may include a communication interface 133, through which the terminal device 102 may be communicatively connected to the communication interface 133 of the server 130, thereby achieving data interaction between the terminal device 102 and the server 130.
[0115] The server 130 may include a processor 132 and a memory 134. The processor 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a controller, a microcontroller, etc.). The memory 134 may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM). The memory 134 may store data 136 and instructions 138 executed by the processor 132 to enable the server 130 to perform operations.
[0116] As described above, the memory 134 can store one or more machine learning models 140. For example, the model 140 can be or can additionally include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.
[0117] Figure 3 is a schematic diagram of a system 100 architecture provided in an embodiment of the present application. Figure 3In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: various tasks to be scheduled, callable resources, and other parameters in the embodiment of the present application.
[0118] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.
[0119] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.
[0120] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.
[0121] exist Figure 3 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.
[0122] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. Figure 3 As shown, a neural network can be obtained by training according to the training device 120.
[0123] The present application also provides a chip, which includes a neural network processor NPU 50. The chip can be set as follows Figure 3 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule.
[0124] The neural network processor NPU 40 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.
[0125] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.
[0126] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 408.
[0127] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.
[0128] In some implementations, the vector calculation unit 407 can store the processed output vector to the unified buffer 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.
[0129] The unified memory 406 is used to store input data and output data.
[0130] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller 405 (DMAC), the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.
[0131] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.
[0132] An instruction fetch buffer 409 connected to the controller 404 and used to store instructions used by the controller 404;
[0133] The controller 404 is used to call the instructions cached in the memory 409 to control the working process of the computing accelerator.
[0134] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.
[0135] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0136] (1) Neural Network
[0137] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:
[0138]
[0139] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.
[0140] The work of each layer in the neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.
[0141] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.
[0142] (2) Back propagation algorithm
[0143] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
[0144] See also Figure 4 , Figure 4 The following is a schematic diagram of the architecture of a model training method provided in an embodiment of the present application, such as Figure 4As shown, the architecture provided by the embodiment of the present application includes: a cloud-side central node, which can be, for example, a cloud-side server. A1, A2, ... are distributed nodes of type A, such as mobile phone products held by users. B1, B2, ... are distributed nodes of type B, such as personal computers held by users. After the administrator of the distributed node (such as a user of a mobile phone or computer) agrees, the administrator of the distributed node voluntarily shares the data generated in the process of daily use of the device while protecting privacy, adds it to the model training plan, and the device becomes a distributed node in the architecture. The system in this embodiment can also include more types of distributed nodes, such as smart watches, etc. In order to protect data privacy, the distributed nodes will not upload data to the central node, but only save data locally. The distributed nodes are connected to the cloud server via a communication network. The cloud-side central node can run a large model, while each distributed node can only run a small model due to hardware capabilities, and A and B can have different data processing capabilities.
[0145] See also Figure 5 , Figure 5 A flow chart of a model training method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the model training method provided in the embodiment of the present application includes:
[0146] 501. The server sends a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained.
[0147] In the embodiment of the present application, the server (also referred to as a central node or a cloud-side training device) may send a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained. The first device may be a terminal device, for example Figure 4 The distributed nodes (A1, A2, B1, B2) shown in FIG.
[0148] In an embodiment of the present application, the server can obtain a model to be trained, which can be a model training object for subsequent federated learning. The model to be trained may include a neural network or other multi-layer nonlinear models. For example, the neural network may include a feedforward neural network, a deep neural network, a recursive neural network, and a convolutional neural network.
[0149] In an embodiment of the present application, the server can compress the model to be trained to obtain a first sub-model. Specifically, the first sub-model can be obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization. Among them, in the weight matrix of the neural network, setting a part of the elements to invalid values is a pruning operation. In one implementation, the elements to be pruned can be set to zero. For example, the weight values in the weight matrix whose absolute values are less than or equal to a certain threshold are set to zero. Regarding knowledge distillation and quantization, the knowledge distillation and quantization methods provided in the prior art can be used.
[0150] Among them, the first sub-model and the model to be trained can satisfy at least one of the following characteristics: the storage resources required to store the first sub-model are less than the storage resources required to store the model to be trained, and the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained, wherein the computing resources can include at least one of the following: memory resources, video memory resources or CPU resources.
[0151] In one implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the model to be trained. The storage resources can be understood as the size of the storage space; for example, the storage space required for the terminal device to store the first sub-model is 200MB, while the storage space required to store the model to be trained is 1GB.
[0152] In one implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained; the computing resources may include at least one of the following: memory resources, video memory resources or CPU resources; it should be noted that the required computing resources may refer to the peak computing resources required to run the model, such as the peak memory occupancy, the peak video memory occupancy or the peak CPU occupancy.
[0153] In an embodiment of the present application, the server may select the first sub-model with reference to the data processing capability of the first device, so that the first device has sufficient resources to run the first sub-model; in one implementation, the server may obtain the data processing capability of the first device, and the data processing capability includes at least one of the following: available storage resources or available computing resources. The server may obtain the first sub-model corresponding to the first device based on a first mapping relationship; wherein the first mapping relationship includes a correspondence between data processing capability and sub-model, and in the first mapping relationship, the data processing capability of the first device corresponds to the first sub-model.
[0154] In one implementation, the data processing capability of the first device may be related to the available computing resources. In one implementation, the data processing capability of the first device may be related to the available computing resources. In one implementation, the data processing capability of the first device may be related to the available computing resources and the available computing resources.
[0155] Specifically, the data processing capability of the first device may be a parameter value, which may indicate the data processing capability of the device. In one implementation, the terminal device may determine the data processing capability of the first device based on the hardware capability information of the first device, such as the system on chip (SOC) model, graphics processing unit (GPU) model, embedded neural network processing unit (NPU) model, storage space size, etc. Optionally, the server may maintain a calculation function for data processing capability, with the input being the above-mentioned SOC model, GPU model, NPU model, storage space size, etc., and the output being the calculated data processing capability.
[0156] In an embodiment of the present application, the server can obtain the first sub-model corresponding to the first device based on the first mapping relationship, wherein the first mapping relationship can be a data processing capability interval-model mapping table, each data processing capability interval corresponds to a model, for example, the data processing capability interval is 0 to 100 points, the data processing capability interval is 0 to 10, 10 to 20, 20 to 30, ..., 90 to 100, and each interval is mapped to a model. In an embodiment of the present application, the server can obtain the data processing capability of the first device, and determine the interval where the data processing capability of the first device is located from the first mapping relationship, wherein the interval where the data processing capability of the first device is located corresponds to the first sub-model.
[0157] It should be understood that the hardware capability information related to the first device, such as the SOC model, GPU model, NPU model, storage space size, etc., can be uploaded by the first device to the server. Optionally, the first device can send a device registration request to the server and carry the above hardware capability information.
[0158] Similarly, the server may also send a second sub-model to the second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model; specifically, the storage resource sizes required to store the first sub-model and the second sub-model are different and / or the computing resource sizes required to run the first sub-model and the second sub-model are different. The second sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation, or quantization.
[0159] In an embodiment of the present application, the server can send different models to devices with different data processing capabilities. For example, the second device and the first device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources. Accordingly, the server can send the first sub-model to the first device and the second sub-model to the second device. The second sub-model is different from the first sub-model. For example, the storage resource sizes required to store the first sub-model and the second sub-model are different and / or the computing resource sizes required to run the first sub-model and the second sub-model are different.
[0160] Next, we will give an application example where a server sends a model to a terminal device:
[0161] First, the user voluntarily registers the terminal device he holds to the training framework of the large-scale pre-trained language model, so that the user is aware of the impact of the training on his daily use of the device, and the training process needs to strictly implement the user data privacy protection strategy. The user can select the data source in his device that can participate in the training plan, for example: specify that the text data generated when using some specific applications (such as Weibo) can participate in the training plan. The terminal device on the user side (such as the first device and the second device in the above embodiment) can send a device registration request to the server and carry hardware capability information (such as the SOC model, GPU model, NPU model, storage space size and other hardware capability information in the above embodiment). Optional: If a user holds multiple personal devices and data is shared between multiple personal devices, the user can register only one device as a distributed node. The server can obtain the hardware capability information of the terminal device, and map the terminal device to a specific group based on the hardware capability information of the terminal device. Different terminal groups correspond to different terminal hardware capabilities, and each group corresponds to a compressed model (for example, in the above embodiment, the first device corresponds to the first sub-model, and the second device corresponds to the second sub-model). Specifically, the server can establish an association relationship between the ID of the terminal device and the corresponding group ID (for example, an association relationship between the ID of the above terminal device and the corresponding group ID is established in the first mapping relationship in the above embodiment).
[0162] After the user generates enough data for training during the daily use of the terminal device, the terminal device sends a model download request (carrying the ID of the terminal device) to the cloud-side server. The cloud-side server indexes the corresponding group ID according to the ID of the terminal device, and the server can send the model corresponding to the group ID to the terminal device through the network. Exemplarily, the server can provide a compressed model TinyBERTi of a specific size for the original model BERT stored in the central server, and distribute it to the terminal device through the network.
[0163] 502. Receive a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model.
[0164] In an embodiment of the present application, after receiving the first sub-model sent by the server, the first device can use local data to train the first sub-model and obtain a first gradient during the training process, and then upload the first gradient to the server. Optionally, in order to protect information privacy during the transmission process, the first device can upload the encrypted first gradient to the server.
[0165] Similarly, after receiving the second sub-model sent by the server, the second device can use local data to train the second sub-model and obtain a second gradient during the training process, and then upload the second gradient to the server. Optionally, in order to protect the privacy of information during the transmission process, the second device can upload the encrypted second gradient to the server.
[0166] In the embodiment of the present application, the server may receive the first gradient sent by the first device. Similarly, the server may also receive the second gradient sent by the second device.
[0167] 503. Perform model training on the model to be trained based at least on the first gradient to obtain an updated model to be trained.
[0168] In an embodiment of the present application, the server may perform model training on the model to be trained based on at least the first gradient to obtain an updated model to be trained. Specifically, the server may train the first sub-model based on the first gradient to obtain an updated first sub-model; at least use the updated first sub-model as a teacher model and the model to be trained as a student model, and perform knowledge distillation on the model to be trained to obtain an updated model to be trained.
[0169] In an embodiment of the present application, the server can also train the second sub-model based on the second gradient to obtain an updated second sub-model; using the updated first sub-model and the updated second sub-model as teacher models, the model to be trained as a student model, and performing knowledge distillation on the model to be trained to obtain an updated model to be trained.
[0170] In the embodiment of the present application, the server can summarize the gradients uploaded by each type of terminal device in groups, and use the federated gradient averaging algorithm to update the corresponding small model. Figure 6 and Figure 7 , Figure 6 and Figure 7An example of a model training method provided in an embodiment of the present application, wherein the model compression module of the server can compress the model to be trained into a first sub-model and a second sub-model, and then the server can send the first sub-model to nodes A1, A2 and A3, and send the second sub-model to nodes B1, B2 and B3.
[0171] Afterwards, the gradients uploaded by nodes A1, A2, and A3 are used to update the parameters of the first sub-model by using the federated gradient average method, and the gradients uploaded by nodes B1, B2, and B3 are used to update the parameters of the second sub-model by using the federated gradient average method. In addition, the server uses various types of small models to fuse and update the central node large model by using the multi-model fusion method. For example, the first sub-model and the second sub-model are fused into the model to be trained. The fusion method can be multi-teacher distillation.
[0172] For example, the server can aggregate multiple updated compression models {TinyBERTi_update}, use multi-teacher distillation, treat {TinyBERTi_update} as multiple teacher models, and distill to obtain the student model BERT_update. Because the distillation process cannot use the user's end-side data, a data-free distillation method (DatafreeDistillation) is used, or synthetic data is used for distillation, or general public data is used for distillation.
[0173] The embodiment of the present application provides a model training method, including: sending a first sub-model to a first device; wherein the first sub-model is obtained by compressing a model to be trained; receiving a first gradient sent by the first device, wherein the first gradient is obtained when the first device trains the first sub-model; and performing model training on the model to be trained based at least on the first gradient to obtain an updated model to be trained. In the above manner, the server compresses the model to be trained and sends it to the terminal device, so that the terminal device does not need to train a large model of the same size as the server.
[0174] In an embodiment of the present application, the server may also send a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model; receive a second gradient sent by the second device, wherein the second gradient is obtained when the second device trains the second sub-model; and based on the first gradient and the second gradient, perform model training on the model to be trained to obtain an updated model to be trained. In the above manner, the server can select different models to be sent to the terminal device according to the different terminal devices.
[0175] Next, the data processing device provided in the embodiment of the present application is described from the perspective of the device. Figure 8 , Figure 8 A schematic diagram of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown in FIG. 8 , the data processing device 800 provided in the embodiment of the present application includes:
[0176] A sending module 801 is used to send a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained;
[0177] A receiving module 802 is configured to receive a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model;
[0178] The model training module 803 is used to perform model training on the model to be trained based on at least the first gradient to obtain an updated model to be trained.
[0179] In one possible implementation, the storage resources required to store the first sub-model are smaller than the storage resources required to store the model to be trained.
[0180] In one possible implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
[0181] In a possible implementation, the computing resources include at least one of the following: memory resources, video memory resources, or CPU resources.
[0182] In a possible implementation, the first sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization.
[0183] In one possible implementation, the model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; at least the updated first sub-model is used as a teacher model, and the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0184] In a possible implementation, the device further includes:
[0185] An acquisition module is used to acquire the data processing capability of the first device, wherein the data processing capability includes at least one of the following: available storage resources or available computing resources; based on a first mapping relationship, acquiring a first sub-model corresponding to the first device; wherein the first mapping relationship includes a correspondence between the data processing capability and the sub-model, and in the first mapping relationship, the data processing capability of the first device corresponds to the first sub-model.
[0186] In a possible implementation, the sending module is used to send a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model;
[0187] The receiving module is used to receive a second gradient sent by the second device, where the second gradient is obtained when the second device trains the second sub-model;
[0188] The model training module is used to perform model training on the model to be trained based on the first gradient and the second gradient to obtain an updated model to be trained.
[0189] In a possible implementation, the first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources.
[0190] In a possible implementation, sizes of storage resources required to store the first sub-model and the second sub-model are different.
[0191] In a possible implementation, the sizes of computing resources required to run the first sub-model and the second sub-model are different.
[0192] In a possible implementation, the second sub-model is obtained by performing at least one of the following processing on the model to be trained: pruning, knowledge distillation or quantization.
[0193] In a possible implementation, the model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; train the second sub-model based on the second gradient to obtain an updated second sub-model; use the updated first sub-model and the updated second sub-model as teacher models, the model to be trained as a student model, and perform knowledge distillation on the model to be trained to obtain an updated model to be trained.
[0194] Reference Fig. 9 , Fig. 9 A schematic diagram of a system provided in an embodiment of the present application is shown as follows: Fig. 9As shown in FIG. 1 , the system 900 provided in the embodiment of the present application includes:
[0195] A first device 901 and a server 902; wherein the server 902 is used to send a first sub-model to the first device 901; wherein the first sub-model is obtained by compressing the model to be trained;
[0196] The first device 901 is used to train the first sub-model to obtain a first gradient; and send the first gradient to the server 902;
[0197] The server 902 is used to perform model training on the model to be trained based on at least the first gradient to obtain an updated model to be trained.
[0198] In one possible implementation, the storage resources required to store the first sub-model are smaller than the storage resources required to store the model to be trained.
[0199] In one possible implementation, the computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
[0200] In a possible implementation, the computing resources include at least one of the following: memory resources, video memory resources, or CPU resources.
[0201] In a possible implementation, the first sub-model is obtained by performing at least one of the following processing methods on the model to be trained: pruning, knowledge distillation or quantization.
[0202] In a possible implementation, the server 902 is configured to train the first sub-model based on the first gradient to obtain an updated first sub-model;
[0203] At least the updated first sub-model is used as the teacher model, the model to be trained is used as the student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0204] In a possible implementation, the server 902 is used to obtain the data processing capability of the first device 901, where the data processing capability includes at least one of the following: available storage resources or available computing resources;
[0205] Based on the first mapping relationship, obtain the first sub-model corresponding to the first device 901; wherein the first mapping relationship includes the correspondence between data processing capabilities and sub-models, and in the first mapping relationship, the data processing capabilities of the first device 901 correspond to the first sub-model.
[0206] In a possible implementation, the system further includes: a second device 903, and the server 902 is configured to send a second sub-model to the second device 903; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model;
[0207] The second device 903 is used to train the second sub-model to obtain a second gradient; and send the second gradient to the server 902;
[0208] The server 902 is configured to perform model training on the model to be trained based at least on the first gradient, including:
[0209] Based on the first gradient and the second gradient, model training is performed on the model to be trained to obtain an updated model to be trained.
[0210] In a possible implementation, the first device 901 and the second device 903 have different data processing capabilities, and the data processing capabilities include at least one of the following: different usable storage resources or different usable computing resources.
[0211] In a possible implementation, sizes of storage resources required to store the first sub-model and the second sub-model are different.
[0212] In a possible implementation, the sizes of computing resources required to run the first sub-model and the second sub-model are different.
[0213] In a possible implementation, the second sub-model is obtained by performing at least one of the following processing on the model to be trained: pruning, knowledge distillation or quantization.
[0214] In a possible implementation, the server 902 is configured to train the first sub-model based on the first gradient to obtain an updated first sub-model;
[0215] Based on the second gradient, training the second sub-model to obtain an updated second sub-model;
[0216] The updated first sub-model and the updated second sub-model are used as teacher models, the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
[0217] Next, an execution device provided by an embodiment of the present application is introduced. Fig.10 , Fig.10This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1000 can be specifically manifested as a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Among them, the execution device 1000 can be deployed with Fig.10 The data processing device described in the corresponding embodiment is used to implement Fig.10 The data processing function in the corresponding embodiment. Specifically, the execution device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003 and a memory 1004 (wherein the number of the processor 1003 in the execution device 1000 can be one or more, Fig.10 In the example of FIG. 1003 , the processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of the present application, the receiver 1001, the transmitter 1002, the processor 1003 and the memory 1004 may be connected via a bus or other means.
[0218] The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1003. A portion of the memory 1004 may also include a non-volatile random access memory (NVRAM). The memory 1004 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.
[0219] The processor 1003 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.
[0220] The method disclosed in the above embodiment of the present application can be applied to the processor 1003, or implemented by the processor 1003. The processor 1003 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1003. The above processor 1003 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1003 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1004, and the processor 1003 reads the information in the memory 1004 and completes the steps of the above method in combination with its hardware.
[0221] The receiver 1001 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1002 can be used to output digital or character information through the first interface; the transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1002 can also include a display device such as a display screen.
[0222] In one embodiment of the present application, the processor 1003 is configured to execute Figure 5 The method executed by the execution device in the corresponding embodiment.
[0223] The present application also provides a training device. Fig.11 , Fig.11 is a structural diagram of a training device provided in an embodiment of the present application. The training device 1100 may be deployed with Fig.10 The neural network training device described in the corresponding embodiment is used to implement Fig.10The functions of the neural network training device in the corresponding embodiment are specifically implemented by one or more servers. The training device 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1111 (for example, one or more processors) and memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1111 can be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the training device 1100.
[0224] The training device 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158; or, one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0225] In the embodiment of the present application, the central processing unit 1111 is used to execute the steps related to the training method in the above embodiment.
[0226] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0227] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0228] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0229] For details, please refer to Fig.12 , Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1200, NPU 1200 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.
[0230] In some implementations, the operation circuit 1203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1203 is a two-dimensional systolic array. The operation circuit 1203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1203 is a general-purpose matrix processor.
[0231] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1201 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 1208.
[0232] The unified memory 1206 is used to store input data and output data. The weight data is directly transferred to the weight memory 1202 through the direct memory access controller (DMAC) 1205. The input data is also transferred to the unified memory 1206 through the DMAC.
[0233] BIU stands for Bus Interface Unit, i.e., bus interface unit 1210 , which is used for the interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 1209 .
[0234] The bus interface unit 1210 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0235] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or to transfer weight data to the weight memory 1202 or to transfer input data to the input memory 1201.
[0236] The vector calculation unit 1207 includes multiple operation processing units, and further processes the output of the operation circuit 1203 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.
[0237] In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector calculation unit 1207 can apply a linear function; or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, for example, for use in a subsequent layer in a neural network.
[0238] An instruction fetch buffer 1209 connected to the controller 1204 is used to store instructions used by the controller 1204;
[0239] Unified memory 1206, input memory 1201, weight memory 1202 and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0240] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
[0241] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0242] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0243] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0244] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A model training method, characterized in that: Applied to a server, the method comprises: Sending a first sub-model to a first device; wherein the first sub-model is obtained by compressing the model to be trained; and the first device is a terminal device; receiving a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model; Based on the first gradient, training the first sub-model to obtain an updated first sub-model; At least the updated first sub-model is used as the teacher model, the model to be trained is used as the student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
2. The method according to claim 1, characterized in that The storage resources required to store the first sub-model are less than the storage resources required to store the model to be trained.
3. The method according to claim 1 or 2, characterized in that: The computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
4. The method according to claim 3, characterized in that The computing resources include at least one of the following: memory resources, video memory resources or CPU resources.
5. The method according to claim 1 or 2, characterized in that: The first sub-model is obtained by performing at least one of the following processing methods on the training model: pruning, knowledge distillation or quantization.
6. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire a data processing capability of the first device, where the data processing capability includes at least one of the following: available storage resources or available computing resources; Based on a first mapping relationship, a first sub-model corresponding to the first device is obtained; wherein the first mapping relationship includes a correspondence between data processing capabilities and sub-models, and in the first mapping relationship, the data processing capabilities of the first device correspond to the first sub-model.
7. The method according to claim 1 or 2, characterized in that: The method further comprises: Sending a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model; receiving a second gradient sent by the second device, where the second gradient is obtained when the second device trains the second sub-model; The performing model training on the model to be trained based at least on the first gradient includes: Based on the first gradient and the second gradient, model training is performed on the model to be trained to obtain an updated model to be trained.
8. The method according to claim 7, characterized in that The first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources.
9. The method according to claim 7, characterized in that: The sizes of storage resources required to store the first sub-model and the second sub-model are different.
10. The method according to claim 7, characterized in that The computing resources required to run the first sub-model and the second sub-model are different in size.
11. The method according to claim 7, characterized in that The second sub-model is obtained by performing at least one of the following processing methods on the training model: pruning, knowledge distillation or quantization.
12. The method according to claim 7, characterized in that The performing model training on the to-be-trained model based on the first gradient and the second gradient includes: Based on the first gradient, training the first sub-model to obtain an updated first sub-model; Based on the second gradient, training the second sub-model to obtain an updated second sub-model; The updated first sub-model and the updated second sub-model are used as teacher models, the model to be trained is used as a student model, and knowledge distillation is performed on the model to be trained to obtain an updated model to be trained.
13. A model training device, characterized in that: Applied to a server, the device comprises: A sending module, configured to send a first sub-model to a first device; wherein the first sub-model is obtained by compressing a model to be trained, and the first device is a terminal device; A receiving module, configured to receive a first gradient sent by the first device, where the first gradient is obtained when the first device trains the first sub-model; A model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; at least use the updated first sub-model as a teacher model and the model to be trained as a student model, and perform knowledge distillation on the model to be trained to obtain an updated model to be trained.
14. The device according to claim 13, characterized in that The storage resources required to store the first sub-model are less than the storage resources required to store the model to be trained.
15. The device according to claim 13 or 14, characterized in that The computing resources required to run the first sub-model are less than the computing resources required to run the model to be trained.
16. The device according to claim 15, characterized in that The computing resources include at least one of the following: memory resources, video memory resources or CPU resources.
17. The device according to claim 13 or 14, characterized in that The first sub-model is obtained by performing at least one of the following processing methods on the training model: pruning, knowledge distillation or quantization.
18. The device according to claim 13 or 14, characterized in that The device also includes: An acquisition module is used to acquire the data processing capability of the first device, wherein the data processing capability includes at least one of the following: available storage resources or available computing resources; based on a first mapping relationship, acquiring a first sub-model corresponding to the first device; wherein the first mapping relationship includes a correspondence between the data processing capability and the sub-model, and in the first mapping relationship, the data processing capability of the first device corresponds to the first sub-model.
19. The device according to claim 13 or 14, characterized in that The sending module is used to send a second sub-model to a second device; wherein the second sub-model is obtained by compressing the model to be trained, and the second sub-model is different from the first sub-model; The receiving module is used to receive a second gradient sent by the second device, where the second gradient is obtained when the second device trains the second sub-model; The model training module is used to perform model training on the model to be trained based on the first gradient and the second gradient to obtain an updated model to be trained.
20. The device according to claim 19, characterized in that The first device and the second device have different data processing capabilities, and the data processing capabilities include at least one of the following: available storage resources or available computing resources.
21. The device according to claim 19, characterized in that The sizes of storage resources required to store the first sub-model and the second sub-model are different.
22. The device according to claim 19, characterized in that The computing resources required to run the first sub-model and the second sub-model are different in size.
23. The device according to claim 19, characterized in that The second sub-model is obtained by performing at least one of the following processing methods on the training model: pruning, knowledge distillation or quantization.
24. The device according to claim 19, characterized in that The model training module is used to train the first sub-model based on the first gradient to obtain an updated first sub-model; train the second sub-model based on the second gradient to obtain an updated second sub-model; use the updated first sub-model and the updated second sub-model as teacher models and the model to be trained as a student model, and perform knowledge distillation on the model to be trained to obtain an updated model to be trained.
25. A model training device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes and implement the method according to any one of claims 1 to 12.
26. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 12.
Citation Information
Patent Citations
Computing device and related product
CN110196735A
Method and device for alliance learning and alliance learning system
CN111340243A
Neural network compression method and device
CN111382863A
Federated learning method and system based on batch size and gradient compression ratio adjustment
CN111401552A
Distributed training of models using stochastic gradient descent
US10152676B1
