Data processing method and related equipment
By introducing a loss function term that evaluates the degree of compression of the specified layer during neural network training and adaptively adjusting the output data dimensions, the problem of accuracy reduction caused by neural network pruning compression is solved, and efficient neural network compression and precision retention are achieved.
Patent Information
- Application Number
- CN202311245496.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-25
AI Technical Summary
When the prior art pruning and compressing neural networks, the accuracy of the neural network will be greatly reduced and the important information cannot be effectively retained.
By introducing the first term of the loss function during training, the compression of the specified layer of the neural network is evaluated, and the dimension of the output data is adaptively adjusted by trainable variables, thereby achieving compression of the specified layer of the neural network.
While ensuring the accuracy of the neural network, effectively compress the neural network, reduce resource consumption, and improve inference efficiency.
Smart Images
Figure CN119692409A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method and related equipment. Background Art
[0002] As artificial intelligence technology develops towards the direction of ultra-large data scale, computing scale and model scale, various large-scale neural networks such as large language models (LLM) have emerged.
[0003] However, large-scale neural networks consume a lot of resources when deployed in practice, making their application difficult. Therefore, how to compress neural networks is an important topic.
[0004] Currently, after obtaining a trained neural network, unimportant or redundant parameters are often selected from the trained neural network for pruning based on experience, etc., so as to compress the trained neural network.
[0005] However, this pruning method often loses a lot of information carried by the pruned parameters, thus affecting the accuracy of the neural network. Summary of the invention
[0006] The embodiment of the present application provides a data processing method, which can solve the problem that the current method of pruning and compressing a neural network greatly reduces the accuracy of the neural network. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.
[0007] In a first aspect, the present application provides a data processing method, which includes: obtaining at least one training data, any training data including images, text or speech; training a neural network through at least one training data and a loss function to obtain a trained neural network, the loss function including a first loss term, the first loss term is used to evaluate the degree of compression of a specified layer in the neural network, in the trained neural network, the dimension of the output data of the specified layer is based on the degree of compression of the specified layer after training, and the specified layer in the neural network is compressed.
[0008] In the first aspect, the neural network can be trained by a first loss term so that a specified layer in the trained neural network can use the minimum number of neuron outputs according to the amount of information required to be represented and the representation capacity of a single neuron, that is, compress the dimension of the corresponding output data to remove neural network coding redundancy and achieve compression of the specified layer of the neural network, so that effective information can be compressed in the retained parameters of the neural network, and the neural network can be compressed as much as possible while ensuring that effective information is not lost or is rarely lost to ensure the accuracy of the neural network.
[0009] In a possible implementation manner of the first aspect, the first loss term is obtained based on a trainable variable, and the trainable variable is used to describe the degree of compression of the dimension of output data of a specified layer.
[0010] In this possible implementation, the parameter in the first loss term that characterizes the compression degree of a specified layer in the neural network can be a trainable variable, so that the trainable variable can be continuously iteratively updated with training, and the compression degree of the specified layer in the neural network can be adaptively learned.
[0011] In a possible implementation manner of the first aspect, the trainable variable can be converted into a discrete variable that describes the degree of compression of the dimension of the output data of the specified layer based on a mapping function.
[0012] The variables that describe the degree of compression of the dimension of the output data of the specified layer are often discrete variables, which are not differentiable and cannot be directly iteratively optimized during the training process. Therefore, in this possible implementation, it is proposed to map the continuously trainable variables through a mapping function to discrete variables that can describe the degree of compression of the dimension of the output data of the specified layer, and the continuously trainable variables can be used for iterative updates based on back-propagation and other methods during the training process.
[0013] In a possible implementation manner of the first aspect, the discrete variable describing the degree of compression of the dimension of the output data of the specified layer is the code length after the output data of the specified layer is compressed.
[0014] In a possible implementation manner of the first aspect, the loss function also includes a second loss term, which is used to evaluate the difference between the data probability distribution corresponding to the specified layer and the target distribution.
[0015] In this possible implementation, the neural network is trained through the second loss term, so that the data probability distribution of the specified layer in the trained neural network can be close to the target distribution such as Gaussian distribution, thereby ensuring the information representation capability of the output data of the specified layer.
[0016] In a possible implementation manner of the first aspect, the data probability distribution corresponding to the specified layer refers to the data probability distribution obtained after compressing the specified layer.
[0017] In this possible implementation, by evaluating the difference between the data probability distribution obtained after compression of the specified layer and the target distribution, the output data of the compressed specified layer in the trained neural network is made close to the target distribution such as Gaussian distribution, thereby ensuring the information representation ability of the neurons in the compressed specified layer.
[0018] In a possible implementation of the first aspect, the trained neural network is used to be deployed in a distributed system, the distributed system includes multiple distributed devices, the trained neural network is divided into multiple segmentation modules, different segmentation modules are used to be deployed in different distributed devices, and the designated layer includes at least one of the connection layers between different segmentation modules.
[0019] In this possible implementation, the dimension of the output data of at least one feature layer in the connection layer between different segmentation modules is compressed, thereby reducing the communication consumption between distributed devices in the distributed system in the actual reasoning process, improving the communication efficiency, and thus improving the corresponding data processing efficiency.
[0020] The second aspect of the present application provides a data processing device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module and a training module.
[0021] The third aspect of the present application provides an electronic device, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0022] The fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0023] The fifth aspect of the present application provides a computer program product storing one or more computer execution instructions, wherein the computer program product includes computer execution instructions. When the computer execution instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.
[0024] In a sixth aspect of the present application, a chip system is provided, which includes a processor for supporting an electronic device to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the electronic device. The chip system may be composed of a chip, or may include a chip and other discrete devices.
[0025] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;
[0027] Figure 2 is an exemplary schematic diagram of a gated discrete variable M provided in an embodiment of the present application;
[0028] Figure 3 It is an exemplary flowchart of an iterative process in the training process provided in the embodiment of the present application;
[0029] Figure 4 is an exemplary schematic diagram of a terminal-cloud collaboration system provided in an embodiment of the present application;
[0030] Figure 5 is an exemplary schematic diagram of a large-scale neural network provided in an embodiment of the present application;
[0031] Figure 6 is a schematic diagram of an embodiment of a data processing device provided in an embodiment of the present application;
[0032] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0034] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0035] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is merely a description of the way of distinguishing objects with the same attributes when describing them in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.
[0036] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0037] (1) Neural Network
[0038] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:
[0039]
[0040] Where s=1, 2, ...n, n is a natural number greater than 1, ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.
[0041] (2) Loss Function
[0042] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.
[0043] (3) Back propagation algorithm
[0044] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.
[0045] At present, artificial intelligence technology is developing towards the direction of super-large data scale, computing scale and model scale, and various large-scale neural networks such as large language models (LLM) have emerged.
[0046] However, large-scale neural networks consume a lot of resources when actually deployed, making their application difficult. For example, if the scale of the neural network is large, it may be difficult to deploy it directly on terminal devices with limited resources. In addition, in scenarios such as federated learning, large-scale neural networks need to be deployed in distributed systems, and distributed deployment will cause certain communication losses between different parts of the neural network. Therefore, how to compress neural networks is an important topic.
[0047] Currently, after obtaining a trained neural network, unimportant or redundant parameters are often selected from the trained neural network for pruning based on experience, etc., so as to compress the trained neural network.
[0048] However, this pruning method is often performed on the basis of a fully trained neural network. Therefore, no matter how it is designed, a large amount of information carried by the pruned parameters will be lost, thus affecting the accuracy of the neural network.
[0049] Based on this, a data processing method is provided in an embodiment of the present application, which can compress all valid information into the retained parameters of the neural network, and compress the neural network as much as possible while ensuring that the valid information is not lost or very little is lost to ensure the accuracy of the neural network.
[0050] The embodiments of the present application can be applied to electronic devices.
[0051] The specific type of the electronic device is not limited here. Exemplarily, the electronic device can be a terminal device, a cloud server, a single server or a server cluster, or a virtual machine (VM) or a container.
[0052] When the electronic device is a terminal device, the type of the terminal device is not limited here. Exemplarily, the terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a terminal in industrial control, a terminal in self-driving (such as a car machine, a vehicle computing platform, a domain controller, a sensor, etc.), a terminal in remote medical, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, a terminal in the Internet of Things (IoT), a wearable device, a robot, etc., or a combination of one or more thereof.
[0053] like Figure 1 As shown, based on the above electronic device, a data processing method in an embodiment of the present application includes steps 101-102.
[0054] Step 101: Obtain at least one training data.
[0055] Any training data includes images, text or speech.
[0056] In the embodiment of the present application, the specific method of obtaining the training data is not limited here.
[0057] Exemplarily, the training data may be obtained from an open source dataset or may be manually collected.
[0058] The specific type of the training data is not limited here. The specific type of the training data is associated with the neural network to be trained.
[0059] Exemplarily, if the neural network is a natural language processing model, the training data may include text; if the neural network is an image processing model or a video processing model, the training data may include images; and if the neural network is a speech processing model, the training data may include speech.
[0060] In some examples, the training data may also correspond to a label. In different application scenarios, the specific content of the label of the training data may be different. For example, if the neural network is used to perform a classification task, the label of the training data is the category corresponding to the training data; and if the neural network is used to perform a generation task (such as a translation task), the label of the training data is the actual generated content (such as the translation content) corresponding to the training data.
[0061] The training data may come from a variety of sources. For example, the training data may be stored locally in the electronic device executing the embodiment of the present application, or may be transmitted to the electronic device by other devices, or may be collected by the electronic device through sensors, web pages, or other applications, or may be obtained by the electronic device after processing image data, text data, voice data, etc. There may also be a variety of specific types of the training data, which are not limited here.
[0062] Step 102: Train the neural network using at least one training data and a loss function to obtain a trained neural network.
[0063] The loss function includes a first loss term, which is used to evaluate the compression degree of a specified layer in a neural network. In the trained neural network, the dimension of the output data of the specified layer is obtained by compressing the specified layer in the neural network based on the compression degree of the specified layer after training.
[0064] In the embodiment of the present application, there can be multiple types of neural networks to be trained, which are not limited here. Exemplarily, the neural network can be one or more of a natural language processing model, an image processing model, a speech processing model, and a graph neural network.
[0065] In many scenarios, the neural network can be a large-scale neural network, for example, the neural network can be a large language model (LLM). Compared with traditional neural networks, large language models have more powerful computing power and more network parameters. For example, the large language model GPT4 uses a network model based on the transformer architecture and mixture of experts (MoE), has billions or even tens of billions of parameters, and has 10T of training data, which can have an intelligent understanding of the complexity of language and contextual information.
[0066] Large language models are widely used in natural language processing, including text generation, machine translation, semantic understanding, dialogue systems, etc. They can generate high-quality, fluent text, and can also generate text of a specific style or theme by controlling conditions. However, due to its huge number of parameters and computing resource consumption, large language models face challenges in practical applications, such as slow reasoning speed and difficulty in application on the client side.
[0067] Based on this, in an embodiment of the present application, large-scale neural networks such as large language models can be trained, and after training, the neural network can be compressed as much as possible while ensuring the accuracy of the trained neural network, thereby facilitating the deployment and reasoning operations of the neural network.
[0068] Among them, the neural network can be considered as a source encoder, which encodes and extracts information about features, and finally obtains feature information about the target task in the data. The information that the neural network ultimately needs to extract is information related to the identification target, and there may be redundant information in it, which has a certain redundancy.
[0069] Based on this, in an embodiment of the present application, the neural network can be trained through the first loss term so that the specified layer in the trained neural network can use the minimum number of neuron outputs according to the amount of information required to be represented and the representation ability of a single neuron, that is, compress the dimension of the corresponding output data to remove the neural network encoding redundancy and achieve compression of the specified layer of the neural network.
[0070] Specifically, in an embodiment of the present application, the neural network can be trained through the first loss term of the loss function to train the neural network to adaptively determine the degree of compression of a specified layer in the neural network while ensuring good accuracy. After training, the trained neural network maintains good capabilities while reducing the scale of the neural network, thereby achieving adaptive compression without loss of accuracy.
[0071] The designated layers in the neural network are the layers to be compressed. The number and position of the designated layers in the neural network can be various, and are not limited here.
[0072] Exemplarily, the designated layer may be a layer in the neural network whose corresponding output data dimension is greater than a preset dimension.
[0073] In addition, in scenarios such as end-cloud collaboration and federated learning, when the neural network is split into multiple segmentation modules and deployed to different distributed devices for operation, the separated layers in the neural network can be considered as connection layers between different segmentation modules, and the connection layers can also be called affinity layers. The output data of the affinity layer needs to be transmitted between the corresponding distributed devices, so the amount of the output data greatly affects the corresponding communication efficiency. Therefore, in this scenario, the specified layer in the neural network can be one or more of the connection layers in each segmentation module, that is, the affinity layer. In this way, the embodiment of the present application can be used to compress the dimension of the output data of the specified layer after training, thereby reducing the corresponding communication consumption and improving the communication efficiency in the actual reasoning process, thereby improving the corresponding data processing efficiency.
[0074] Specifically, the first loss term of the loss function is used to evaluate the degree of compression of a specified layer in a neural network.
[0075] The specific form of the first loss item is not limited here.
[0076] Exemplarily, the loss value of the first loss term of the loss function is negatively correlated with the degree of compression of a specified layer in the neural network. That is, the greater the degree of compression of the specified layer in the neural network, the smaller the loss value of the first loss term, thereby instructing the neural network to learn in the direction of a greater degree of compression of the specified layer.
[0077] There are many ways to characterize the parameters of the degree of compression of a specified layer in a neural network. If the value of the parameter characterizing the degree of compression of a specified layer in a neural network increases as the degree of compression of the specified layer in the neural network increases, then the loss value of the first loss term of the loss function is negatively correlated with the value of the parameter characterizing the degree of compression of the specified layer in the neural network; if the value of the parameter characterizing the degree of compression of a specified layer in a neural network decreases as the degree of compression of the specified layer in the neural network increases, then the loss value of the first loss term of the loss function is positively correlated with the value of the parameter characterizing the degree of compression of the specified layer in the neural network.
[0078] In one embodiment, the first loss term is obtained based on a trainable variable, which is used to describe the degree of compression of the dimension of the output data of the specified layer.
[0079] It can be seen that in the embodiment of the present application, the parameter in the first loss term that characterizes the compression degree of the specified layer in the neural network can be a trainable variable, so that the trainable variable can be continuously iteratively updated with training, and the compression degree of the specified layer in the neural network can be adaptively learned.
[0080] In some embodiments, the trainable variables can be converted into discrete variables that describe the degree of compression of the dimensions of the output data of a specified layer based on a mapping function.
[0081] In an embodiment of the present application, the variable describing the degree of compression of the dimension of the output data of a specified layer is often a discrete variable, and discrete variables are not differentiable and cannot be directly iteratively optimized during the training process. Therefore, in an embodiment of the present application, it is proposed to map the continuously trainable variables through a mapping function to discrete variables that can describe the degree of compression of the dimension of the output data of the specified layer, and the continuously trainable variables can be used for iterative updates based on back propagation and other methods during the training process.
[0082] In some embodiments, the discrete variable describing the compression degree of the dimension of the output data of the specified layer is the code length after the output data of the specified layer is compressed.
[0083] In some other examples, the discrete variable describing the degree of compression of the dimension of the output data of a specified layer may also be a variable in other forms, for example, a vector, and the trainable variable may be mapped to the value of an element in the vector through a mapping function.
[0084] The following introduces an exemplary derivation process from discrete variables to the first loss term, taking the discrete variable describing the degree of compression of the dimension of the output data of the specified layer as the code length after the output data of the specified layer is compressed as an example.
[0085] In this example, the code length of the compressed output data of the specified layer refers to the length of valid data of the specified output data in one or more dimensions.
[0086] In this example, assuming that the minimum amount of information transmitted by the neural network is I(X; Y) to ensure processing accuracy, according to the Gaussian maximum principle of continuous distribution entropy, the minimum number of output neurons for output data is However, mutual information is difficult to count directly, so the mutual information statistics and neural network compression problem are transformed into the Gaussian code length minimization problem.
[0087] However, since the code length is a discrete variable and is not differentiable, it cannot be directly optimized iteratively during the training process. Therefore, it is proposed to map the continuous trainable variables through a mapping function to a discrete variable that can describe the compression degree of the dimension of the output data of the specified layer, and the continuous trainable variables can be used for iterative updates based on back propagation and other methods during the training process.
[0088] Exemplarily, the mapping function may be a nonlinear function mapping, and a two-level sigmoid function may be used to convert a continuous and trainable variable into a compressed code length of output data of a specified layer.
[0089] Specifically, suppose that the gated discrete variable M corresponding to the output data of the specified layer to be compressed in the neural network is {m0, m1, ..., m L}, where L is the number of neurons in the output data of this layer. During the training process, M needs to be compressed as much as possible, and the number of neurons actually retained is reduced as much as possible under the premise of lossless or minimal loss of accuracy. The elements in the gated discrete variable correspond one-to-one to the neurons in the output data, where m i The value of is 0 or 1, indicating whether the neuron i in the output data passes through the gated discrete variable, that is, if m i The value of is 0, then the neuron i in the output data cannot pass through the gated discrete variable and needs to be compressed and filtered out; if m i If the value of is 1, the neuron i in the output data is retained in the compressed output data through the gated discrete variable.
[0090] Assume that l is a trainable variable whose initial value can be a random variable. The trainable variable 1 represents the compression degree of the specified layer in the neural network, giving 1 to m i The mapping function is:
[0091] m i =sigmoid(γ(i-sigmoid(l)L))
[0092] Here γ is a value set to a large value, thereby ensuring that the output is 0 or 1.
[0093] It can be understood that in the mapping function, by training the trainable variable 1, if i is less than the specified value, then m i is 0, and if i is greater than the specified value, then m i is 1. That is to say, by adjusting the value of the trainable variable l, the number of elements with a value of 1 in the controllable discrete variable M can be determined, thereby correspondingly determining the code length of the compressed output data of the specified layer.
[0094] like Figure 2As shown, it is an exemplary form of the gated discrete variable M obtained by transforming the trainable variable l through the mapping function.
[0095] At this time, one or more elements on the left side of the gated discrete variable M are 1, and the number of elements on the left side that are 1 is the code length of the output data of the specified layer after compression. One or more elements on the right side of the gated discrete variable M are 0, indicating that the elements corresponding to the one or more elements of the gated discrete variable that are 0 in the output data of the specified layer are compressed and filtered out.
[0096] In this way, the gated discrete variable M obtained by transforming the trainable variable l through the mapping function can filter the neurons in the output data of the corresponding specified layer to obtain compressed output data.
[0097] In this example, the mapping function based on this example can determine that the larger the trainable variable l is, the greater the compression degree of the specified layer is. In other examples, if the mapping function adopts other forms of functions, the relationship between the trainable variable l and the compression degree of the specified layer may be different, and this embodiment of the application does not limit this.
[0098] In addition, in some embodiments, the loss function may also include other loss terms to ensure the processing accuracy of the neural network for related tasks.
[0099] Specifically, in some embodiments, the loss function also includes a second loss term, which is used to evaluate the difference between the data probability distribution corresponding to the specified layer and the target distribution.
[0100] Exemplarily, the target distribution may be a Gaussian distribution, and the second loss term is used to evaluate the Kullback-Leibler (KL) distance between the data probability distribution corresponding to the specified layer and the target distribution. In the embodiment of the present application, the neural network is trained by the second loss term, so that the data probability distribution of the specified layer in the trained neural network can be close to the target distribution such as the Gaussian distribution, thereby ensuring the information representation capability of the output data of the specified layer.
[0101] In some examples, the data probability distribution corresponding to the specified layer may be the data probability distribution obtained by the specified layer when it is not compressed.
[0102] In other examples, the data probability distribution corresponding to the specified layer refers to the data probability distribution obtained after compressing the specified layer.
[0103] In this example, by evaluating the difference between the data probability distribution obtained after compression of the specified layer and the target distribution, the output data of the compressed specified layer in the trained neural network is made close to the target distribution such as the Gaussian distribution, thereby ensuring the information representation ability of the neurons in the compressed specified layer.
[0104] Exemplarily, in one example, the loss function including the first loss term and the second loss term may be:
[0105]
[0106] In addition, exemplarily, the loss function may also include a third loss term to evaluate the accuracy of the processing results of the neural network on the training data, that is, to describe the label error corresponding to the training data, for example, to evaluate the difference between the processing results of the neural network on the training data and the label corresponding to the training data.
[0107] Among them, D train is a dataset including at least one training data, is the third loss term, specifically the cross entropy loss between the processing result of the neural network on the training data and the label corresponding to the training data, l is the first loss term, specifically the trainable variable, KL(p, N) is the second loss term, that is, the Kullback-Leibler (KL) distance between the data probability distribution corresponding to the specified layer and the target distribution, α and β are preset coefficients.
[0108] In the actual training process, according to the training data, based on the above-mentioned first loss term and / or second loss term, etc., there can be many ways to train the neural network, which are not limited here.
[0109] Exemplarily, the training process may include multiple iterations, and the following takes the specific process of the nth iteration of the multiple iterations as an example to specifically introduce the training process, where n is a positive integer, that is, the nth iteration may be any iteration of the multiple iterations.
[0110] Specifically, during the nth iteration, the method comprises the following steps:
[0111] Get the training data for the nth iteration process;
[0112] Obtaining the processing result of the training data of the nth iteration process through the neural network of the nth iteration process;
[0113] According to the processing result of the nth iteration process and the loss function, the neural network of the nth iteration process and the trainable variables corresponding to the specified layer of the neural network in the nth iteration process are updated.
[0114] like Figure 3 The figure shows an exemplary flowchart of an iterative process in the training process.
[0115] It can be seen that after obtaining the processing result of the training data of the nth iteration process through the neural network of the nth iteration process, the sum of the first loss term for evaluating the compression degree of the specified layer in the neural network, the second loss term for evaluating the difference between the data probability distribution of the specified layer and the target distribution such as Gaussian distribution, and the third loss term for evaluating the label error between the processing result of the training data and the label of the training data can be calculated, wherein the first loss term is obtained based on the monotonic mapping of the trainable variable, and the trainable variable can be converted into a gated discrete variable through a mapping function, and the gated discrete variable is used to describe the compression degree of the output data of the specified layer.
[0116] In this way, the corresponding loss value is obtained according to the sum of the first loss term, the second loss term and the third loss term, and the trainable variables of the neural network in the nth iteration process are updated through the back propagation algorithm.
[0117] If the neural network after the nth iteration process meets the preset convergence conditions (for example, converges to a specified state or the number of iterations reaches a preset number, etc.), the training can be completed, and the dimension of the output data of the specified layer is determined according to the trainable variables after the nth iteration process, that is, the neural network after the nth iteration process is compressed according to the trainable variables after the nth iteration process, so as to obtain the trained neural network. Conversely, the neural network after the nth iteration process can be used as the neural network for the n+1th iteration process, and training can continue.
[0118] After the training is completed, the compression degree of the specified layer after training can be determined based on the trained variables, so as to compress the specified layer and obtain the compressed neural network as the trained neural network. The trained neural network can be fine-tuned according to the downstream task and then deployed to the device of the relevant reasoning scenario, or it can be directly deployed to the device of the relevant reasoning scenario.
[0119] Through any of the above embodiments of the present application, it is possible to compress the neural network in various application scenarios while ensuring good accuracy of the neural network, thereby improving the reasoning efficiency.
[0120] Some possible exemplary application scenarios are introduced below.
[0121] 1. Scenario of end-cloud collaborative reasoning.
[0122] like Figure 4 The figure shows an exemplary schematic diagram of the end-cloud collaborative system.
[0123] Any of the above embodiments of the present application can be applied to an end-cloud collaborative system to perform lossless compression on the affinity layer in a neural network deployed in the end-cloud collaborative system, thereby solving the communication consumption problem in the collaborative process and the resource consumption problem of the end-side model.
[0124] like Figure 4 In an exemplary scenario shown, the cloud device can be a server or a server cluster. The end side can be a host computer deployed at each node in the network, which can belong to a distributed node provided by a service provider, used to perform calculations and provide services on the distributed nodes. The end-cloud collaborative system in this example can be a product deployment form of an information and communications technology (ICT) network.
[0125] In this example, when the neural network is divided into multiple segmentation modules and needs to be deployed to different distributed devices for operation, the separated layers in the neural network can be considered as the connection layer between different segmentation modules, and the connection layer can also be called affinity layer. The output data of the affinity layer needs to be transmitted between the corresponding distributed devices. Therefore, the amount of the output data greatly affects the corresponding communication efficiency.
[0126] One or more affinity layers are used as designated layers of the neural network in any of the above embodiments, and the neural network is trained based on any of the above embodiments to obtain a trained neural network, in which the dimension of the output data of the affinity layer is compressed, and the accuracy of the compressed neural network can be guaranteed during the training process. In this way, the affinity layer can be pruned (for example, the neurons in the corresponding output data and the corresponding weights in the affinity layer are reset to zero or deleted), and then the pruned affinity layer is deployed to the corresponding cloud side and / or terminal side.
[0127] In this way, the end-side device processes local data through the segmentation module deployed on the end-side device in the neural network. The output data of the pruned affinity layer in the end-side device is output data with compressed dimensions. When the compressed output data is sent to the cloud-side device, the communication consumption can be reduced and the communication efficiency can be improved due to the small data volume.
[0128] Similarly, the cloud-side device can process the global data and the data received from the end-side device through the segmentation model deployed on the cloud-side device in the neural network. The output data of the pruned affinity layer in the cloud-side device is also the output data with compressed dimension. When the compressed output data is sent to the end-side device, the data volume is also small, so it can also reduce communication consumption and improve communication efficiency, thereby improving the end-cloud collaboration efficiency and the reasoning efficiency of the corresponding neural network.
[0129] 2. Multiplexing layer compression scenario of large-scale neural network.
[0130] like Figure 5 , which is an exemplary schematic diagram of a large-scale neural network such as a large prediction model.
[0131] The large-scale neural network can be deployed on a cloud server or server cluster. Based on the training process of any of the above embodiments, the key feature layers such as the reuse layer in the large-scale neural network can be compressed and trained, so as to learn the compression redundancy of the key feature layers such as the reuse layer with no or little loss of accuracy, so as to achieve lossless compression of the accuracy of the key feature layers after training.
[0132] exist Figure 5 In the example shown, the large-scale neural network includes an encoder, multiple expert models (such as expert model 1, expert model 2 to expert model n in the figure) and a classifier. Among them, multiple expert models process input features in a sparse parallel manner, and the dimension of the expert model is largely affected by the dimension of the features output by the encoder.
[0133] Therefore, by using the last layer of the encoder as the designated layer, and thereby reducing the dimension of the features output by the encoder through any of the above embodiments, the dimension of the input features of each expert model can be reduced, thereby reducing the number of parameters of the expert model, and achieving compression of the expert model, thereby reducing the redundancy of the overall neural network, reducing the number of parameters of the neural network, reducing the resource consumption of the neural network, and improving the reasoning efficiency.
[0134] 3. Information compression of knowledge graph.
[0135] In this example, information compression based on the knowledge graph can be achieved.
[0136] In the process of AI reasoning, the addition of some expert experience can improve the reasoning accuracy. The form of storing such expert experience can adopt the knowledge graph model. This type of algorithm is more commonly used in the recommendation field. When applied, the knowledge graph can be mapped to a feature layer through a model or linear mapping, and then the feature layer can be input into the neural network to assist reasoning.
[0137] In this example, the feature layer obtained by knowledge graph mapping can be used as the designated layer in any of the above embodiments. By using any of the above embodiments, the neural network including the feature layer obtained by knowledge graph mapping is trained to compress the output data of the feature layer, thereby reducing the dimension of the output data of the feature layer and improving the reasoning efficiency.
[0138] The data processing method provided in the embodiments of the present application is described above from multiple aspects. The data processing device provided in the embodiments of the present application is described below in conjunction with the accompanying drawings.
[0139] like Figure 6 As shown, the embodiment of the present application provides a data processing device 60, and the data processing device 60 includes:
[0140] An acquisition module 601 is used to acquire at least one training data, where any training data includes an image, text or voice;
[0141] The training module 602 is used to train the neural network through at least one training data and a loss function to obtain a trained neural network, wherein the loss function includes a first loss term, and the first loss term is used to evaluate the compression degree of a specified layer in the neural network. In the trained neural network, the dimension of the output data of the specified layer is obtained by compressing the specified layer in the neural network based on the compression degree of the specified layer after training.
[0142] Optionally, the first loss term is obtained based on a trainable variable, which is used to describe the degree of compression of the dimension of the output data of the specified layer.
[0143] Optionally, the trainable variables can be converted into discrete variables describing the degree of compression of the dimensions of the output data of a specified layer based on a mapping function.
[0144] Optionally, the discrete variable describing the degree of compression of the dimension of the output data of the specified layer is the code length after the output data of the specified layer is compressed.
[0145] Optionally, the loss function also includes a second loss term, which is used to evaluate the difference between the data probability distribution corresponding to the specified layer and the target distribution.
[0146] Optionally, the data probability distribution corresponding to the specified layer refers to the data probability distribution obtained after compressing the specified layer.
[0147] Optionally, the trained neural network is used to be deployed in a distributed system, the distributed system includes multiple distributed devices, the trained neural network is divided into multiple segmentation modules, different segmentation modules are used to be deployed in different distributed devices, and the designated layer includes at least one of the connection layers between different segmentation modules.
[0148] Figure 7, which is a possible logical structure diagram of an electronic device 70 provided in an embodiment of the present application. The electronic device 70 is used to implement the functions of the electronic device involved in any of the above embodiments. The electronic device 70 includes: a memory 701, a processor 702, a communication interface 703 and a bus 704. Among them, the memory 701, the processor 702, and the communication interface 703 are connected to each other through the bus 704.
[0149] The memory 701 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 701 may store a program. When the program stored in the memory 701 is executed by the processor 702, the processor 702 and the communication interface 703 are used to execute one or more steps in the above data processing method embodiment.
[0150] The processor 702 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, to execute relevant programs to implement the functions required to be executed by the acquisition module and the training module in the data processing device in the above embodiment, or to execute one or more steps in the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701, and executes one or more steps in the above data processing method embodiment in combination with its hardware.
[0151] The communication interface 703 uses a transceiver device such as but not limited to a transceiver to implement communication between the electronic device 70 and other devices or a communication network.
[0152] The bus 704 can realize a path for transmitting information between the components of the electronic device 70 (for example, the memory 701, the processor 702, and the communication interface 703). The bus 704 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0153] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the above Figure 7 The steps performed by the processor in .
[0154] In another embodiment of the present application, a computer program product is further provided, the computer program product comprising computer execution instructions, the computer execution instructions being stored in a computer readable storage medium; when the processor of the device executes the computer execution instructions, the device executes the above Figure 7 The steps performed by the processor in .
[0155] In another embodiment of the present application, a chip system is further provided, the chip system comprising a processor, the processor being used to implement the above Figure 7 In one possible design, the chip system may also include a memory, which is used to store the necessary program instructions and data for the device to write data. The chip system may be composed of a chip or may include a chip and other discrete devices.
[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0157] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc., various media that can store program codes.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire at least one training data, any of which includes an image, text, or voice; The neural network is trained by using the at least one training data and the loss function to obtain a trained neural network, wherein the loss function includes a first loss term, and the first loss term is used to evaluate the degree of compression of a specified layer in the neural network. In the trained neural network, the dimension of the output data of the specified layer is obtained by compressing the specified layer in the neural network based on the degree of compression of the specified layer after training.
2. The method according to claim 1, characterized in that The first loss term is obtained based on a trainable variable, and the trainable variable is used to describe the degree of compression of the dimension of the output data of the specified layer.
3. The method according to claim 2, characterized in that The trainable variable can be converted into a discrete variable describing the degree of compression of the dimension of the output data of the specified layer based on a mapping function.
4. The method according to claim 3, characterized in that The discrete variable describing the degree of compression of the dimension of the output data of the specified layer is the code length after the output data of the specified layer is compressed.
5. The method according to any one of claims 1 to 4, characterized in that: The loss function also includes a second loss term, which is used to evaluate the difference between the data probability distribution corresponding to the specified layer and the target distribution.
6. The method according to claim 5, characterized in that The data probability distribution corresponding to the specified layer refers to the data probability distribution obtained after compressing the specified layer.
7. The method according to any one of claims 1 to 6, characterized in that: The trained neural network is used to be deployed in a distributed system, which includes multiple distributed devices. The trained neural network is divided into multiple segmentation modules, and different segmentation modules are used to be deployed in different distributed devices. The designated layer includes at least one of the connection layers between different segmentation modules.
8. A data processing device, characterized in that: include: An acquisition module, used to acquire at least one training data, any of which includes an image, text or voice; A training module is used to train a neural network through the at least one training data and a loss function to obtain a trained neural network, wherein the loss function includes a first loss term, and the first loss term is used to evaluate the degree of compression of a specified layer in the neural network. In the trained neural network, the dimension of the output data of the specified layer is obtained by compressing the specified layer in the neural network based on the degree of compression of the specified layer after training.
9. The device according to claim 8, characterized in that The first loss term is obtained based on a trainable variable, and the trainable variable is used to describe the degree of compression of the dimension of the output data of the specified layer.
10. The device according to claim 9, characterized in that The trainable variable can be converted into a discrete variable describing the degree of compression of the dimension of the output data of the specified layer based on a mapping function.
11. The device according to claim 10, characterized in that The discrete variable describing the compression degree of the dimension of the output data of the specified layer is the code length after the output data of the specified layer is compressed.
12. The device according to any one of claims 8 to 11, characterized in that: The loss function also includes a second loss term, which is used to evaluate the difference between the data probability distribution corresponding to the specified layer and the target distribution.
13. The device according to claim 12, characterized in that The data probability distribution corresponding to the specified layer refers to the data probability distribution obtained after compressing the specified layer.
14. The device according to any one of claims 8 to 13, characterized in that: The trained neural network is used to be deployed in a distributed system, which includes multiple distributed devices. The trained neural network is divided into multiple segmentation modules, and different segmentation modules are used to be deployed in different distributed devices. The designated layer includes at least one of the connection layers between different segmentation modules.
15. An electronic device, characterized in that: The electronic device includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 1-7.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.