A model distillation method and related devices

Abstract

The application relates to the field of artificial intelligence, and discloses a model distillation method, which comprises the following steps: at a first computing node of a computing node cluster, distilling a student model by using part of the student model and part of a teacher model, and performing gradient back propagation in the distillation process in the first computing node, without relying on other computing nodes to complete the distillation of a network layer responsible by the first computing node, so that greater computing resource utilization is achieved, and the distillation process is accelerated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model distillation method and related equipment. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] With the continuous development of artificial intelligence technology, natural language human-computer interaction systems, which enable interaction between humans and machines through natural language, are becoming increasingly important. For humans and machines to interact through natural language, the system needs to be able to recognize the specific meaning of human natural language. Typically, systems identify the specific meaning of sentences by extracting key information from the sentences.

[0004] The transformer structure possesses powerful semantic expressive capabilities, enabling it to capture long dependencies within text. Since its inception, it has significantly outperformed previous models in a range of natural language processing tasks, particularly translation. Pre-trained language models based on the transformer structure have also achieved excellent results in question-answering systems, voice assistants, and other fields.

[0005] However, the Transformer model has many parameters, posing a significant challenge for inference on cloud or edge devices where latency and accuracy are critical. Therefore, some studies have proposed methods for parallel compression of the Transformer model on distributed computing nodes, such as pruning and quantization, but these require knowledge distillation to achieve good results. In existing knowledge distillation gradient backpropagation processes, updates are performed progressively from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the next or multiple layers of computing nodes have not yet completed their updates, many computing nodes are idle, resulting in long training times. Summary of the Invention

[0006] In existing distributed distillation methods, each computing node needs to rely on the gradient backpropagation results of subsequent computing nodes to update the model, resulting in many computing nodes being idle. This application provides a model distillation method in which the gradient backpropagation process of each computing node is carried out internally, without relying on subsequent computing nodes to complete the distillation of the network layer it is responsible for, and the gradients calculated by each computing node are not backpropagated to the previous computing node, thereby achieving greater utilization of computing resources and accelerating the distillation process.

[0007] Firstly, this application provides a model distillation method. The method is applied to a first computing node, which is part of a computing node cluster. The computing node cluster can perform knowledge distillation of a student model based on a teacher model. Each computing node can be responsible for distilling a portion of the student model, and the computing nodes can coordinate with each other to achieve the distillation of the student model. The computing node cluster can include multiple serially connected computing nodes, which may include a first computing node and a second computing node. The first and second computing nodes are communicatively connected.

[0008] The computing nodes (e.g., the first computing node, the second computing node) can be terminal devices such as mobile phones, tablets, laptops, and smart wearable devices, or devices with data processing capabilities such as servers and chips; there are no limitations on this.

[0009] The first computing node may be deployed with a first sub-model and a second sub-model. The first sub-model is a part of the student model and the student model also includes a third sub-model connected to the first sub-model. The second sub-model is a part of the teacher model and the teacher model also includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to the second computing node, and the second computing node is deployed with the third sub-model and the fourth sub-model.

[0010] In the student model, the third sub-model is connected before the first sub-model; in the teacher model, the fourth sub-model is connected before the second sub-model.

[0011] The phrase "the third sub-model is connected before the first sub-model" can be understood as follows: the third sub-model is closer to the input layer of the student model than the first sub-model. In other words, the first sub-model is closer to the output layer of the student model than the third sub-model. That is, the output of the third sub-model is used as the input of the first sub-model.

[0012] The phrase "the fourth sub-model is connected before the second sub-model" can be understood as follows: the fourth sub-model is closer to the input layer of the teacher model than the second sub-model. In other words, the second sub-model is closer to the output layer of the teacher model than the fourth sub-model. That is, the output of the fourth sub-model is used as the input of the second sub-model.

[0013] The method includes:

[0014] Acquire first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model; wherein the first input data may be the output obtained by the second computing node through the data processed by the third sub-model and the second input data may be the output obtained by the second computing node through the data processed by the fourth sub-model.

[0015] The student model can be obtained by compressing the teacher model, for example, by pruning the teacher model, or by quantizing the teacher model.

[0016] When performing knowledge distillation for the student model based on the teacher model, each iteration of training can include a feedforward process and a gradient backpropagation process. The feedforward process is the process by which the student model and the teacher model process training samples. Since the teacher model and the student model in each embodiment of this application are divided into multiple modules and are each handled by different computing nodes, each computing node can obtain the calculation results of the computing nodes connected to the previous level (the first computing node can obtain training samples), and perform calculations on the calculation results of the computing nodes connected to the previous level based on its own responsible module, and pass the calculation results to the computing nodes connected to the next level. Since the serial connection order of the computing nodes is consistent with the network layer arrangement of the model, from the perspective of multiple nodes as a whole, the feedforward process is still equivalent to the process of processing training samples through the complete student model and teacher model.

[0017] In this embodiment, each computing node is used to perform feedforward (or forward propagation) based on the part of the network it is responsible for, and obtain output data. Each computing node can flow the output data in a serial order. Taking the first computing node and the second computing node as an example, in the feedforward process, the second computing node can process the third input data through the third sub-model to obtain the first input data. The first input data can be used as the input of the first sub-model. Then, the first computing node can obtain the first input data. Similarly, the second computing node can process the fourth input data through the fourth sub-model to obtain the second input data. The second input data can be used as the input of the second sub-model. Then, the first computing node can obtain the second input data.

[0018] The first input data is processed through the first sub-model to obtain the first intermediate output;

[0019] The second sub-model processes the second input data to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient.

[0020] For the student model (first sub-model) under its responsibility, the first input data can be processed through the first sub-model to obtain the first intermediate output. For the teacher model (second sub-model) under its responsibility, the second input data can be processed through the second sub-model to obtain the second intermediate output.

[0021] In one possible implementation, the first sub-model and the second sub-model may include multiple network layers (e.g., multiple transformer layers), and the first intermediate output may be the output of the last layer in the first sub-model (i.e., the network layer closest to the output layer of the student model), and correspondingly, the second intermediate output may be the output of the last layer in the second sub-model (i.e., the network layer closest to the output layer of the teacher model); in another possible implementation, the first intermediate output may also be the output of each layer in the first sub-model, and correspondingly, the second intermediate output may be the output of each layer in the second sub-model.

[0022] Based on the first gradient, the first sub-model is distilled to obtain the updated first sub-model.

[0023] In the existing gradient backpropagation process of knowledge distillation, updates are performed step by step from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the network layer of the next or multiple computing nodes has not yet been updated, many computing nodes are in a resource idle state. For example, computing node 1 is responsible for the distillation of network layer 1, and computing node 2 is responsible for the distillation of network layer 2. Network layer 2 is connected after network layer 1. In the feedforward process, the output of network layer 1 is used as the input of network layer 2. During gradient backpropagation, network layer 1 is updated only after network layer 2 has been updated. That is, computing node 1 cannot update network layer 1 before computing node 2 has completed the update of network layer 2. In this embodiment, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0024] It should be understood that in the asynchronous parallel training process described above, the error generated by the student model propagates layer by layer (propagating during feedforward). During parallel training, when a previous segment is not completed, subsequent segments will be affected by the previous ones. On the other hand, although queuing eliminates the need for computation nodes to wait for each other, non-sequential computation introduces a larger error to the model compared to ordinary pipelined parallel training. Therefore, in one implementation, a teacher model can be used to mitigate the accumulation of error caused by the student model. Specifically, at the beginning of distillation training, the input data obtained by the computation node can be the output of the teacher model of the connected computation node at the previous level (at the beginning of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error propagates layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model as input at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model). As training progresses, the input is gradually changed to samples from the output queue of the student model to maintain consistency with the inference phase (using only the student model).

[0025] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0026] In one possible implementation, the first sub-model does not include the output layer of the student model, and correspondingly, the second sub-model does not include the output layer of the teacher model. When the first sub-model includes the output layer of the student model and the second sub-model includes the output layer of the teacher model, the first computation node itself does not need to wait for gradient backpropagation from the next-level computation node (it is itself the last-level computation node).

[0027] In one possible implementation, the first gradient is used only to update the first sub-model. Regarding the first gradient obtained by the first computing node, since it is only used to update the first sub-model, it may not be propagated back to the second computing node, meaning it will not be used by the second computing node when distilling the network layers it is responsible for. This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0028] In practical applications, the time taken for each computing node to execute the feedforward process of its respective network layer is not uniform. In order to further reduce the waiting time between different computing nodes, during forward propagation, the next computing node does not wait for the previous computing node to finish its calculation before starting its calculation.

[0029] In one possible implementation, an output queue I can be deployed between adjacent computing nodes (e.g., the nth computing node and the (n+1)th computing node) to collect the output of the nth computing node in the most recent t0 steps. This output queue can have a pre-set memory size and be updated according to a first-in, first-out (FIFO) principle. The (n+1)th computing node can always obtain data from the output queue as its own input without waiting. This design avoids the load imbalance problem from lagging modules, achieving a near-theoretical linear speedup of N times with N training devices.

[0030] Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, a first queue (for the student model) can be deployed between the second computing node and the first computing node. The second computing node can store the output data (including the first input data) into the first queue. Then, the first computing node can obtain the first input data from the first queue. The first queue is used to store at least one piece of first data from the second computing node. Each piece of first data is the output obtained by the second computing node through the processing of data by the third sub-model.

[0031] In one possible implementation, the first computing node is also communicatively connected to a third computing node, and a third queue can be deployed (e.g., a storage location between the first and third computing nodes, or on the first computing node, or on the third computing node). The student model also includes a fifth sub-model connected after the first sub-model. After the first computing node processes the first input data through the first sub-model, the processed first intermediate output can be stored in the third queue (the entity performing the storage action is not limited to the first computing node). That is, the first intermediate output can be passed to the third queue, which is used to store the first intermediate output so that the third computing node can obtain the first intermediate output from the third queue. The first intermediate output is used as input data for the feedforward process when performing model distillation on the third sub-model.

[0032] Since the distillation process between computing nodes is independent, after completing the update process with one gradient backpropagation, the computing node can directly obtain the next input data and perform feedforward without waiting. Furthermore, since the next input data is already stored in the queue deployed with the previous level computing node, there is no need to wait for the feedforward process of the previous level computing node to complete, which further improves the utilization of computing resources of the computing node and accelerates the distillation process.

[0033] Taking the first computing node as an example, in the previous iteration, the first computing node can distill the network layers of the student network it is responsible for to obtain the first sub-model, and directly obtain the first input data from the first queue in response to obtaining the first sub-model. After completing the distillation of the first sub-model, in response to obtaining the updated first sub-model, it can obtain the third input data (i.e., the next input data) from the first queue, and then perform a feedforward process for model distillation of the updated first sub-model based on the third input data.

[0034] Similarly, for the teacher model, an output queue can also be deployed. Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, a second queue can be deployed (for the teacher model, the deployment location can be a storage location between the first and second computing nodes, or deployed on the first computing node, or deployed on the second computing node). The second computing node can store the output data (including the second input data) into the second queue. Then, the first computing node can obtain the second input data from the second queue. The second queue is used to store at least one second data from the second computing node, and each second data is the output obtained by the second computing node through the data processing of the fourth sub-model.

[0035] Secondly, this application provides a model distillation method, which is applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model, wherein the first sub-model is a partial model of a student model, the second sub-model is a partial model of a teacher model, and the teacher model further includes a third sub-model connected to the second sub-model. The first computing node and the second computing node are communicatively connected, and the second computing node deploys the third sub-model. The method includes:

[0036] Obtain first input data from the second computing node, wherein the first input data is the output data of the third sub-model;

[0037] In this embodiment, the computing node cluster can perform knowledge distillation of the student model based on the teacher model. Each computing node can be responsible for distilling a portion of the student model, and the computing nodes can coordinate with each other to achieve the distillation of the student model. The computing node cluster can include multiple serially connected computing nodes, which may include a first computing node and a second computing node, and the first and second computing nodes are communicatively connected.

[0038] Among them, the student model can be obtained by compressing the teacher model, for example, the student model can be obtained by pruning the teacher model, for example, the student model can be obtained by quantizing the teacher model;

[0039] In this embodiment of the application, the teacher model may include a second sub-model and a third sub-model, wherein the output of the third sub-model is used as the input of the second sub-model. That is, the second sub-model and the third sub-model are part of the teacher model, and the third sub-model is the model in the teacher model that is connected before the second sub-model.

[0040] The first input data is processed through the first sub-model to obtain the first intermediate output;

[0041] The first input data is processed by the second sub-model to obtain a second intermediate output. The first intermediate output and the second intermediate output are used to determine the first gradient.

[0042] Based on the first gradient, the first sub-model is distilled to obtain the updated first sub-model.

[0043] In the existing gradient backpropagation process of knowledge distillation, updates are performed step by step from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the network layer of the next or multiple computing nodes has not yet been updated, many computing nodes are in a resource idle state. For example, computing node 1 is responsible for the distillation of network layer 1, and computing node 2 is responsible for the distillation of network layer 2. Network layer 2 is connected after network layer 1. In the feedforward process, the output of network layer 1 is used as the input of network layer 2. During gradient backpropagation, network layer 1 is updated only after network layer 2 has been updated. That is, computing node 1 cannot update network layer 1 before computing node 2 has completed the update of network layer 2. In the embodiments described in the first aspect, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for), thereby achieving greater utilization of computing resources and thus accelerating the distillation process.

[0044] However, in the asynchronous parallel training process described above, the error generated by the student model propagates layer by layer (propagating during feedforward). During parallel training, when a previous segment is not completed, subsequent segments will be affected by the previous ones. On the other hand, although queuing eliminates the need for computation nodes to wait for each other, non-sequential computation introduces a larger error to the model compared to ordinary pipelined parallel training. Therefore, in one implementation, a teacher model can be used to mitigate the accumulation of error caused by the student model. Specifically, at the beginning of distillation training, the input data obtained by the computation node can be the output of the teacher model of the connected computation node at the previous level (at the beginning of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error propagates layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model (the output of the third sub-model) as the input (as the input of the first sub-model) at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model). As training progresses, the input is gradually changed to samples from the student model's output queue to maintain consistency with the inference phase (using only the student model).

[0045] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0046] In one possible implementation, the first loss is used only to update the first sub-model.

[0047] In one possible implementation, obtaining the first input data from the second computing node includes:

[0048] The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the data, and the at least one first data includes the first input data.

[0049] In one possible implementation, the first queue is a first-in, first-out queue.

[0050] In one possible implementation, before distilling the first sub-model according to the first gradient, the at least one first data includes second input data, the second input data being the output data of the third sub-model; after distilling the first sub-model according to the first gradient, the method further includes:

[0051] In response to obtaining the updated first sub-model, the second input data is obtained from the first queue, and the second input data is used as input data for the feedforward process when performing model distillation on the updated first sub-model.

[0052] In one possible implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the second sub-model.

[0053] This application also provides a model distillation method, including:

[0054] Receive performance requirement information sent by the terminal device;

[0055] In this embodiment of the application, the terminal device can send performance requirement information to the cloud-side device, wherein the performance requirement information may include at least one of the following: accuracy requirement, latency requirement, or model compression ratio requirement;

[0056] In some scenarios, when a terminal device needs to obtain a model for inference from a cloud-based device, it can send a model acquisition request to the cloud-based device. This request may include performance requirement information. The cloud-based device can receive this performance requirement information and determine a suitable student model based on it. When high accuracy is required, the student model can be larger (e.g., with a larger number of parameters or a larger storage space) to provide a high-performance model. When high latency is required, the student model can be smaller to provide a model that can be quickly derived. When a high model compression ratio (the ratio of the size of the model before scaling to the size of the model after scaling) is required, the student model can be larger to provide a model with a high compression ratio.

[0057] Obtain the teacher model and, based on the performance requirement information, determine the student model, wherein the student model is obtained by compressing the teacher model;

[0058] Knowledge distillation of a student network is performed using multiple computing nodes to obtain a knowledge-distilled student network. Each computing node is responsible for the knowledge distillation of a portion of the student network. Each computing node performs feedforward based on its own portion of the network to obtain output data. Each computing node also determines gradients based on the feedforward data of its own portion of the network and performs gradient backpropagation within each computing node to distill its own portion of the network. The gradients determined by each computing node are not propagated to connected computing nodes.

[0059] The knowledge-distilled student network is sent to the terminal device.

[0060] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0061] In one possible implementation, a first-in-first-out (FIFO) queue is also deployed between the multiple serially connected computing nodes to pass the output data to the FIFO queue between the connected computing nodes, so that the connected computing nodes can obtain the output data from the FIFO queue.

[0062] In this embodiment, once the cloud-side device determines the knowledge-distilled student network, it can send the determined knowledge-distilled student network to the terminal device. The terminal device can then perform inference based on the received student model. It should be noted that the terminal device can also perform further model compression processing on the received knowledge-distilled student network; this is not a limitation.

[0063] In this way, cloud-side devices can determine a student model that suits the needs of terminal devices based on the performance requirements of the terminal devices.

[0064] Thirdly, this application provides a computing node cluster, including multiple computing nodes connected in series, wherein the multiple computing nodes are used to perform knowledge distillation for a student network, and each computing node is responsible for knowledge distillation of a portion of the student network; wherein...

[0065] Each computing node is used to perform feedforward based on its own portion of the network to obtain output data;

[0066] Each of the computing nodes is also used to determine a gradient based on the output data, and to distill the portion of the network it is responsible for according to the gradient.

[0067] In one possible implementation, each of the computing nodes is used to perform gradient backpropagation within each computing node, and the gradients determined by each computing node are not propagated to the connected computing nodes.

[0068] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0069] In one possible implementation, a first-in-first-out queue is further deployed among the multiple serially connected computing nodes, and the method further includes:

[0070] Each compute node passes the output data to a first-in-first-out queue among connected compute nodes, so that the connected compute nodes can retrieve the output data from the first-in-first-out queue.

[0071] Fourthly, this application provides a model distillation apparatus, which is applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model, wherein the first sub-model is a partial model of a student model, and the student model further includes a third sub-model connected to the first sub-model; the second sub-model is a partial model of a teacher model, and the teacher model further includes a fourth sub-model connected to the second sub-model; the first computing node is communicatively connected to a second computing node, and the second computing node deploys the third sub-model and the fourth sub-model; the apparatus includes:

[0072] The acquisition module is used to acquire first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model;

[0073] The feedforward module is used to process the first input data through the first sub-model to obtain the first intermediate output;

[0074] The second sub-model processes the second input data to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient.

[0075] The distillation module is used to distill the first sub-model according to the first gradient to obtain the updated first sub-model.

[0076] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0077] In one possible implementation, the first sub-model does not include the output layer of the student model.

[0078] In one possible implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the second sub-model.

[0079] In one possible implementation, the first gradient is used only to update the first sub-model.

[0080] In one possible implementation, the acquisition module is specifically used for:

[0081] The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the input data, and the at least one first data includes the first input data.

[0082] In one possible implementation, the first queue is a first-in, first-out queue.

[0083] In one possible implementation, before distilling the first sub-model according to the first gradient, the at least one first data includes third input data, the third input data being the output data of the third sub-model; the acquisition module is further configured to:

[0084] In response to obtaining the updated first sub-model, the third input data is obtained from the first queue, the third input data being used as input data for the feedforward process when performing model distillation on the updated first sub-model.

[0085] In one possible implementation, the acquisition module is specifically used for:

[0086] The second input data is obtained from the second queue, wherein the second queue is used to store at least one second data from the second computing node, each second data being the output obtained by the second computing node through the processing of input data by the fourth sub-model, and the at least one second data includes the second input data.

[0087] In one possible implementation, the first computing node is also communicatively connected to a third computing node, the student model further includes a fifth sub-model connected after the first sub-model, the first intermediate output is used to be passed to a third queue, the third queue is used to store the first intermediate output so that the third computing node can obtain the first intermediate output from the third queue, and the first intermediate output is used as input data for the feedforward process when performing model distillation on the third sub-model.

[0088] Fifthly, this application provides a model distillation apparatus applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model, wherein the first sub-model is a partial model of a student model, the second sub-model is a partial model of a teacher model, and the teacher model further includes a third sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, and the second computing node deploys the third sub-model. The apparatus includes:

[0089] The acquisition module is used to acquire first input data from the second computing node, wherein the first input data is the output data of the third sub-model;

[0090] The feedforward module is used to process the first input data through the first sub-model to obtain the first intermediate output;

[0091] The first input data is processed by the second sub-model to obtain a second intermediate output. The first intermediate output and the second intermediate output are used to determine the first gradient.

[0092] The distillation module is used to distill the first sub-model according to the first gradient to obtain the updated first sub-model.

[0093] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0094] In one possible implementation, the first loss is used only to update the first sub-model.

[0095] In one possible implementation, the acquisition module is specifically used for:

[0096] The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the data, and the at least one first data includes the first input data.

[0097] In one possible implementation, the first queue is a first-in, first-out queue.

[0098] In one possible implementation, before distilling the first sub-model according to the first gradient, the at least one first data includes second input data, the second input data being the output data of the third sub-model; the acquisition module is further configured to:

[0099] After distilling the first sub-model according to the first gradient, in response to obtaining the updated first sub-model, the second input data is obtained from the first queue, and the second input data is used as input data for the feedforward process when performing model distillation on the updated first sub-model.

[0100] In one possible implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the second sub-model.

[0101] This application provides a model distillation apparatus, comprising:

[0102] The receiving module is used to receive performance requirement information sent by the terminal device.

[0103] The acquisition module is used to acquire the teacher model and determine the student model based on the performance requirement information, wherein the student model is obtained by compressing the teacher model;

[0104] The knowledge distillation module is used to perform knowledge distillation on a student network based on multiple computing nodes to obtain a knowledge-distilled student network. Each computing node is responsible for the knowledge distillation of a portion of the student network. Each computing node is used to perform feedforward based on the portion of the network it is responsible for to obtain output data. Each computing node is also used to determine the gradient based on the data obtained from the feedforward of the portion of the network it is responsible for, and to perform gradient backpropagation within each computing node to distill the portion of the network it is responsible for. The gradients determined by each computing node are not propagated to the connected computing nodes.

[0105] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0106] In one possible implementation, a first-in-first-out (FIFO) queue is also deployed between the multiple serially connected computing nodes to pass the output data to the FIFO queue between the connected computing nodes, so that the connected computing nodes can obtain the output data from the FIFO queue.

[0107] The sending module is used to send the knowledge-distilled student network to the terminal device.

[0108] In this way, cloud-side devices can determine a student model that suits the needs of terminal devices based on the performance requirements of the terminal devices.

[0109] In a sixth aspect, embodiments of this application provide a model distillation apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect and any optional methods thereof, and the second aspect and any optional methods thereof.

[0110] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods of the first aspect and any optional method thereof, and the methods of the second aspect and any optional method thereof.

[0111] Eighthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any optional method thereof, and the second aspect and any optional method thereof.

[0112] Ninthly, this application provides a chip system including a processor for supporting a model distillation apparatus in performing the functions involved in the foregoing aspects, such as transmitting or processing data or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution or training device. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0113] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0114] Figure 2 It is a natural language processing system;

[0115] Figure 3 It is a natural language processing system;

[0116] Figure 4A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0117] Figure 5 A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0118] Figure 6 A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0119] Figure 7 A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0120] Figure 8 A schematic diagram of the natural language processing related devices provided in the embodiments of this application;

[0121] Figure 9 This is a schematic diagram of a transformer layer architecture;

[0122] Figure 10 A schematic diagram of the relevant equipment provided in the embodiments of this application;

[0123] Figure 11 This application provides an example of a model distillation method.

[0124] Figure 12 This is a schematic diagram of the structure of a neural network model in an embodiment of this application;

[0125] Figure 13 This is a schematic diagram of a transformer layer structure;

[0126] Figure 14 A schematic diagram of the operation of an attention head;

[0127] Figure 15 A schematic diagram of model width scaling provided for an embodiment of this application;

[0128] Figure 16 A schematic diagram of model depth scaling provided for an embodiment of this application;

[0129] Figure 17 This is a schematic diagram illustrating an embodiment of a model distillation method provided in this application.

[0130] Figure 18 This is a schematic diagram illustrating an embodiment of a model distillation method provided in this application.

[0131] Figure 19 This is a schematic diagram illustrating an embodiment of a model distillation method provided in this application.

[0132] Figure 20A schematic diagram of a model distillation apparatus provided in an embodiment of this application;

[0133] Figure 21 A schematic diagram of a model distillation apparatus provided in an embodiment of this application;

[0134] Figure 22 A schematic diagram of a model distillation apparatus provided in an embodiment of this application;

[0135] Figure 23 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0136] Figure 24 This is a schematic diagram of a training device provided in an embodiment of this application;

[0137] Figure 25 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0138] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0139] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0140] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0141] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0142] (1) Infrastructure

[0143] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0144] (2) Data

[0145] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0146] (3) Data processing

[0147] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0148] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0149] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0150] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0151] (4) General ability

[0152] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0153] (5) Smart Products and Industry Applications

[0154] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0155] This application can be applied, but is not limited to, to the field of natural language processing in the field of artificial intelligence. Specifically, it can be applied to neural network compression and neural network inference in the field of natural language processing. The following will introduce several application scenarios that have been implemented in products.

[0156] To better understand the solutions of the embodiments of this application, the following will first combine... Figures 1 to 8 A brief introduction to the possible application scenarios of the embodiments of this application is provided.

[0157] Scenario 1: Neural Network Compression

[0158] This application can be applied to neural network compression-related services, specifically neural network compression services provided by cloud-side servers. In this case, users can transmit information related to model compression to the cloud-side neural network compression system (e.g., a cloud server) through their user devices. This information may include the user's performance requirements for the compressed model. The cloud-side server can then use a specific neural network compression method based on the performance requirements uploaded by the user to obtain a compression result (e.g., the distilled student model in this embodiment) and send the compression result to the user device.

[0159] Figure 2 A neural network compression system is illustrated. This neural network search system may include a user device and a neural network search device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of neural network compression; typically, the user initiates the neural network compression request through the user device.

[0160] The aforementioned neural network compression device can be a cloud server, network server, application server, or management server, or any device or server with neural network compression capabilities. The neural network compression device receives neural network compression requests from smart terminals via an interactive interface, then performs neural network compression using machine learning, deep learning, search, inference, and decision-making methods through a storage device and processor. The compressed result (e.g., the distilled student model in this embodiment) is then fed back to the user device. The storage device in the neural network compression device can be a general term, including local storage and a database storing historical data. The database can be located on the neural network compression device or on other network servers.

[0161] exist Figure 2 In the neural network compression system shown, the user equipment can receive user instructions, such as receiving the user's input of model performance requirements for neural network compression, and then sending a request to the neural network compression device.

[0162] exist Figure 2 In this application, the neural network compression device can execute the model distillation method of the embodiments of this application.

[0163] Figure 3 Another neural network compression system is shown, in Figure 3 In this context, the user equipment (UE) directly functions as a neural network compression device. This UE can directly receive model performance requirements for neural network compression from user input and perform neural network compression directly on its own hardware. The specific process is similar to... Figure 2 Similar to the description above, it will not be repeated here.

[0164] exist Figure 3 In this application, the user equipment itself can execute the model distillation method of the present application embodiment.

[0165] In one possible implementation, the compression service provided in this application can be applied to the compression of transformer models. This compression service can be used to quickly and efficiently help users compress a given or specified base model (transformer model) to produce a new model (transformer model) that meets hardware constraints. For example... Figure 4 As shown, the user inputs the hardware constraints of the required model, such as computational constraints, calculates the need to use the proposed efficient parallel training scheme to compress the basic model, and outputs a new model that meets the user's requirements.

[0166] It should be understood, and referenced Figure 5The neural network compression device in this embodiment can be in the form of a computing node cluster, which may include multiple serially connected computing nodes that can collaborate to perform compression. These computing nodes can be different devices on the cloud or edge, or different chips from the same device or different devices. The computing nodes can communicate with each other. During the training phase, multiple edge computing nodes can collaborate to deploy a large model in blocks across each device, collaboratively performing compression training to obtain the final smaller model. Finally, the smaller model is deployed to various platforms for inference via communication.

[0167] Scenario 2: Natural Language Processing

[0168] Figure 6 A natural language processing (NLP) system is illustrated, comprising user devices and data processing devices. The user devices include smart terminals such as mobile phones, personal computers, or information processing centers. The user devices are the initiators of natural language data processing, acting as the initiators of requests such as language question answering or queries; typically, users initiate requests through their user devices.

[0169] The aforementioned data processing device can be a cloud server, network server, application server, or management server, or any device or server with data processing capabilities. The data processing device receives query statements / voice / text questions (e.g., the data to be processed in this embodiment) from smart terminals via an interactive interface. It then performs language data processing (e.g., data processing using the distilled student model in this embodiment) through a storage device and a data processing processor, employing methods such as machine learning, deep learning, search, reasoning, and decision-making. The processing results (e.g., the data processing results in this embodiment) are then fed back to the user device. The storage device in the data processing device can be a general term, including local storage and a database storing historical data. The database can be located on the data processing device or on other network servers.

[0170] exist Figure 6In the natural language processing system shown, the user device can receive user instructions. For example, the user device can receive a piece of text input by the user and then send a request to the data processing device, causing the data processing device to perform natural language processing applications (such as text classification, text reasoning, named entity recognition, translation, etc.) on the text received by the user device, thereby obtaining the processing results of the corresponding natural language processing application for the text (such as classification results, reasoning results, named entity recognition results, translation results, etc.). For example, the user device can receive a piece of Chinese input by the user and then send a request to the data processing device, causing the data processing device to perform entity classification on the Chinese text, thereby obtaining the entity classification result for the Chinese text; for example, the user device can receive a piece of Chinese input by the user and then send a request to the data processing device, causing the data processing device to translate the Chinese text into English, thereby obtaining the English translation of the Chinese text.

[0171] Figure 7 This demonstrates another natural language processing system, in Figure 7 In this context, the user equipment directly functions as a data processing device. This user equipment can directly receive input from the user (e.g., the data to be processed in this embodiment) and process it directly using its own hardware. The specific process is similar to... Figure 6 Similar to the description above, it will not be repeated here.

[0172] exist Figure 7 In the natural language processing system shown, the user device can receive user instructions, such as receiving a piece of text input by the user, and then performing natural language processing applications (e.g., text classification, text reasoning, named entity recognition, translation, etc.) on that text to obtain the processing results of the corresponding natural language processing application (e.g., classification results, reasoning results, named entity recognition results, translation results, etc.). For example, the user device can receive a piece of Chinese input by the user and perform entity classification on that piece of Chinese to obtain the entity classification result for that piece of Chinese; for example, the user device can receive a piece of Chinese input by the user and translate that piece of Chinese into English to obtain the English translation of that piece of Chinese.

[0173] In this embodiment of the application, the user equipment may store a target neural network and perform inference tasks based on the target neural network each time the operating system (OS) or application (APP) calls the model.

[0174] Figure 8 This is a schematic diagram of the natural language processing related device 300 provided in the embodiments of this application.

[0175] The above Figure 6 and Figure 7 The user equipment in the context can specifically be Figure 8 Local device 301 or local device 302 in the system. Figure 6 The data processing equipment in the middle can specifically be Figure 8 The execution device 310 in the process includes a data storage system 350 that can store the data to be processed by the execution device 310. The data storage system 350 can be integrated into the execution device 310 or set up in the cloud or on other network servers.

[0176] Figure 6 and Figure 7 The processor in the system can perform data training / machine learning / deep learning using neural network models or other models, and use the trained model (such as the distilled student model in the embodiments of this application) to perform natural language processing applications (such as text classification, sequence labeling, reading comprehension, text generation, text reasoning, translation, etc.) on text sequences to obtain the corresponding processing results.

[0177] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0178] (1) Neural Network

[0179] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0180]

[0181] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0182] (2) Transformer layer

[0183] Reference Figure 9 , Figure 9 This is a schematic diagram of a transformer layer architecture, such as Figure 9 As shown, the neural network includes an embedding layer and at least one transformer layer. The at least one transformer layer can be N transformer layers (N being an integer greater than 0). Each transformer layer includes sequentially adjacent attention layers, add and normalize layers, feed-forward layers, and add and normalize layers. In the embedding layer, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the layer above the transformer layer. Using any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained from the last transformer layer in at least one transformer layer are used as feature representations of the current input.

[0184] Next, we will explain each of the above steps in detail with specific examples.

[0185] First, in the embedding layer, the current input is embedded to obtain multiple feature vectors.

[0186] The embedding layer can be called the input embedding layer. The current input can be text, such as a paragraph or a sentence. The text can be Chinese, English, or other languages. After obtaining the current input, the embedding layer can perform embedding processing on each word in the current input to obtain the feature vector of each word. In some embodiments, such as... Figure 1As shown, the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input to obtain word embedding vectors for each word. In the positional encoding layer, the position of each word in the current input can be obtained, and then position vectors can be generated for the positions of each word. In some examples, the position of each word can be the absolute position of each word in the current input. Taking the current input "When should I pay off Huabei" as an example, the position of "When" can be represented as the first position, the position of "should" can be represented as the second position, and so on. In some examples, the position of each word can be the relative position between each word. Still taking the current input "When should I pay off" as an example, the position of "When" can be represented as before "should", the position of "should" can be represented as after "When" and before "pay off", and so on. When the word embedding vectors and position vectors of each word in the current input are obtained, the position vectors of each word can be combined with the corresponding word embedding vectors to obtain word feature vectors, that is, multiple feature vectors corresponding to the current input are obtained. The multiple feature vectors can be represented as an embedding matrix with a preset dimension. It can be set that the number of feature vectors in the multiple feature vectors is M, and the preset dimension is H dimensions, then the multiple feature vectors can be represented as an M×H embedding matrix.

[0187] Secondly, P input vectors are obtained from the layer above the first transformer layer. Taking any first input vector among the P input vectors as the center, based on the correlation degree between each input vector within the preset attention window range and the first input vector, an intermediate vector corresponding to the first input vector is obtained, and thus P intermediate vectors corresponding to the P input vectors are determined. The attention layer can also be referred to as a multi-head attention layer. In one example, the attention layer can be a fixed window multi-head attention layer.

[0188] In some embodiments, the first transformer layer can be the layer below the aforementioned embedding layer, and the P input vectors are the plurality of feature vectors obtained from the embedding layer. In some embodiments, at least one transformer layer in the neural network provided in this specification further includes a second transformer layer. This second transformer layer is the layer above the first self-attention layer, and the P input vectors are the P output vectors output by the second transformer layer. In the last transformer layer of this neural network, the plurality of output vectors obtained through the above steps can be used as a feature representation of the current input. This feature representation is a feature representation of the current input suitable for computer processing and can be used for tasks such as text similarity, text classification, reading comprehension, and machine translation.

[0189] (3) Attention mechanism

[0190] Attention mechanisms mimic the internal processes of biological observation—aligning internal experience with external senses to increase the precision of observation in specific areas. They enable the rapid sifting of high-value information from a large volume of data using limited attentional resources. Attention mechanisms can quickly extract important features from sparse data and are therefore widely used in natural language processing tasks, particularly machine translation. Self-attention mechanisms, an improvement on attention mechanisms, reduce reliance on external information and are better at capturing the internal correlations of data or features. The core idea of ​​attention mechanisms can be rewritten as follows:

[0191] In this formula, Lx = ||Source|| represents the length of the Source. The meaning is that the elements in the Source are imagined as a series of data pairs. Given a Query element in the Target, the similarity or relevance between the Query and each Key is calculated to obtain the weight coefficient of the Value corresponding to each Key. Then, the Values ​​are weighted and summed to obtain the final Attention value. Therefore, the Attention mechanism essentially performs a weighted sum of the Values ​​of the elements in the Source, while the Query and Key are used to calculate the weight coefficients of their corresponding Values. Conceptually, Attention can be understood as selectively filtering a small amount of important information from a large amount of information and focusing on this important information, ignoring most of the unimportant information. The focusing process is reflected in the calculation of the weight coefficients; the larger the weight, the more focused it is on its corresponding Value. That is, the weight represents the importance of the information, and the Value is the corresponding information. Self-attention can be understood as intra attention. The attention mechanism occurs between the elements of the Target (Query) and all elements of the Source. Self-attention refers to the attention mechanism that occurs between elements within the Source or between elements within the Target. It can also be understood as the attention calculation mechanism in the special case where Target = Source. The specific calculation process is the same, only the calculation object changes.

[0192] (4) Natural Language Processing (NLP)

[0193] Natural language is human language, and Natural Language Processing (NLP) is the processing of human language. NLP is a systematic process of analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform numerous automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering systems, and topic segmentation, among others.

[0194] For example, natural language processing tasks can be categorized as follows.

[0195] Sequence labeling: For each word in a sentence, the model is required to provide a classification category based on the context. Examples include Chinese word segmentation, part-of-speech tagging, named entity recognition, and semantic role labeling.

[0196] Classification task: Output a classification value for the entire sentence, such as text classification.

[0197] Sentence relation inference: Given two sentences, determine whether they have a certain nominal relation. Examples include entailment, QA, semantic rewriting, and natural language inference.

[0198] Generative tasks: Output one piece of text and generate another piece of text. Examples include machine translation, text summarization, poetry writing and sentence construction, and picture description.

[0199] (5) Neural network compression

[0200] Compression can include at least one of the following methods: model pruning, weight sharing, kernel sparsity (regularization), quantization, binarization, Low-rank decomposition, etc.

[0201] Quantization, specifically neural network quantization, is a model compression technique that converts floating-point storage (operations) into integer storage (operations). For example, if the model parameters of a model were originally represented using float32 (32-bit floating-point), after quantization, the model parameters would be represented using int8 (8-bit fixed-point). Through model quantization, the computation speed of the model is improved at the cost of a small loss of precision.

[0202] The essence of model quantization is the conversion / mapping between two data types. One implementation of converting floating-point data (data of floating-point data type) to fixed-point data (data of fixed-point data type) can be achieved using the following formula:

[0203]

[0204] Where R is the input floating-point data, Q is the fixed-point data after quantization of the floating-point data R, Z represents the zero-point value, and S represents the scale. Therefore, once S and Z are determined, the conversion between these two data types can be performed. There are many ways to determine S and Z, for example:

[0205]

[0206] Z = Q max -R max / S;

[0207] Where Rmax represents the maximum value of the input floating-point data, Rmin represents the minimum value of the input floating-point data, Qmax represents the maximum value of the fixed-point data, and Rmin represents the minimum value of the fixed-point data.

[0208] The conversion between fixed-point data with different numbers of bits (number of bits, 1 bit = 1 bit) can refer to the above conversion method between floating-point data and fixed-point data, or it can be other conversion methods in the prior art, which will not be elaborated here.

[0209] In one implementation, the conversion between 4 bits and 8 bits can be performed using the methods described above, while the conversion between floating-point data and 2 bits (1 bit) can be performed using the following formula:

[0210]

[0211] Two bits can be represented as three numbers: -1, 0, and 1. T is a threshold; when the floating-point data is greater than or equal to T, the converted 2-bit fixed-point data is 1. When the floating-point data is less than -T, its value is converted to -1. When the floating-point data is any other value, its value is converted to 0. The conversion method for 1 bits is similar to that of 2 bits, but its fixed-point values ​​are only -1 and 1, where T is 0.

[0212] Pruning can be understood as removing parts of the network structure.

[0213] After model compression, the storage resources required to store the compressed model will be reduced. In order to ensure the accuracy of the compressed model, knowledge distillation can be performed on the compressed model. Specifically, the uncompressed model can be used as the teacher model, and the compressed model can be used as the student model. Knowledge distillation (or simply distillation) can be performed on the student model.

[0214] The teacher model can also be called a teacher model, guidance model, etc., and this is not a limitation.

[0215] In knowledge distillation, a compressed model (a simpler network) can be trained using the uncompressed model (a pre-trained complex network), allowing the simpler network to have the same or similar data processing capabilities as the complex network. Knowledge distillation essentially transfers the "knowledge" of a trained complex network to a simpler network. The simpler network, in particular, can have fewer parameters than the complex network.

[0216] It should be noted that the aforementioned identical or similar data processing capabilities can be understood as the student model and teacher model obtaining the same or similar processing results after knowledge distillation when processing the same data.

[0217] (6) Loss Function

[0218] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0219] When performing knowledge distillation, it is necessary to construct the loss based on the output of the teacher model and the output of the student model. The model output used to construct the loss can be the output of the output layer of the model, the intermediate feature map output of the intermediate network layer, or the result obtained by processing the output of the output layer and / or the intermediate feature map output of the intermediate network layer.

[0220] (7) Backpropagation algorithm

[0221] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0222] The following describes a more detailed architecture of the execution entity that performs the neural network search method in the embodiments of this application.

[0223] The following is combined with Figure 10 The system architecture provided in the embodiments of this application will be described in detail. Figure 10 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 10As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0224] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0225] The data acquisition device 560 is used to collect training samples. Training samples can be image data, text data, audio data, etc. In this embodiment, the training samples are the data used when distilling the student model. After collecting the training samples, the data acquisition device 560 stores them in the database 530.

[0226] Training device 520 can maintain training samples in database 530 and distill the student model based on the teacher model to obtain target model / rule 501. In this embodiment, target model / rule 501 can be the distilled student model.

[0227] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0228] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 10 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud, etc.

[0229] Specifically, the training device 520 can transfer the distilled student model to the execution device 510.

[0230] exist Figure 10In the process, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. Users can input data (such as the data to be processed in the embodiments of this application) into the I / O interface 512 through the client device 540.

[0231] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0232] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0233] Finally, the I / O interface 512 presents the processing results (such as the data processing results in this embodiment) to the client device 540, thereby providing them to the user.

[0234] exist Figure 10 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0235] It is worth noting that, Figure 10 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 10In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0236] From the inference side of the model:

[0237] In this embodiment, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the data processing method in this embodiment.

[0238] In this embodiment of the application, the computing module 511 of the execution device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0239] Specifically, the computing module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The data processing method provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the data processing method provided in this application embodiment.

[0240] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the data processing method provided in the embodiments of this application can also be implemented by the hardware system without the function of executing instructions in the computing module 511 of the execution device 520, which is not limited here.

[0241] From the training side of the model:

[0242] In this embodiment of the application, the training device 520 can access the memory ( Figure 10 The model distillation method in this application embodiment can be implemented by code stored in the training device 520 (not shown in the figure) or separately deployed from the training device 520.

[0243] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0244] Specifically, the training device 520 can be a hardware system with instruction execution capabilities. The data processing method provided in this application embodiment can be software code stored in a memory. The training device 520 can retrieve the software code from the memory and execute the retrieved software code to implement the model distillation method provided in this application embodiment.

[0245] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the model distillation method provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0246] It should be understood that there can be multiple training devices (each serving as a computing node).

[0247] Reference Figure 11 , Figure 11 This is an illustration of an embodiment of a model distillation method provided in this application. The model distillation method provided in this application can be applied to a first computing node. The first computing node can be a terminal device such as a mobile phone, tablet, laptop, or smart wearable device. The first computing node can also be a server, chip, or other device with data processing capabilities. Figure 11 As shown, an embodiment of this application provides a model distillation method comprising:

[0248] 1101. Obtain first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model.

[0249] The first computing node may be deployed with a first sub-model and a second sub-model. The first sub-model is a part of the student model, and the student model may include a third sub-model connected to the first sub-model. The second sub-model is a part of the teacher model, and the teacher model may include a fourth sub-model connected before the second sub-model. The first computing node is communicatively connected to the second computing node, and the second computing node may be deployed with the third sub-model and the fourth sub-model.

[0250] The first computing node includes a first sub-model and a second sub-model. This can be understood as the first computing node storing the first sub-model and the second sub-model (the storage location can be in the first computing node or outside the first computing node; for details, please refer to the description of the training device in the above embodiments).

[0251] The second computing node includes the third sub-model and the fourth sub-model. It can be understood that the second computing node can store the third sub-model and the fourth sub-model (the storage location can be in the second computing node or outside the second computing node, and you can refer to the description of the training device in the above embodiment for details).

[0252] In this embodiment, the computing node cluster can perform knowledge distillation of the student model based on the teacher model. Each computing node can be responsible for distilling a portion of the student model, and the computing nodes can coordinate with each other to achieve the distillation of the student model. The computing node cluster can include multiple serially connected computing nodes, which may include a first computing node and a second computing node, and the first and second computing nodes are communicatively connected.

[0253] The student model can be obtained by compressing the teacher model, for example, by pruning the teacher model, or by quantizing the teacher model.

[0254] The student and teacher models will be described next:

[0255] In one possible implementation, the terminal device can send compression performance requirements to the cloud-side device. Specifically, the terminal device can send a request to the cloud-side device to obtain NLP-related task models and upload performance requirements to be met, wherein the performance requirements include, but are not limited to, at least one of accuracy requirements, latency requirements, or model compression ratio requirements.

[0256] Taking pruning as an example, in one possible implementation, the cloud-side device can store an initial neural network model based on a transformer structure (the teacher model, which can be a pre-stored model with high data processing accuracy or a user-specified model). After receiving performance requirement information from the terminal device, the cloud-side device can determine the size information of a suitable model (student model) based on the received performance requirement information. Specifically, when the accuracy requirement included in the performance requirement information is high, the size of the student model can be determined to be larger; when the latency requirement included in the performance requirement information is high, the size of the suitable student model can be determined to be smaller; and when the model compression ratio included in the performance requirement information is high, the size of the suitable student model can be determined to be larger. Specifically, the cloud-side device can determine the size information of the student model based on a preset functional relationship or a preset correspondence (e.g., through a table lookup).

[0257] Reference Figure 12 , Figure 12 This is a schematic diagram of the structure of a neural network model in an embodiment of this application, such as... Figure 12 As illustrated, a transformer-based neural network model can include sequentially connected embedding layers and multiple transformer layers. As those skilled in the art will understand, transformer models are commonly used for performing natural language processing (NLP) tasks. It is important to understand that... Figure 12 The structure shown is merely an example; the number of transformer layers can be set as needed. The neural network model determines the feature vector corresponding to the current node based on the N output vectors obtained from each transformer layer.

[0258] In the embedding layer, the current input is embedded to obtain multiple feature vectors. The core feature of the Transformer model lies in its unique attention mechanism. When processing natural language, such as a sentence, the Transformer model uses this attention mechanism to assign different attention coefficients to the word vectors in the sentence, thus more comprehensively considering the influence of the context on each word. The embedding layer obtains N embedding vectors X based on the node features and positional encoding of each node in the current sequence. l The attention layer is connected to the embedding layer. It takes N embedding vectors from the embedding layer as input vectors, and combines them based on the correlation between the N input vectors to obtain N output vectors, which are then fed to the subsequent transformer layer. The transformer layer takes the output of the previous layer as its input vector and performs similar operations as the previous transformer layer.

[0259] Reference Figure 13 , Figure 13 This is a schematic diagram of a transformer layer structure. The transformer layers of various neural networks in the embodiments of this application can be referenced from this diagram. Figure 13 The structure shown in the figure, such as Figure 13 As shown, the transformer layer consists of a multi-head attention layer, an add & normalization layer, a feed forward layer, and another add & normalization layer, which are sequentially adjacent to each other.

[0260] The multi-head attention layer obtains N input vectors X from the layer above it. l This can also be represented as matrix X. Employing a self-attention mechanism, it transforms each vector based on the correlation between them, resulting in N output vectors, which can also be represented as matrix Y. It can be understood that when this multi-head attention layer is directly connected to the embedding layer, for example... Figure 13 In a transformer layer directly connected to the embedding layer, the input vector it receives is the embedding vector output by the embedding layer; when this multi-head attention layer is a multi-head attention layer included in subsequent transformer layers, for example... Figure 13 The transformer layer directly connected to the previous transformer layer includes a multi-head attention layer, whose input vector is the output vector of the previous transformer layer. In the multi-head attention layer, the MHA layer includes multiple attention heads (e.g., ...). Figure 14 The following are Head 1, Head 2, ..., Head N shown in the figure.

[0261] Figure 14 This is a schematic diagram illustrating the operation of an attention head, showing how the attention head transforms an input matrix X into an output matrix Y. For example... Figure 14As shown, the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V are applied to N input vectors respectively.<X1,X2,…,XN> The input vectors Xi are transformed to obtain the first intermediate vector (q vector), second intermediate vector (k vector), and third intermediate vector (v vector) corresponding to each input vector. Operationally, the input matrix X, composed of N input vectors, can be linearly transformed using the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V, respectively, to obtain the Q matrix, K matrix, and V matrix of the input matrix. These matrices are then split to obtain the q vector, k vector, and v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, the correlation degree between the i-th input vector Xi and each input vector Xj is determined based on the dot product operation between the first intermediate vector (q vector, qi) corresponding to the i-th input vector and each second intermediate vector (k vector, kj) corresponding to each input vector Xj. Although the dot product result of qi and kj can be directly used to determine the correlation degree, a more classic approach is to first divide the dot product result by a constant, then perform a softmax operation, and use the result as the correlation degree between the input vector Xi and Xj, i.e.:

[0262]

[0263] Therefore, the correlation degrees αi,j between the i-th input vector Xi and each input vector Xj can be used as weighting factors to perform a weighted combination of the third intermediate vectors (v vector, vj) corresponding to each input vector Xj, resulting in the i-th combined vector Ci corresponding to the i-th input vector Xi:

[0264]

[0265] Therefore, we can obtain a vector sequence of N combined vectors corresponding to N input vectors.<C1,C2,…,CN> Or matrix C. Based on this combined vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combined vectors can be directly used as N output vectors, i.e., Yi = Ci. In this case, the output matrix Y is the combined vector matrix C, which can also be written as:

[0266]

[0267] The above describes the processing flow of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K, and third transformation matrix V, allowing the above operations to be performed in parallel to obtain m combined vector sequences (i.e., m matrices C). Each vector sequence includes N combined vectors obtained based on a set of transformation matrices. In this case, the MHA layer concatenates the m combined vector sequences to obtain a concatenated matrix; then, it transforms this concatenated matrix using the fourth transformation matrix W to obtain the final output matrix Y. This output matrix Y can be split into N output vectors.<Y1,Y2,…,YN> Through the above operations, the MHA layer performs transformation operations based on the correlation between the N input vectors to obtain N output vectors.

[0268] like Figure 13 As shown, the transformer layer includes a feedforward layer, which comprises an input layer, an intermediate layer, and an output layer, wherein the intermediate layer comprises multiple neurons.

[0269] As mentioned earlier, a neural network model can contain multiple transformer layers. In one embodiment, these multiple transformer layers can be stacked and connected in a residual network manner to form a neural network model.

[0270] In the case of multiple transformer layers, in one embodiment, the neural network model can combine the N output vectors obtained from each of the multiple transformer layers to obtain the feature vector corresponding to the current node. In another embodiment, the neural network model can also extract only the N output vectors obtained from the last transformer layer, combine these N output vectors to obtain the feature vector of the current node.

[0271] In this embodiment of the application, the cloud-side device can store the aforementioned neural network model (teacher model) based on the transformer structure. After receiving the performance requirement information sent by the terminal device, it can determine the size information of the student model based on the received performance requirement information. The size information can include the width and depth of the student model. Specifically, the width information can include the number of attention heads included in each transformer layer in the neural network model and the number of neurons included in the intermediate layer in the feedforward layer. The depth information can include the number of transformer layers included in the neural network model.

[0272] In this embodiment, the computation in the multi-head attention mechanism can be broken down into computations performed by each attention head and then summed. Therefore, the scaling of the MHA layer can be adjusted based on the number of attention heads. The intermediate layer of a fully connected network (feedforward layer) can also be scaled by changing the number of neurons it contains. For a transformer layer, the width can be scaled between the attention heads in the MHA and the neurons in the intermediate layer of the feedforward layer. For example, the BERT base model has 12 attention heads, so there are 12 possible width scaling options, meaning the width can be any one of 1, 2, ..., 12. Similarly, the intermediate layer of the feedforward layer can also retain any number of neurons.

[0273] For example, refer to Figure 15 , Figure 15 This application provides an illustration of a model width scaling method, such as... Figure 15 As shown in the figure, before scaling, the first neural network model includes a transformer layer with an intermediate layer containing M neurons and an MHA layer containing N attention heads. After scaling the width of the first neural network model, a second neural network model is obtained, wherein the second neural network model includes a transformer layer corresponding to the transformer layer included in the first neural network model. The intermediate layer of the transformer layer includes M / 3 neurons and the MHA layer includes N / 3 attention heads.

[0274] For example, refer to Figure 16 , Figure 16 This is a schematic diagram of model depth scaling provided in an embodiment of this application. Before compression, the teacher model includes M transformer layers, and after compression, the number of transformer layers in the student model is reduced to M / 3.

[0275] In this embodiment of the application, the student model may include a first sub-model and a third sub-model, wherein the output of the third sub-model is used as the input of the first sub-model. That is, the first sub-model and the third sub-model are part of the student model, and the third sub-model is the model in the student model that is connected before the first sub-model.

[0276] In this embodiment of the application, the teacher model may include a second sub-model and a fourth sub-model, wherein the output of the fourth sub-model is used as the input of the second sub-model. That is, the second sub-model and the fourth sub-model are part of the teacher model, and the fourth sub-model is the model in the teacher model that is connected before the second sub-model.

[0277] In one possible implementation, the student and teacher models can be transformer models, and the first and second sub-models can include one or more transformer layers. Similarly, the third and fourth sub-models can include one or more transformer layers. Specifically, the transformer model can be divided into multiple modules (the number of which can be determined based on the number of nodes in the computing node cluster or the training speed requirements), where each module contains one or more transformer layers, and each computing node in the computing node cluster can be responsible for compressing one module.

[0278] Firstly, during distillation, all transformer layers within each module can be minimized. Compared to previous locally supervised learning signals, this partitioning allows for greater granularity and considers more internal layer-by-layer dependencies. Secondly, module-based partitioning allows for the design of a new model parallelism strategy to further accelerate training. By assigning each module to different computational nodes, all modules can be trained in parallel, significantly reducing training time compared to previous sequential training.

[0279] Among them, the modules of the transformer model (student model) can be deployed according to the connection order in the transformer model, based on the order of the serial connection of computing nodes;

[0280] For example, a transformer model (a student model, which is obtained by quantizing and pruning the width of the teacher model, i.e., the number of transformer layers remains unchanged, where the width pruning can be pruning the number of attention heads, the number of neurons in the intermediate layers, etc.) can include transformer layers A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, and A12. The compute node cluster can include compute node 1, compute node 2, compute node 3, etc. The transformer model can be divided into three modules: module A1 (including transformer layers A1, A2, A3, and A4), module A2 (including transformer layers A5, A6, A7, and A8), and module A3 (including transformer layers A9, A10, A11, and A12). Computation node 1 can be responsible for knowledge distillation of module A1, computation node 2 can be responsible for knowledge distillation of module A2, and computation node 3 can be responsible for knowledge distillation of module A3.

[0281] Similarly, the transformer model (teacher model) can include transformer layers B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, and B12. The computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules, namely module 1 (…). The system includes transformer layers B1, B2, B3, and B4; module 2 (including transformer layers B5, B6, B7, and B8); and module 3 (including transformer layers B9, B10, B11, and B12). Computing node 1 can be responsible for knowledge distillation of module A1 (based on modules A1 and B1), computing node 2 can be responsible for knowledge distillation of module A2 (based on modules A2 and B2), and computing node 3 can be responsible for knowledge distillation of module A3 (based on modules A3 and B3).

[0282] For example, a transformer model (a student model, which is obtained by deep pruning of the teacher model, i.e., reducing the number of transformer layers) can include transformer layers A1, A2, A3, A4, A5, and A6. A computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules: module A1 (including transformer layers A1 and A2), module A2 (including transformer layers A3 and A4), and module A3 (including transformer layers A5 and A6). Computing node 1 can be responsible for knowledge distillation of module A1, computing node 2 can be responsible for knowledge distillation of module A2, and computing node 3 can be responsible for knowledge distillation of module A3.

[0283] Similarly, the transformer model (teacher model) can include transformer layers B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, and B12. The computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules, namely module 1 (…). The system includes transformer layers B1, B2, B3, and B4; module 2 (including transformer layers B5, B6, B7, and B8); and module 3 (including transformer layers B9, B10, B11, and B12). Computing node 1 can be responsible for knowledge distillation of module A1 (based on modules A1 and B1), computing node 2 can be responsible for knowledge distillation of module A2 (based on modules A2 and B2), and computing node 3 can be responsible for knowledge distillation of module A3 (based on modules A3 and B3).

[0284] When performing knowledge distillation for the student model based on the teacher model, each iteration of training can include a feedforward process and a gradient backpropagation process. The feedforward process is the process by which the student model and the teacher model process the training samples. Since the teacher model and the student model in each embodiment of this application are divided into multiple modules and are each handled by different computing nodes, each computing node can obtain the calculation results of the computing nodes connected to the previous level (the first computing node can obtain the training samples), and perform calculations on the calculation results of the computing nodes connected to the previous level based on its own responsible module, and pass the calculation results to the computing nodes connected to the next level. Since the serial connection order of the computing nodes is consistent with the network layer arrangement of the model, from the perspective of multiple nodes as a whole, the feedforward process is still equivalent to the process of processing training samples through the complete student model and teacher model.

[0285] In this embodiment, each computing node is used to perform feedforward (or forward propagation) based on its own part of the network to obtain output data. Each computing node can flow the output data in a serial order. Taking the first computing node and the second computing node as an example, during the feedforward process, the second computing node can process the data through the third sub-model to obtain the first input data. The first input data can be used as the input of the first sub-model, and thus the first computing node can obtain the first input data. Similarly, the second computing node can process the data through the fourth sub-model to obtain the second input data. The second input data can be used as the input of the second sub-model, and thus the first computing node can obtain the second input data.

[0286] The following describes how the first input data is obtained:

[0287] In practical applications, the time taken for each computing node to execute the feedforward process of its respective network layer is not uniform. In order to further reduce the waiting time between different computing nodes, during forward propagation, the next computing node does not wait for the previous computing node to finish its calculation before starting its calculation.

[0288] In one possible implementation, an output queue I can be deployed between adjacent computing nodes (e.g., the nth computing node and the (n+1)th computing node) to collect the output of the nth computing node in the most recent t0 steps. The input queue has a pre-set memory size and is updated according to a first-input first-output (FIFO) principle. The (n+1)th computation node can always draw from the queue. Get from This allows the system to use its own input without waiting. This design avoids the load imbalance problem from lagging modules, achieving a near-theoretical linear speedup of N times with N training devices.

[0289] Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, a first queue (for the student model) can be deployed between the second computing node and the first computing node. The second computing node can store the output data (including the first input data) into the first queue. Then, the first computing node can obtain the first input data from the first queue. The first queue is used to store at least one piece of first data from the second computing node. Each piece of first data is the output obtained by the second computing node through the processing of the input data by the third sub-model.

[0290] In one possible implementation, the first computing node is also communicatively connected to a third computing node, and a third queue can be deployed (e.g., a storage location between the first and third computing nodes, or on the first computing node, or on the third computing node). The student model also includes a fifth sub-model connected after the first sub-model. After the first computing node processes the first input data through the first sub-model, the processed first intermediate output can be stored in the third queue (the entity performing the storage action is not limited to the first computing node). That is, the first intermediate output can be passed to the third queue, which is used to store the first intermediate output so that the third computing node can obtain the first intermediate output from the third queue. The first intermediate output is used as input data for the feedforward process when performing model distillation on the third sub-model.

[0291] The timing of obtaining the first input data is described below:

[0292] In existing knowledge distillation gradient backpropagation processes, updates begin from the output layer and gradually move towards the input layer. The update of the preceding network layer depends on the completion of the subsequent layer's update. Therefore, when the network layers of subsequent or multiple computational nodes have not yet been updated, many computational nodes are in a resource-idle state. For example, computational node 1 is responsible for distilling network layer 1, and computational node 2 is responsible for distilling network layer 2. Network layer 2 is connected after network layer 1. During the feedforward process, the output of network layer 1 serves as the input of network layer 2. During gradient backpropagation, network layer 2 is updated before network layer 1 is updated. In other words, computational node 1 cannot update network layer 1 until computational node 2 has completed its update. The inventive concept of this application is that the gradient backpropagation process of each computational node is performed internally, independent of the subsequent (or subsequent N levels, where N is greater than 1) computational nodes completing the distillation of their assigned network layers. Furthermore, the gradients calculated by each computational node are not backpropagated to the preceding computational node. This asynchronous parallel training achieves greater utilization of computational resources, thereby accelerating the distillation process.

[0293] Since the distillation process between computing nodes is independent, after completing the update process with one gradient backpropagation, the computing node can directly obtain the next input data and perform feedforward without waiting. Furthermore, since the next input data is already stored in the queue deployed with the previous level computing node, there is no need to wait for the feedforward process of the previous level computing node to complete, which further improves the utilization of computing resources of the computing node and accelerates the distillation process.

[0294] Taking the first computing node as an example, in the previous iteration, the first computing node can distill the network layers of the student network it is responsible for to obtain the first sub-model, and directly obtain the first input data from the first queue in response to obtaining the first sub-model. After completing the distillation of the first sub-model, it can obtain the third input data (i.e., the next input data) from the first queue based on the obtained updated first sub-model, and then perform a feedforward process for model distillation of the updated first sub-model based on the third input data.

[0295] Similarly, for the teacher model, an output queue can also be deployed. Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, a second queue can be deployed (for the teacher model, the deployment location can be a storage location between the first and second computing nodes, or deployed on the first computing node, or deployed on the second computing node). The second computing node can store the output data (including the second input data) into the second queue. Then, the first computing node can obtain the second input data from the second queue. The second queue is used to store at least one second data from the second computing node, and each second data is the output obtained by the second computing node through the data processing of the fourth sub-model.

[0296] 1102. Process the first input data through the first sub-model to obtain the first intermediate output.

[0297] 1103. The second input data is processed through the second sub-model to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient.

[0298] Steps 1102 and 1103 describe the feedforward process performed by the first computing node based on the input data. For the student model it is responsible for (first sub-model), the first input data can be processed through the first sub-model to obtain the first intermediate output. For the teacher model it is responsible for (second sub-model), the second input data can be processed through the second sub-model to obtain the second intermediate output.

[0299] In one possible implementation, the first sub-model and the second sub-model may include multiple network layers (e.g., multiple transformer layers), and the first intermediate output may be the output of the last layer in the first sub-model (i.e., the network layer closest to the output layer of the student model), and correspondingly, the second intermediate output may be the output of the last layer in the second sub-model (i.e., the network layer closest to the output layer of the teacher model).

[0300] For the nth computation node, assume that the learnable parameters in the small model it is responsible for (a part of the student model it is responsible for) are w. n Its training objective function is to minimize the distance L between the features of the compressed small model and the features of the uncompressed large model (a portion of the teacher's model). (n) For the nth computation node, a common approach to constructing the loss function is to use the final output of that computation node as a characteristic. Assuming... and Let be the outputs of the nth computational node of the model before and after compression, respectively. Then, the training objective function of the nth computational node can be expressed as:

[0301]

[0302] In one possible implementation, the first intermediate output can also be the output of each layer in the first sub-model, and correspondingly, the second intermediate output can be the output of each layer in the second sub-model. For compression methods that do not change the number of network layers (e.g., transformer layers) (e.g., model quantization, width-direction pruning), a common approach is to construct the loss function as the output of each network layer (e.g., transformer layer) in each computation node. Assume f l and Let be the outputs of the l-th transformer layer before and after compression, respectively. Then, the objective function of the n-th computation node can be expressed as:

[0303]

[0304] In this embodiment of the application, after obtaining the first intermediate output and the second intermediate output, a loss can be constructed based on the first intermediate output and the second intermediate output, and a first gradient can be determined based on the loss. The first gradient can be used to update the first sub-model.

[0305] It should be understood that in one possible implementation, the first sub-model does not include the output layer of the student model, and correspondingly, the second sub-model does not include the output layer of the teacher model. When the first sub-model includes the output layer of the student model and the second sub-model includes the output layer of the teacher model, the first computation node itself does not need to wait for gradient backpropagation from the next-level computation node (it is itself the last-level computation node).

[0306] 1104. Distill the first sub-model according to the first gradient to obtain the updated first sub-model.

[0307] In the existing gradient backpropagation process of knowledge distillation, updates are performed step by step from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the network layer of the next or multiple computing nodes has not yet been updated, many computing nodes are in a resource idle state. For example, computing node 1 is responsible for the distillation of network layer 1, and computing node 2 is responsible for the distillation of network layer 2. Network layer 2 is connected after network layer 1. In the feedforward process, the output of network layer 1 is used as the input of network layer 2. During gradient backpropagation, network layer 1 is updated only after network layer 2 has been updated. That is, computing node 1 cannot update network layer 1 before computing node 2 has completed the update of network layer 2. In this embodiment, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0308] It should be understood that in the asynchronous parallel training process described above, the error generated by the student model propagates layer by layer (propagating during feedforward). During parallel training, when a previous segment is not completed, subsequent segments will be affected by the previous ones. On the other hand, although queues can prevent computing nodes from waiting for each other, non-sequential computation introduces a larger error to the model compared to ordinary pipelined parallel training. Therefore, in one implementation, a teacher model can be used to mitigate the accumulation of error caused by the student model. Specifically, at the beginning of distillation training, the input data obtained by the computing node can be the output of the teacher model of the connected computing node at the previous level (at the beginning of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error propagates layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model as input at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model). As training progresses, the input is gradually changed to use the samples from the output queue of the student model to maintain consistency with the inference stage (using only the student model). For details, please refer to... Figure 18 The descriptions in the corresponding embodiments are not repeated here.

[0309] In the specific implementation, the queue corresponding to the teacher model can be used. In the queue corresponding to the student model The convex combination between them serves as the input to the (n+1)th computation node, and the control coefficient λ changes from 1 to 0 during training.

[0310]

[0311] There are many decay functions that can make λ change from 1 to 0 during training. Assuming the total number of training iterations is T0, a commonly used linear decay function can be expressed as:

[0312] λ t =max(1-t / T0, 0);

[0313] Reference Figure 17 , Figure 17 As a system illustration, to accelerate compressed training, the various modules of the model can be placed on N computing devices. Given a training set D, during the forward propagation of the small network, each block independently calculates its own module loss L. n Let n = 1, ..., N. In backpropagation, the gradient of each module is restricted to propagate locally within that module and not to previous modules. The number of modules N can be adjusted based on memory constraints. Furthermore, it is best to partition the model evenly to ensure balanced memory usage across different modules.

[0314] Using the student network as the quantization source, the distillation method proposed in this application, compared to ordinary non-parallel quantization training, significantly improves training speed (by more than 100 times) while consuming only 1 / 3 of the GPU memory, relying on 4096 training data points, and the final compressed model accuracy only slightly decreases. The results are shown in the table below:

[0315]

[0316] Furthermore, this application embodiment also provides a computing node cluster, including multiple computing nodes connected in series, wherein the multiple computing nodes are used to perform knowledge distillation for a student network, and each computing node is responsible for knowledge distillation of a portion of the student network; wherein each computing node is used to perform feedforward based on the portion of the network it is responsible for to obtain output data;

[0317] Each computing node is also used to determine gradients based on the data obtained by feedforwarding the part of the network it is responsible for, and to perform gradient backpropagation within each computing node to distill the part of the network it is responsible for, and the gradients determined by each computing node are not propagated to the connected computing nodes.

[0318] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0319] In one possible implementation, the output data can be passed to a first-in-first-out queue between connected computing nodes so that the connected computing nodes can retrieve the output data from the first-in-first-out queue.

[0320] This application provides a model distillation method applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, which further includes a third sub-model connected to the first sub-model. The second sub-model is a partial model of a teacher model, which further includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, which deploys the third and fourth sub-models. The method includes: acquiring first input data and second input data from the second computing node, where the first input data is output data of the third sub-model and the second input data is output data processed by the fourth sub-model; processing the first input data through the first sub-model to obtain a first intermediate output; processing the second input data through the second sub-model to obtain a second intermediate output, wherein the first and second intermediate outputs are used to determine a first gradient; and distilling the first sub-model according to the first gradient to obtain an updated first sub-model. In this embodiment, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0321] Reference Figure 18 , Figure 18 This is an illustration of a model distillation method provided in an embodiment of this application. The method can be applied to a first computing node, which is communicatively connected to a second computing node. The first computing node can be a terminal device such as a mobile phone, tablet, laptop, or smart wearable device. The first computing node can also be a server, chip, or other device with data processing capabilities. Figure 18 As shown, an embodiment of this application provides a model distillation method comprising:

[0322] 1801. Obtain first input data from the second computing node, wherein the first input data is the output data of the third sub-model.

[0323] The first computing node may be deployed with a first sub-model and a second sub-model. The first sub-model is a part of the student model, and the second sub-model is a part of the teacher model. The teacher model also includes a third sub-model connected to the second sub-model. The first computing node is communicatively connected to the second computing node, and the second computing node may be deployed with the third sub-model.

[0324] In this embodiment, the computing node cluster can perform knowledge distillation of the student model based on the teacher model. Each computing node can be responsible for distilling a portion of the student model, and the computing nodes can coordinate with each other to achieve the distillation of the student model. The computing node cluster can include multiple serially connected computing nodes, which may include a first computing node and a second computing node, and the first and second computing nodes are communicatively connected.

[0325] The student model can be obtained by compressing the teacher model, for example, by pruning the teacher model, or by quantizing the teacher model.

[0326] In this embodiment of the application, the teacher model may include a second sub-model and a third sub-model, wherein the output of the third sub-model is used as the input of the second sub-model. That is, the second sub-model and the third sub-model are part of the teacher model, and the third sub-model is the model in the teacher model that is connected before the second sub-model.

[0327] It should be understood that Figure 11 The fourth sub-model in the corresponding embodiment can be the third sub-model in the embodiment of this application.

[0328] In one possible implementation, the student and teacher models can be transformer models, and the first and second sub-models can each include one or more transformer layers. Similarly, the third sub-model can include one or more transformer layers. Specifically, the transformer model can be divided into multiple modules (the number of which can be determined based on the number of nodes in the computing node cluster or the training speed requirements), where each module contains one or more transformer layers, and each computing node in the computing node cluster can be responsible for compressing one module.

[0329] Firstly, during distillation, all transformer layers within each module can be minimized. Compared to previous locally supervised learning signals, this partitioning allows for greater granularity and considers more internal layer-by-layer dependencies. Secondly, module-based partitioning allows for the design of a new model parallelism strategy to further accelerate training. By assigning each module to different computational nodes, all modules can be trained in parallel, significantly reducing training time compared to previous sequential training.

[0330] Among them, the modules of the transformer model (student model) can be deployed according to the connection order in the transformer model, based on the order of the serial connection of computing nodes;

[0331] For example, a transformer model (a student model, which is obtained by quantizing and pruning the width of the teacher model, i.e., the number of transformer layers remains unchanged, where the width pruning can be pruning the number of attention heads, the number of neurons in the intermediate layers, etc.) can include transformer layers A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, and A12. The compute node cluster can include compute node 1, compute node 2, compute node 3, etc. The transformer model can be divided into three modules: module A1 (including transformer layers A1, A2, A3, and A4), module A2 (including transformer layers A5, A6, A7, and A8), and module A3 (including transformer layers A9, A10, A11, and A12). Computation node 1 can be responsible for knowledge distillation of module A1, computation node 2 can be responsible for knowledge distillation of module A2, and computation node 3 can be responsible for knowledge distillation of module A3.

[0332] Similarly, the transformer model (teacher model) can include transformer layers B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, and B12. The computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules, namely module 1 (…). The system includes transformer layers B1, B2, B3, and B4; module 2 (including transformer layers B5, B6, B7, and B8); and module 3 (including transformer layers B9, B10, B11, and B12). Computing node 1 can be responsible for knowledge distillation of module A1 (based on modules A1 and B1), computing node 2 can be responsible for knowledge distillation of module A2 (based on modules A2 and B2), and computing node 3 can be responsible for knowledge distillation of module A3 (based on modules A3 and B3).

[0333] For example, a transformer model (a student model, which is obtained by deep pruning of the teacher model, i.e., reducing the number of transformer layers) can include transformer layers A1, A2, A3, A4, A5, and A6. A computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules: module A1 (including transformer layers A1 and A2), module A2 (including transformer layers A3 and A4), and module A3 (including transformer layers A5 and A6). Computing node 1 can be responsible for knowledge distillation of module A1, computing node 2 can be responsible for knowledge distillation of module A2, and computing node 3 can be responsible for knowledge distillation of module A3.

[0334] Similarly, the transformer model (teacher model) can include transformer layers B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, and B12. The computing node cluster can include computing node 1, computing node 2, and computing node 3. Computing node 1 and computing node 2 are connected, and computing node 2 and computing node 3 are connected. The transformer model can be divided into three modules, namely module 1 (…). The system includes transformer layers B1, B2, B3, and B4; module 2 (including transformer layers B5, B6, B7, and B8); and module 3 (including transformer layers B9, B10, B11, and B12). Computing node 1 can be responsible for knowledge distillation of module A1 (based on modules A1 and B1), computing node 2 can be responsible for knowledge distillation of module A2 (based on modules A2 and B2), and computing node 3 can be responsible for knowledge distillation of module A3 (based on modules A3 and B3).

[0335] When performing knowledge distillation for the student model based on the teacher model, each iteration of training may include a feedforward process and a gradient backpropagation process. The feedforward process is the process by which the student model and the teacher model process training samples. Since the teacher model and the student model in each embodiment of this application are divided into multiple modules and are each handled by different computing nodes, each computing node can obtain the calculation results of the computing node connected to the previous level (the first computing node can obtain the training samples), and perform calculations on the calculation results of the computing node connected to the previous level based on the module it is responsible for, and pass the calculation results to the computing node connected to the next level.

[0336] In the existing gradient backpropagation process of knowledge distillation, updates are performed progressively from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the network layers of the next or multiple computational nodes have not yet been updated, many computational nodes are in a resource-idle state. For example, computational node 1 is responsible for the distillation of network layer 1, and computational node 2 is responsible for the distillation of network layer 2. Network layer 2 is connected after network layer 1. During the feedforward process, the output of network layer 1 serves as the input of network layer 2. During gradient backpropagation, network layer 1 is only updated after network layer 2 has been updated. In other words, computational node 1 cannot update network layer 1 until computational node 2 has completed its update. Figure 11 In the corresponding embodiment, the gradient backpropagation process of each computing node is carried out internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and can be avoided by being backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0337] However, in the asynchronous parallel training process described above, the error generated by the student model propagates layer by layer (propagating during feedforward). During parallel training, when a previous segment is not completed, subsequent segments will be affected by the previous ones. On the other hand, although queuing eliminates the need for computation nodes to wait for each other, non-sequential computation introduces a larger error to the model compared to ordinary pipelined parallel training. Therefore, in one implementation, a teacher model can be used to mitigate the accumulation of error caused by the student model. Specifically, at the beginning of distillation training, the input data obtained by the computation node can be the output of the teacher model of the connected computation node at the previous level (at the beginning of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error propagates layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model (the output of the third sub-model) as the input (as the input of the first sub-model) at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model). As training progresses, the input is gradually changed to samples from the student model's output queue to maintain consistency with the inference phase (using only the student model).

[0338] In this embodiment, each computing node is used to perform feedforward (or forward propagation) based on the part of the network it is responsible for, and obtain output data. Each computing node can flow the output data in a serial order. Taking the first computing node and the second computing node as an example, in the feedforward process, the second computing node can process the data through the third sub-model (teacher model) to obtain the first input data. The first input data can be used as the input of the first sub-model, and thus the first computing node can obtain the first input data. Similarly, the second computing node can process the data through the fourth sub-model to obtain the second input data. The second input data can be used as the input of the second sub-model, and thus the first computing node can obtain the second input data.

[0339] The following describes how the first input data is obtained:

[0340] In practical applications, the time taken for each computing node to execute the feedforward process of its respective network layer is not uniform. In order to further reduce the waiting time between different computing nodes, during forward propagation, the next computing node does not wait for the previous computing node to finish its calculation before starting its calculation.

[0341] In one possible implementation, an output queue I can be deployed between adjacent computing nodes (e.g., the nth computing node and the (n+1)th computing node) to collect the output of the nth computing node in the most recent t0 steps. The input queue has a pre-set memory size and is updated according to a first-input first-output (FIFO) principle. The (n+1)th computation node can always draw from the queue. Get from This allows the system to use its own input without waiting. This design avoids the load imbalance problem from lagging modules, achieving a near-theoretical linear speedup of N times with N training devices.

[0342] Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, a first queue (for the teacher model) can be deployed between the second computing node and the first computing node. The second computing node can store the output data (including the first input data) into the first queue. Then, the first computing node can obtain the first input data from the first queue. The first queue is used to store at least one piece of first data from the second computing node. Each piece of first data is the output obtained by the second computing node through the processing of data by the third sub-model.

[0343] The timing of obtaining the first input data is described below:

[0344] In existing knowledge distillation gradient backpropagation processes, updates begin from the output layer and gradually move towards the input layer. The update of the preceding network layer depends on the completion of the subsequent layer's update. Therefore, when the network layers of subsequent or multiple computational nodes have not yet been updated, many computational nodes are in a resource-idle state. For example, computational node 1 is responsible for distilling network layer 1, and computational node 2 is responsible for distilling network layer 2. Network layer 2 is connected after network layer 1. During the feedforward process, the output of network layer 1 serves as the input of network layer 2. During gradient backpropagation, network layer 2 is updated before network layer 1 is updated. In other words, computational node 1 cannot update network layer 1 until computational node 2 has completed its update. The inventive concept of this application is that the gradient backpropagation process of each computational node is performed internally, independent of the subsequent (or subsequent N levels, where N is greater than 1) computational nodes completing the distillation of their assigned network layers. Furthermore, the gradients calculated by each computational node are not backpropagated to the preceding computational node. This asynchronous parallel training achieves greater utilization of computational resources, thereby accelerating the distillation process.

[0345] Since the distillation process between computing nodes is independent, after completing the update process with one gradient backpropagation, the computing node can directly obtain the next input data and perform feedforward without waiting. Furthermore, since the next input data is already stored in the queue deployed with the previous level computing node, there is no need to wait for the feedforward process of the previous level computing node to complete, which further improves the utilization of computing resources of the computing node and accelerates the distillation process.

[0346] Taking the first computing node as an example, in the previous iteration, the first computing node can distill the network layer of the student network it is responsible for to obtain the first sub-model, and directly obtain the first input data from the first queue based on the obtained first sub-model. After completing the distillation of the first sub-model, it can obtain the second input data (i.e., the next input data) from the first queue based on the obtained updated first sub-model, and then perform a feedforward process for model distillation of the updated first sub-model based on the second input data. Alternatively, a second queue (for the student model) can be deployed between the second computing node and the first computing node. After completing the distillation of the first sub-model, it can obtain the fifth input data (i.e., the next input data) from the first queue based on the obtained updated first sub-model, and then perform a feedforward process for model distillation of the updated first sub-model based on the fifth input data.

[0347] Similarly, for the teacher model, queues can be deployed between the various computing nodes. Taking the nth computing node as the second computing node and the (n+1)th computing node as the first computing node as an example, the first computing node can obtain the first input data from the first queue. The first queue is used to store at least one second data from the second computing node, and each second data is the output obtained by the second computing node through the data processing of the third sub-model.

[0348] 1802. Process the first input data through the first sub-model to obtain the first intermediate output;

[0349] 1803. The first input data is processed through the second sub-model to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient;

[0350] Steps 1802 and 1803 describe the feedforward process performed by the first computing node based on the input data. For the student model it is responsible for (first sub-model), the first input data can be processed through the first sub-model to obtain the first intermediate output. For the teacher model it is responsible for (second sub-model), the first input data can be processed through the second sub-model to obtain the second intermediate output.

[0351] In one possible implementation, the first sub-model and the second sub-model may include multiple network layers (e.g., multiple transformer layers), and the first intermediate output may be the output of the last layer in the first sub-model (i.e., the network layer closest to the output layer of the student model), and correspondingly, the second intermediate output may be the output of the last layer in the second sub-model (i.e., the network layer closest to the output layer of the teacher model).

[0352] For the nth computation node, assume that the learnable parameters in the small model it is responsible for (a part of the student model it is responsible for) are w. n Its training objective function is to minimize the distance L between the features of the compressed small model and the features of the uncompressed large model (a portion of the teacher's model). (n) For the nth computation node, a common approach to constructing the loss function is to use the final output of that computation node as a characteristic. Assuming... and Let be the outputs of the nth computational node of the model before and after compression, respectively. Then, the training objective function of the nth computational node can be expressed as:

[0353]

[0354] In one possible implementation, the first intermediate output can also be the output of each layer in the first sub-model, and correspondingly, the second intermediate output can be the output of each layer in the second sub-model. For compression methods that do not change the number of network layers (e.g., transformer layers) (e.g., model quantization, width-direction pruning), a common approach is to construct the loss function as the output of each network layer (e.g., transformer layer) in each computation node. Assume f l and Let be the outputs of the l-th transformer layer before and after compression, respectively. Then, the objective function of the n-th computation node can be expressed as:

[0355]

[0356] In this embodiment of the application, after obtaining the first intermediate output and the second intermediate output, a loss can be constructed based on the first intermediate output and the second intermediate output, and a first gradient can be determined based on the loss. The first gradient can be used to update the first sub-model.

[0357] It should be understood that in one possible implementation, the first sub-model does not include the output layer of the student model, and correspondingly, the second sub-model does not include the output layer of the teacher model. When the first sub-model includes the output layer of the student model and the second sub-model includes the output layer of the teacher model, the first computation node itself does not need to wait for gradient backpropagation from the next-level computation node (it is itself the last-level computation node).

[0358] 1804. Distill the first sub-model according to the first gradient to obtain the updated first sub-model.

[0359] In the existing gradient backpropagation process of knowledge distillation, updates are performed step by step from the output layer to the input layer. The update of the previous layer depends on the completion of the update of the next layer. Therefore, when the network layer of the next or multiple computing nodes has not yet been updated, many computing nodes are in a resource idle state. For example, computing node 1 is responsible for the distillation of network layer 1, and computing node 2 is responsible for the distillation of network layer 2. Network layer 2 is connected after network layer 1. In the feedforward process, the output of network layer 1 is used as the input of network layer 2. During gradient backpropagation, network layer 1 is updated only after network layer 2 has been updated. That is, computing node 1 cannot update network layer 1 before computing node 2 has completed the update of network layer 2. In this embodiment, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0360] It should be understood that in the asynchronous parallel training process described above, the error generated by the student model propagates layer by layer (propagating during feedforward). During parallel training, when a previous segment is not completed, subsequent segments will be affected by the previous ones. On the other hand, although queuing eliminates the need for computation nodes to wait for each other, non-sequential computation introduces a larger error to the model compared to ordinary pipelined parallel training. Therefore, in one implementation, a teacher model can be used to mitigate the accumulation of error caused by the student model. Specifically, at the beginning of distillation training, the input data obtained by the computation node can be the output of the teacher model of the connected computation node at the previous level (at the beginning of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error propagates layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model as input at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model). As training progresses, the input is gradually changed to samples from the output queue of the student model to maintain consistency with the inference phase (using only the student model).

[0361] In the specific implementation, the queue corresponding to the teacher model can be used. In the queue corresponding to the student model The convex combination between them serves as the input to the (n+1)th computation node, and the control coefficient λ changes from 1 to 0 during training.

[0362]

[0363] There are many decay functions that can make λ change from 1 to 0 during training. Assuming the total number of training iterations is T0, a commonly used linear decay function can be expressed as:

[0364] λ t =max(1-t / T0, 0);

[0365] This application provides a model distillation method applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model, wherein the first sub-model is a partial model of a student model, the second sub-model is a partial model of a teacher model, and the teacher model further includes a third sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, and the second computing node deploys the third sub-model. The method includes: acquiring first input data from the second computing node, the first input data being output data of the third sub-model; processing the first input data through the first sub-model to obtain a first intermediate output; processing the first input data through the second sub-model to obtain a second intermediate output, the first intermediate output and the second intermediate output being used to determine a first gradient; and distilling the first sub-model according to the first gradient to obtain an updated first sub-model. At the start of distillation training, the input data obtained by the computing node can be the output of the teacher model of the computing node connected to the previous level (at the start of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error will propagate layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model as the input at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model).

[0366] Reference Figure 19 , Figure 19 This is a schematic diagram of an embodiment of a model distillation method provided in this application, as shown below. Figure 19 As shown in the figure, the model distillation method provided in this application embodiment includes:

[0367] 1901. Receive performance requirement information sent by the terminal device.

[0368] In this embodiment of the application, the terminal device can send performance requirement information to the cloud-side device, wherein the performance requirement information may include at least one of the following: accuracy requirement, latency requirement, or model compression ratio requirement.

[0369] In some scenarios, when a terminal device needs to obtain a model for inference from a cloud-based device, it can send a model retrieval request to the cloud-based device. This request may include performance requirement information. The cloud-based device can receive this performance requirement information and determine a suitable student model based on it. When high accuracy is required, the student model can be larger (e.g., with a larger number of parameters or requiring more storage space) to provide a high-performance model. When high latency is required, the student model can be smaller to provide a model that can be quickly derived. When a high model compression ratio (the ratio of the size of the model before scaling to the size of the model after scaling) is required, the student model can be larger to provide a model with a high compression ratio.

[0370] 1902. Obtain the teacher model and determine the student model based on the performance requirement information, wherein the student model is obtained by compressing the teacher model;

[0371] 1903. Knowledge distillation is performed on a student network based on multiple computing nodes to obtain a knowledge-distilled student network, wherein each computing node is responsible for knowledge distillation of a portion of the student network; each computing node is used to feed forward based on the portion of the network it is responsible for to obtain output data; each computing node is also used to determine gradients based on the data obtained from the feedforward of the portion of the network it is responsible for, and to perform gradient backpropagation within each computing node to distill the portion of the network it is responsible for, and the gradients determined by each computing node are not propagated to the connected computing nodes.

[0372] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0373] In one possible implementation, a first-in-first-out (FIFO) queue is also deployed between the multiple serially connected computing nodes to pass the output data to the FIFO queue between the connected computing nodes, so that the connected computing nodes can obtain the output data from the FIFO queue.

[0374] The description of step 1303 can be found above. Figure 11 and Figure 18 The model distillation method provided in the corresponding embodiments will not be described again here.

[0375] 1904. Send the knowledge-distilled student network to the terminal device.

[0376] In this embodiment, once the cloud-side device determines the knowledge-distilled student network, it can send the determined knowledge-distilled student network to the terminal device. The terminal device can then perform inference based on the received student model. It should be noted that the terminal device can further perform model compression processing on the received knowledge-distilled student network, for example, by performing the aforementioned... Figure 11 and Figure 18 The model distillation method provided in the corresponding embodiments is not limited here.

[0377] In this way, cloud-side devices can determine a student model that suits the needs of terminal devices based on the performance requirements of the terminal devices.

[0378] exist Figures 1 to 19 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 20 , Figure 20 This is a schematic diagram of a model distillation apparatus 2000 provided in an embodiment of this application. The model distillation apparatus 2000 can be a terminal device or a server. The apparatus can be applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, and the student model further includes a third sub-model connected to the first sub-model. The second model is a partial model of a teacher model, and the teacher model further includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node. The second computing node deploys the third sub-model and the fourth sub-model. The apparatus includes:

[0379] The acquisition module 2001 is used to acquire first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model;

[0380] The specific description of the acquisition module 2001 can be found in the description of step 1101 in the above embodiments, and will not be repeated here.

[0381] The feedforward module 2002 is used to process the first input data through the first sub-model to obtain the first intermediate output;

[0382] The second sub-model processes the second input data to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient.

[0383] The specific description of the feedforward module 2002 can be found in the descriptions of steps 1102 and 1103 in the above embodiments, and will not be repeated here.

[0384] Distillation module 2003 is used to distill the first sub-model according to the first gradient to obtain an updated first sub-model.

[0385] The specific description of the distillation module 2003 can be found in the description of step 1104 in the above embodiments, and will not be repeated here.

[0386] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0387] In one possible implementation, the first sub-model does not include the output layer of the student model.

[0388] In one possible implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the second sub-model.

[0389] In one possible implementation, the first gradient is used only to update the first sub-model.

[0390] In one possible implementation, the acquisition module 2001 is specifically used for:

[0391] The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the input data, and the at least one first data includes the first input data.

[0392] In one possible implementation, the first queue is a first-in, first-out queue.

[0393] In one possible implementation, before distilling the first sub-model according to the first gradient, the at least one first data includes third input data, the third input data being the output data of the third sub-model; the acquisition module 2001 is further configured to:

[0394] In response to obtaining the updated first sub-model, the third input data is obtained from the first queue, the third input data being used as input data for the feedforward process when performing model distillation on the updated first sub-model.

[0395] In one possible implementation, the acquisition module 2001 is specifically used for:

[0396] The second input data is obtained from the second queue, wherein the second queue is used to store at least one second data from the second computing node, each second data being the output obtained by the second computing node through the processing of input data by the fourth sub-model, and the at least one second data includes the second input data.

[0397] In one possible implementation, the first computing node is also communicatively connected to a third computing node, the student model further includes a fifth sub-model connected after the first sub-model, the first intermediate output is used to be passed to a third queue, the third queue is used to store the first intermediate output so that the third computing node can obtain the first intermediate output from the third queue, and the first intermediate output is used as input data for the feedforward process when performing model distillation on the third sub-model.

[0398] This application provides a model distillation apparatus applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, which further includes a third sub-model connected to the first sub-model. The second sub-model is a partial model of a teacher model, which further includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, which deploys the third and fourth sub-models. The apparatus includes: an acquisition module for acquiring first input data and second input data from the second computing node, wherein the first input data is output data of the third sub-model and the second input data is output data processed by the fourth sub-model; a feedforward module for processing the first input data through the first sub-model to obtain a first intermediate output; and processing the second input data through the second sub-model to obtain a second intermediate output, wherein the first and second intermediate outputs are used to determine a first gradient; and a distillation module for distilling the first sub-model according to the first gradient to obtain an updated first sub-model. In this embodiment, the gradient backpropagation process of each computing node is performed internally, without relying on the next level (or the next N levels, where N is greater than 1) computing node to complete the distillation of the network layer it is responsible for. Furthermore, the gradients calculated by each computing node are not backpropagated to the previous level computing node (for the first gradient obtained by the first computing node, the first gradient is only used to update the first sub-model and may not be backpropagated to the second computing node, i.e., it will not be used by the second computing node when distilling the network layer it is responsible for). This achieves greater utilization of computing resources and thus accelerates the distillation process.

[0399] See Figure 21 , Figure 21 This is a schematic diagram of a model distillation apparatus 2100 provided in an embodiment of this application. The model distillation apparatus 2100 can be a terminal device or a server. The apparatus can be applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, and the second sub-model is a partial model of a teacher model. The teacher model further includes a third sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, and the second computing node deploys the third sub-model. The model distillation apparatus 2100 includes:

[0400] The acquisition module 2101 is used to acquire first input data from the second computing node, wherein the first input data is the output data of the third sub-model;

[0401] The specific description of the acquisition module 2101 can be found in the description of step 1804 in the above embodiment, and will not be repeated here.

[0402] The feedforward module 2102 is used to process the first input data through the first sub-model to obtain the first intermediate output;

[0403] The first input data is processed by the second sub-model to obtain a second intermediate output. The first intermediate output and the second intermediate output are used to determine the first gradient.

[0404] The specific description of the feedforward module 2102 can be found in the descriptions of steps 1802 and 1803 in the above embodiments, and will not be repeated here.

[0405] Distillation module 2103 is used to distill the first sub-model according to the first gradient to obtain an updated first sub-model.

[0406] The specific description of the distillation module 2103 can be found in the description of step 1804 in the above embodiments, and will not be repeated here.

[0407] In one possible implementation, the student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

[0408] In one possible implementation, the first loss is used only to update the first sub-model.

[0409] In one possible implementation, the acquisition module 2101 is specifically used for:

[0410] The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the data, and the at least one first data includes the first input data.

[0411] In one possible implementation, the first queue is a first-in, first-out queue.

[0412] In one possible implementation, before distilling the first sub-model according to the first gradient, the at least one first data includes second input data, the second input data being the output data of the third sub-model; the acquisition module 2101 is further configured to:

[0413] In response to obtaining the updated first sub-model, the second input data is obtained from the first queue, and the second input data is used as input data for the feedforward process when performing model distillation on the updated first sub-model.

[0414] In one possible implementation, the storage resources required to store the first sub-model are less than the storage resources required to store the second sub-model.

[0415] This application provides a model distillation apparatus applied to a first computing node. The first computing node deploys a first sub-model and a second sub-model, wherein the first sub-model is a partial model of a student model, the second sub-model is a partial model of a teacher model, and the teacher model further includes a third sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, and the second computing node deploys the third sub-model. The apparatus includes: an acquisition module for acquiring first input data from the second computing node, wherein the first input data is output data of the third sub-model; a feedforward module for processing the first input data through the first sub-model to obtain a first intermediate output; and processing the first input data through the second sub-model to obtain a second intermediate output, wherein the first intermediate output and the second intermediate output are used to determine a first gradient; and a distillation module for distilling the first sub-model according to the first gradient to obtain an updated first sub-model. At the start of distillation training, the input data obtained by the computing node can be the output of the teacher model of the computing node connected to the previous level (at the start of distillation, the error of the student model is extremely large, and the input of the teacher model is more accurate. Since the error will propagate layer by layer in this embodiment, the convergence speed of the student model is very slow. Using the output of the teacher model as the input at the beginning can reduce the error propagation during feedforward and improve the convergence speed of the student model).

[0416] Reference Figure 22 , Figure 22 This is a schematic diagram of an embodiment of a model distillation apparatus provided in this application, as shown below. Figure 22 As shown in the figure, the model distillation apparatus 2200 provided in this application embodiment includes:

[0417] The receiving module 2201 is used to receive performance requirement information sent by the terminal device.

[0418] The specific description of the receiving module 2201 can be found in the description of step 1901 in the above embodiments, and will not be repeated here.

[0419] The acquisition module 2202 is used to acquire the teacher model and determine the student model based on the performance requirement information, wherein the student model is obtained by compressing the teacher model;

[0420] The specific description of the acquisition module 2202 can be found in the description of step 1902 in the above embodiment, and will not be repeated here.

[0421] The knowledge distillation module 2203 is used to perform knowledge distillation on a student network based on multiple computing nodes to obtain a knowledge-distilled student network. Each computing node is responsible for the knowledge distillation of a portion of the student network. Each computing node is used to perform feedforward based on the portion of the network it is responsible for to obtain output data. Each computing node is also used to determine the gradient based on the data obtained from the feedforward of the portion of the network it is responsible for, and to perform gradient backpropagation within each computing node to distill the portion of the network it is responsible for. The gradients determined by each computing node are not propagated to the connected computing nodes.

[0422] The specific description of the knowledge distillation module 2203 can be found in the description of step 1903 in the above embodiments, and will not be repeated here.

[0423] In one possible implementation, the student model is a transformer model, and the partial model includes one or more transformer layers.

[0424] In one possible implementation, a first-in-first-out (FIFO) queue is also deployed between the multiple serially connected computing nodes to pass the output data to the FIFO queue between the connected computing nodes, so that the connected computing nodes can obtain the output data from the FIFO queue.

[0425] The sending module 2204 is used to send the knowledge-distilled student network to the terminal device.

[0426] The specific description of the sending module 2204 can be found in the description of step 1904 in the above embodiment, and will not be repeated here.

[0427] In this way, cloud-side devices can determine a student model that suits the needs of terminal devices based on the performance requirements of the terminal devices.

[0428] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 23 , Figure 23 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 2300 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, or a server, etc., and is not limited thereto. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303, and a memory 2304 (wherein the execution device 2300 may have one or more processors 2303). Figure 23 (Taking a processor as an example), the processor 2303 may include an application processor 23031 and a communication processor 23032. In some embodiments of this application, the receiver 2301, transmitter 2302, processor 2303, and memory 2304 may be connected via a bus or other means.

[0429] Memory 2304 may include read-only memory and random access memory, and provides instructions and data to processor 2303. A portion of memory 2304 may also include non-volatile random access memory (NVRAM). Memory 2304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0430] Processor 2303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0431] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 2303. The processor 2303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 2303 or by instructions in software form. The processor 2303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 2303 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 2304. Processor 2303 reads the information in memory 2304 and, in conjunction with its hardware, completes the steps of the above method.

[0432] Receiver 2301 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 2302 can be used to output digital or character information through the first interface; transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 2302 may also include a display device such as a display screen.

[0433] In one embodiment of this application, the processor 2303 is configured to execute the above-described... Figures 11 to 19 The student model obtained by the model distillation method is a knowledge distillation product.

[0434] This application also provides a training device; please refer to [link / reference]. Figure 24 , Figure 24This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 2400 is implemented by one or more servers. The training device 2400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 2424 (e.g., one or more processors) and memory 2432, and one or more storage media 2430 (e.g., one or more mass storage devices) for storing application programs 2442 or data 2444. The memory 2432 and storage media 2430 can be temporary or persistent storage. The program stored in the storage media 2430 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 2424 may be configured to communicate with the storage media 2430 and execute the series of instruction operations in the storage media 2430 on the training device 2400.

[0435] The training device 2400 may also include one or more power supplies 2426, one or more wired or wireless network interfaces 2450, one or more input / output interfaces 2458; or, one or more operating systems 2441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0436] In this embodiment of the application, the central processing unit 2424 is used to execute the above-described... Figures 11 to 19 The model distillation method.

[0437] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0438] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0439] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0440] For details, please refer to Figure 25 , Figure 25 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 2500. The NPU 2500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2503, which is controlled by the controller 2504 to retrieve matrix data from the memory and perform multiplication operations.

[0441] In some implementations, the arithmetic circuit 2503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 2503 is a two-dimensional pulsating array. The arithmetic circuit 2503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2503 is a general-purpose matrix processor.

[0442] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2508.

[0443] Unified memory 2506 is used to store input and output data. Weight data is directly transferred to weight memory 2502 via Direct Memory Access Controller (DMAC) 2505. Input data is also transferred to unified memory 2506 via DMAC.

[0444] BIU stands for Bus Interface Unit 2510, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2509.

[0445] The Bus Interface Unit (BIU) 2510 is used by the instruction fetch memory 2509 to fetch instructions from external memory, and also by the memory access controller 2505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0446] The DMAC is mainly used to move input data from external memory DDR to unified memory 2506, or to weight data to weight memory 2502, or to input data to input memory 2501.

[0447] The vector computation unit 2507 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0448] In some implementations, the vector computation unit 2507 can store the processed output vector in the unified memory 2506. For example, the vector computation unit 2507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 2503, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 2503, for example, for use in subsequent layers of the neural network.

[0449] The instruction fetch buffer 2509 connected to the controller 2504 is used to store the instructions used by the controller 2504.

[0450] Unified memory 2506, input memory 2501, weighted memory 2502, and instruction fetch memory 2509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0451] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0452] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0453] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0454] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0455] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model distillation method, characterized in that, The method is applied to a first computing node, which deploys a first sub-model and a second sub-model. The first sub-model is a portion of a student model, and the student model further includes a third sub-model connected to the first sub-model. The second model is a portion of a teacher model, and the teacher model further includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, which deploys the third and fourth sub-models. The data used by the student model during distillation includes at least one of image data, text data, and audio data. The method includes: Obtain first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model; The first input data is processed through the first sub-model to obtain the first intermediate output; The second sub-model processes the second input data to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient. Based on the first gradient, the first sub-model is distilled to obtain the updated first sub-model.

2. The method according to claim 1, characterized in that, The student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

3. The method according to claim 1 or 2, characterized in that, The first sub-model does not include the output layer of the student model.

4. The method according to any one of claims 1 to 2, characterized in that, The step of obtaining the first input data from the second computing node includes: The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being output data obtained by the second computing node through the third sub-model processing the input data, and the at least one first data includes the first input data.

5. The method according to claim 4, characterized in that, The first queue is a first-in, first-out queue.

6. The method according to claim 5, characterized in that, Before distilling the first sub-model according to the first gradient, the at least one first data includes third input data, which is the output data of the third sub-model; After distilling the first sub-model according to the first gradient, the method further includes: In response to obtaining the updated first sub-model, the third input data is obtained from the first queue, the third input data being used as input data for the feedforward process when performing model distillation on the updated first sub-model.

7. The method according to any one of claims 1 to 2, characterized in that, Obtaining second input data from the second computing node includes: The second input data is obtained from the second queue, wherein the second queue is used to store at least one second data from the second computing node, each second data being the output obtained by the second computing node through the processing of input data by the fourth sub-model, and the at least one second data includes the second input data.

8. The method according to any one of claims 1 to 2, characterized in that, The first computing node is also communicatively connected to the third computing node. The student model also includes a fifth sub-model connected after the first sub-model. The first intermediate output is used to be passed to a third queue, which is used to store the first intermediate output so that the third computing node can obtain the first intermediate output from the third queue. The first intermediate output is used as input data for the feedforward process when performing model distillation on the third sub-model.

9. A model distillation method, characterized in that, The method is applied to a first computing node, which deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, and the second sub-model is a partial model of a teacher model. The teacher model further includes a third sub-model connected to the second sub-model. The first computing node and the second computing node are communicatively connected, and the second computing node deploys the third sub-model. The data used by the student model during distillation includes at least one of image data, text data, and audio data. The method includes: Obtain first input data from the second computing node, wherein the first input data is the output data of the third sub-model; The first input data is processed through the first sub-model to obtain the first intermediate output; The first input data is processed by the second sub-model to obtain a second intermediate output. The first intermediate output and the second intermediate output are used to determine the first gradient. Based on the first gradient, the first sub-model is distilled to obtain the updated first sub-model.

10. The method according to claim 9, characterized in that, The student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

11. The method according to claim 9 or 10, characterized in that, The step of obtaining the first input data from the second computing node includes: The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the data, and the at least one first data includes the first input data.

12. The method according to claim 11, characterized in that, Before distilling the first sub-model according to the first gradient, the at least one first data includes second input data, which is the output data of the third sub-model; After distilling the first sub-model according to the first gradient, the method further includes: In response to obtaining the updated first sub-model, the second input data is obtained from the first queue, and the second input data is used as input data for the feedforward process when performing model distillation on the updated first sub-model.

13. A model distillation apparatus, characterized in that, The device is applied to a first computing node, which deploys a first sub-model and a second sub-model. The first sub-model is a portion of a student model, and the student model further includes a third sub-model connected to the first sub-model. The second model is a portion of a teacher model, and the teacher model further includes a fourth sub-model connected to the second sub-model. The first computing node is communicatively connected to a second computing node, which deploys the third and fourth sub-models. The data used by the student model during distillation includes at least one of image data, text data, and audio data. The device includes: The acquisition module is used to acquire first input data and second input data from the second computing node, wherein the first input data is the output data of the third sub-model and the second input data is the output data processed by the fourth sub-model; The feedforward module is used to process the first input data through the first sub-model to obtain the first intermediate output; The second sub-model processes the second input data to obtain a second intermediate output, and the first intermediate output and the second intermediate output are used to determine the first gradient. The distillation module is used to distill the first sub-model according to the first gradient to obtain the updated first sub-model.

14. The apparatus according to claim 13, characterized in that, The student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

15. The apparatus according to claim 13 or 14, characterized in that, The first sub-model does not include the output layer of the student model.

16. The apparatus according to any one of claims 13 to 14, characterized in that, The acquisition module is specifically used for: The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being output data obtained by the second computing node through the third sub-model processing the input data, and the at least one first data includes the first input data.

17. The apparatus according to claim 16, characterized in that, The first queue is a first-in, first-out queue.

18. The apparatus according to claim 17, characterized in that, Before distilling the first sub-model according to the first gradient, the at least one first data includes third input data, the third input data being the output data of the third sub-model; the acquisition module is further configured to: In response to obtaining the updated first sub-model, the third input data is obtained from the first queue, the third input data being used as input data for the feedforward process when performing model distillation on the updated first sub-model.

19. The apparatus according to any one of claims 13 to 14, characterized in that, The acquisition module is specifically used for: The second input data is obtained from the second queue, wherein the second queue is used to store at least one second data from the second computing node, each second data being the output obtained by the second computing node through the processing of input data by the fourth sub-model, and the at least one second data includes the second input data.

20. A model distillation apparatus, characterized in that, An apparatus is applied to a first computing node, which deploys a first sub-model and a second sub-model. The first sub-model is a partial model of a student model, and the second sub-model is a partial model of a teacher model. The teacher model further includes a third sub-model connected to the second sub-model. The first computing node and the second computing node are communicatively connected, and the second computing node deploys the third sub-model. The data used by the student model during distillation includes at least one of image data, text data, and audio data. The apparatus includes: The acquisition module is used to acquire first input data from the second computing node, wherein the first input data is the output data of the third sub-model; The feedforward module is used to process the first input data through the first sub-model to obtain the first intermediate output; The first input data is processed by the second sub-model to obtain a second intermediate output. The first intermediate output and the second intermediate output are used to determine the first gradient. The distillation module is used to distill the first sub-model according to the first gradient to obtain the updated first sub-model.

21. The apparatus according to claim 20, characterized in that, The student model and the teacher model are transformer models, and the first sub-model and the second sub-model include one or more transformer layers.

22. The apparatus according to claim 20 or 21, characterized in that, The acquisition module is specifically used for: The first input data is obtained from the first queue, wherein the first queue is used to store at least one first data from the second computing node, each first data being the output obtained by the second computing node through the third sub-model processing the data, and the at least one first data includes the first input data.

23. The apparatus according to claim 22, characterized in that, Before distilling the first sub-model according to the first gradient, the at least one first data includes second input data, the second input data being the output data of the third sub-model; the acquisition module is further configured to: After distilling the first sub-model according to the first gradient, in response to obtaining the updated first sub-model, the second input data is obtained from the first queue, and the second input data is used as input data for the feedforward process when performing model distillation on the updated first sub-model.

24. A model distillation apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 12.

25. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

26. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1 to 12.

Images