Model distillation method and device, model inference method and inference model

Abstract

The embodiment of the present disclosure provides a model distillation method and device, a model inference method and an inference model, and relates to the technical field of artificial intelligence. The model distillation method comprises the following steps: inputting sample data into a pre-trained teacher model, and determining a first output of an attention layer of the teacher model; inputting the sample data into an initial student model, dividing an input sequence of an attention layer of the initial student model into multiple blocks, selecting a target block from the multiple blocks according to a distillation feature loss increment corresponding to each block, and determining a second output of the attention layer of the initial student model based on the target block; and updating the initial student model according to the difference between the first output and the second output to obtain a target student model. The embodiment can significantly reduce the calculation cost of model inference.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence technology, specifically to model distillation methods and apparatus, model inference methods and inference models, electronic devices, storage media, and computer program products. Background Technology

[0002] In recent years, generative artificial intelligence (AI) technology has made breakthroughs in image and video generation, mainly thanks to diffusion models and the Transformer architecture. Diffusion models dominate image generation, while the Transformer, with its powerful sequence processing capabilities and long-range dependency capture ability, has penetrated from Natural Language Processing (NLP) into Computer Vision (CV). The Diffusion Transformer (DiT) model is a diffusion model based on the Transformer architecture, which utilizes the scalability of the Transformer to improve the quality of generation by scaling the model.

[0003] However, as the scale of inference models such as Wensheng video continues to expand, attention computation in large-scale models has become a major bottleneck restricting inference performance. Summary of the Invention

[0004] Embodiments of this disclosure provide model distillation methods and apparatus, model inference methods and inference models, electronic devices, storage media, and computer program products.

[0005] In a first aspect, embodiments of this disclosure provide a model distillation method, comprising: inputting sample data into a pre-trained teacher model and determining a first output of the attention layer of the teacher model; inputting sample data into an initial student model and dividing the input sequence of the attention layer of the initial student model into multiple blocks, selecting a target block from the multiple blocks according to the distillation feature loss increment corresponding to each block, and determining a second output of the attention layer of the initial student model based on the target block; wherein, the distillation feature loss increment corresponding to each block is the difference between a first distillation feature loss between the output of the attention layer of the initial student model containing the block and the first output, and a second distillation feature loss between the output of the attention layer of the initial student model not containing the block and the first output; updating the initial student model according to the difference between the first output and the second output to obtain a target student model.

[0006] In some embodiments, the input sequence of the attention layer of the initial student model includes a query vector sequence and a key vector sequence. When the initial student model includes multiple attention layers, a target block is selected from multiple blocks based on the distillation feature loss increment corresponding to each block. This includes: for each attention head in each attention layer, determining a corresponding set of candidate key vector blocks for each query vector block of the query vector sequence; for each candidate key vector block in the candidate key vector block set, determining the distillation feature loss increment corresponding to the candidate key vector block based on the difference between the first distillation feature loss and the second distillation feature loss when the candidate key vector block is included and not included; and determining the target key vector block corresponding to the query vector block from the candidate key vector block set based on the distillation feature loss increment corresponding to each candidate key vector block, and using this target block as the target block.

[0007] In some embodiments, determining the target key vector block corresponding to the query vector block from the set of candidate key vector blocks based on the distillation feature loss increment corresponding to each candidate key vector block includes: determining the importance score of each candidate key vector block based on the distillation feature loss increment corresponding to each candidate key vector block; sorting the candidate key vector blocks in the set of candidate key vector blocks by importance according to the importance score corresponding to each candidate key vector block; and determining the target key vector block corresponding to the query vector block from the set of candidate key vector blocks in descending order of importance based on the set number of sparse constraints or the full layer sparsity.

[0008] In some embodiments, updating the initial student model based on the difference between the first output and the second output to obtain the target student model includes: determining a block sparse pattern table based on the target block; wherein the block sparse pattern table records the target key vector block corresponding to each query vector block; updating the quantization parameters of the initial student model based on the block sparse pattern table; determining the second output based on the target block and the updated quantization parameters, and determining the target distillation feature loss based on the difference between the first output and the second output; in response to the determined target distillation feature loss not meeting a preset accuracy threshold, re-determining the target block from multiple blocks, and re-performing the step of determining the block sparse pattern table based on the re-determined target block to re-determine the target distillation feature loss, until the target distillation feature loss meets the preset accuracy threshold, obtaining the target student model, and outputting the block sparse pattern table and the quantization parameter table; wherein the quantization parameter table includes the updated quantization parameters.

[0009] In some embodiments, updating the quantization parameters of the initial student model according to the block sparse pattern table includes: according to the block sparse pattern table, skipping blocks other than the target block among multiple blocks, and updating the quantization parameters of the target block.

[0010] In some embodiments, the sample data comes from a calibration set; the initial values ​​of the quantization parameters are determined based on statistics or clustering of the calibration set.

[0011] In some embodiments, determining the target distillation feature loss based on the difference between the first output and the second output includes: determining the target distillation feature loss based on the Euclidean distance between the first output and the second output.

[0012] In some embodiments, the teacher model includes a converter model or a diffusion-converter model for generating images or videos; the initial student model is obtained based on the teacher model.

[0013] Secondly, embodiments of this disclosure provide a model inference method, which includes: inputting data to be processed into a target student model; wherein the target student model is obtained according to the model distillation method mentioned in any of the first aspects; performing attention operations only on target blocks in the input sequence based on the input sequence of the attention layer of the target student model to obtain a sparse attention output result; and generating target content based on the sparse attention output result, wherein the target content includes an image or a video.

[0014] In some embodiments, attention operations are performed only on the target blocks in the input sequence to obtain sparse attention output results, including: determining the target key vector block corresponding to each query vector block of the query vector sequence for each attention head in each attention layer of the target student model according to a determined block sparse pattern table; performing block quantization on the input sequence according to the quantization parameters in a determined quantization parameter table; wherein the block size of the block quantization is consistent with the block size of the input sequence; and performing matrix multiplication and aggregation on each query vector block and the corresponding target key vector block based on the block quantization input sequence to obtain the sparse attention output results of each attention layer.

[0015] Thirdly, embodiments of this disclosure provide a model distillation apparatus, comprising: a model determination module configured to determine an initial student model based on a pre-trained teacher model, wherein the teacher model includes an attention layer; a block sparse selection module configured to input sample data into the teacher model and determine a first output of the attention layer of the teacher model, and input sample data into the initial student model, divide the input sequence of the attention layer of the initial student model into multiple blocks, select a target block from the multiple blocks according to the distillation feature loss increment corresponding to each block, and determine a second output of the attention layer of the initial student model based on the target block; wherein the distillation feature loss increment corresponding to each block is the difference between a first distillation feature loss between the output of the attention layer of the initial student model containing the block and the first output, and a second distillation feature loss between the output of the attention layer of the initial student model not containing the block and the first output; and an update module configured to update the initial student model based on the difference between the first output and the second output to obtain a target student model.

[0016] Fourthly, embodiments of this disclosure provide an inference model, which is a target student model obtained according to the model distillation method of any one of the first aspects. The inference model includes: an attention operation module configured to perform attention operation only on target blocks in the input sequence based on the input sequence of the attention layer to obtain sparse attention output results; and a content generation module configured to generate target content based on the sparse attention output results, wherein the target content includes images or videos.

[0017] In some embodiments, the inference model further includes a quantization module configured to perform block quantization on the input sequence according to quantization parameters in a determined quantization parameter table; wherein the block size of the block quantization is consistent with the block size of the input sequence; the attention operation module is further configured to determine the target key vector block corresponding to each query vector block of the query vector sequence for each attention head in each attention layer of the inference model according to a determined block sparsity pattern table; and, based on the block quantized input sequence, perform matrix multiplication and aggregation on each query vector block and the corresponding target key vector block to obtain the sparse attention output result of each attention layer.

[0018] Fifthly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having stored one or more computer programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement a model distillation method as mentioned in any of the first aspects, or a model inference method as mentioned in any of the second aspects.

[0019] In a sixth aspect, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a model distillation method as mentioned in any of the first aspects, or a model inference method as mentioned in any of the second aspects.

[0020] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements a model distillation method as mentioned in any of the first aspects, or a model inference method as mentioned in any of the second aspects.

[0021] The model distillation method and apparatus, model inference method and inference model provided in the embodiments of this disclosure divide the input sequence of the initial student model into blocks and determine the distillation feature loss increment corresponding to each block. Since the distillation feature loss increment can reflect the real impact of removing a block on the final output, the sparse selection decision can be made sensitive to the final output. Then, based on the distillation loss feature increment corresponding to each block, the most important block is selected from multiple blocks for sparse attention operation, which can significantly reduce the amount of computation and memory usage while retaining the ability of the attention layer to capture key information.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied; Figure 2 This is a flowchart of one embodiment of the model distillation method according to the present disclosure; Figure 3 This is a flowchart of the model distillation method according to this disclosure in an application scenario; Figure 4 This is a flowchart of an embodiment of the model reasoning method according to the present disclosure; Figure 5 This is a schematic diagram of the structure of one embodiment of the model distillation apparatus according to the present disclosure; Figure 6 This is a schematic diagram of a structure based on an embodiment of the reasoning model of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0024] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Currently, attention computation in the Transformer / DiT architecture has gradually become a bottleneck for inference in large models. According to relevant technologies, the semantic sequence of the input is represented by a vector matrix in the computation of the Transformer attention layer. The input sequence includes a query vector (Query, Q), a key vector (Key, K), and a value vector (Value, V). Attention computation based on the Transformer architecture mainly includes the following processes: ; Among them, K T Let d represent the transpose of key vector K, d represent the dimension of K, and S represent the similarity score between query vector Q and K. for and The dot product represents the correlation or attention weight between each element in the sequence and other elements.

[0027] Then, the attention weight P is obtained by normalizing the similarity score S: ; Where P is the attention weight and softmax is the normalization function.

[0028] Finally, by aggregating the information in V based on P, the attention output result O is obtained: ; Where O represents the output result. It is the multiplication of attention weight P and value vector V, that is, the aggregation of information in V based on P to obtain the final calculation result of the attention mechanism.

[0029] Currently, the following attention quantization scheme is mainly used in accelerating inference in the model: 1. Block Quantization Attention Mechanism This scheme targets block-quantized attention operators (Flash Attention, FA), such as the Sparse Attention with Graph Embeddings (SageAttention) series, and addresses the two core matrix multiplications in attention computation. PV implements block-level low-bit quantization, uses low-precision data types such as integers INT8, INT4 or floating-point numbers FP8 for computation, and combines smoothing and quantization strategies for Q / K / V tensors to accelerate the computation of standard attention operators while maintaining model accuracy.

[0030] 2. Combining sparse attention with quantification

[0031] Represented by the Sparse Guided Attention (SpargeAttn) mechanism, this method proposes a sparse quantization attention mechanism that does not require training. By predicting and filtering the attention map online, it skips some computations and achieves a combination of sparsity and quantization techniques.

[0032] 3. Post-training quantification

[0033] Post-Training Quantization (PTQ) techniques typically use a small amount of calibration data to set quantization parameters, such as scaling factors, clipping thresholds, and zero-points, aiming to minimize accuracy loss when converting from a floating-point model to a low-precision model.

[0034] 4. The idea of ​​sparse / quantized calibration based on distillation

[0035] For example, in the PTQ process of the video diffusion model, the use of significant data selection and sparse distillation schemes effectively alleviates the challenges brought by long sequences.

[0036] However, the attention quantification schemes mentioned above generally suffer from at least one of the following problems: (1) The selection criteria for sparse blocks are not sensitive enough to the output: Heuristic scores based on attention scores or top-k scores cannot truly reflect the impact of a block being cut off on the final output, making generative models prone to local quality degradation.

[0037] (2) Decoupling of sparsity and quantization: The sparsity mode and quantization parameters are determined independently. Under sparse conditions, the quantization error may be amplified, resulting in unstable model generation accuracy / quality.

[0038] (3) Insufficient integration in the inference stage: Mask processing, quantization / dequantization, matrix multiplication and softmax operations are executed by multiple kernels, which brings additional memory access and scheduling overhead, and significantly increases the amount of attention computation and GPU memory usage.

[0039] The model distillation method and apparatus, model reasoning method and reasoning model provided according to the embodiments of this disclosure at least partially solve the above-mentioned problems existing in the related art.

[0040] The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present disclosure can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this disclosure are not limited to the order in which they are described, but can be performed in any order or in parallel.

[0041] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the model distillation method or model distillation apparatus and model inference method or inference model of the present disclosure can be applied.

[0042] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays that support receiving messages from the server, including but not limited to smartphones, tablets, e-book readers, electronic players, laptops, and desktop computers, etc.

[0044] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software modules used to provide distributed services), or as a single software program or software module, such as a client / server application or a web browser application. No specific limitations are made here.

[0045] Server 105 can be a server providing various services. Taking a model distillation application that can provide acceleration services for inference models as an example, server 105 can achieve the following effects when running a model distillation application: First, sample data is input into a pre-trained teacher model, and the first output of the attention layer of the teacher model is determined; then, sample data is input into an initial student model, and the input sequence of the attention layer of the initial student model is divided into multiple blocks, and a target block is selected from the multiple blocks according to the distillation feature loss increment corresponding to each block, and the second output of the attention layer of the initial student model is determined based on the target block; wherein, the distillation feature loss increment corresponding to each block is the difference between the first distillation feature loss between the output of the attention layer of the initial student model containing the block and the first output, and the second distillation feature loss between the output of the attention layer of the initial student model not containing the block and the first output; finally, the initial student model is updated according to the difference between the first output and the second output to obtain the target student model.

[0046] In addition, taking a model inference application that can provide model inference services as an example, the server 105 can achieve the following effects when running the model inference application: First, the data to be processed is input into the pre-obtained target student model; then, according to the input sequence of the attention layer of the target student model, attention operation is performed only on the target block in the input sequence to obtain the sparse attention output result; finally, the target content is generated based on the sparse attention output result, and the target content may include images or videos.

[0047] It's important to note that a server can be either hardware or software. When a server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When a server is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here. A server can also be a server for a distributed system, or a server integrated with blockchain technology. A server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0048] It should be noted that the model distillation method or model inference method provided in this embodiment of the present disclosure is generally executed by the server 105, and correspondingly, the model distillation device or inference model is generally set in the server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] Figure 2A flow 200 of an embodiment of the model distillation method according to this disclosure is shown. The entity performing the model distillation method may be, for example, Figure 1 The server 105 shown, process 200 includes the following steps: Step 201: Input the sample data into the pre-trained teacher model and determine the first output of the attention layer of the teacher model.

[0051] In the model distillation method according to embodiments of this disclosure, the teacher model can be a pre-trained model with fixed parameters. This teacher model can generate high-quality images or videos, exhibiting high accuracy and generalization ability, but it is typically large in scale, has many parameters, and high computational complexity. Embodiments of this disclosure aim to use knowledge distillation, with the teacher model as a knowledge source, to provide guidance signals and transfer the knowledge of the teacher model to the student model. This allows the student model to mimic the output behavior of the teacher model, reducing computational requirements while maintaining good inference performance and generation quality / accuracy.

[0052] According to some exemplary implementations of this disclosure, the model architecture of the teacher model may adopt, for example, the Transformer architecture or the DiT model architecture.

[0053] In practical applications, teacher models can be used to generate text, images, or videos. For example, a teacher model can be an Artificial Intelligence Generated Content (AIGC) model that processes input sample data to generate text, images, videos, and other content. Taking image or video generation as an example, the input sample data can include at least one of image data and text data. The text data can be, for example, prompt text, used to generate a target image or video. The image data can be, for example, an initial noisy image or its latent representation, used to generate a target image or video. The initial noisy image or its latent representation is a two-dimensional structure. By dividing the two-dimensional image or latent representation into fixed-size image patches and embedding these patches into a high-dimensional vector, the two-dimensional graphic is transformed into a one-dimensional sequence of tokens, enabling the model to process it.

[0054] Processed token sequence x The input is fed into the attention layer of the teacher model for processing. xIt's a matrix containing all token information, obtained by linearly transforming the input token vector matrix to form the Q / K / V matrix. Inside the attention layer, each token corresponds to a Q, K, and V matrix. Simply put, a token is a semantic unit, a vectorized representation of the input tokens of the attention layer.

[0055] Attention mechanisms in Transformer-based architectures can include self-attention and cross-attention. Self-attention handles internal dependencies, calculating the relevance of each element (e.g., a token) to all other elements within a single sequence. Cross-attention, on the other hand, handles external condition injection; for example, between two sequences, a text-described query can be used to focus on image feature keys / values ​​to generate an image that matches the text description.

[0056] In some exemplary implementations, the network structure of the teacher model may include multiple attention layers, each of which may include multiple attention heads. The multi-head attention output of that attention layer is obtained by concatenating the features of the outputs of all attention heads and performing a linear transformation. As the core part of the model, the stacked attention layers are responsible for processing serialized tokens to capture the dependencies between different tokens in the sequence. They propagate the output of the previous layer as the input of the next layer through a hierarchical manner, ultimately obtaining the output of the attention layer, which is also referred to as the first output in this paper.

[0057] In some implementations, the teacher model can also receive input time step information and conditional information such as image category labels or text descriptions. The teacher model can then perform image generation and transformation processes based on the input conditional information, achieving consistency between the generated image and the text description.

[0058] It should be understood that the model structure of the teacher model will vary depending on different application scenarios or different model output requirements. This disclosure does not impose further specific restrictions on the model structure of the teacher model.

[0059] Step 202: Input sample data into the initial student model, divide the input sequence of the attention layer of the initial student model into multiple blocks, select the target block from the multiple blocks according to the distillation feature loss increment corresponding to each block, and determine the second output of the attention layer of the initial student model based on the target block.

[0060] According to the embodiments of this disclosure, the initial student model can be determined based on the teacher model. For example, the basic model structure of the initial student model can be the same as that of the teacher model, and the model parameters of the teacher model can be used as initial parameters. That is, the initial student model also includes multiple attention layers, and each attention layer can include multiple attention heads. The main difference between the two is that the attention layers of the teacher model use standard attention operators or dense attention operators, while the initial student model uses more efficient sparse attention operators. Here, sparse attention operators refer to those used in the initial student model. , PV When performing attention calculations such as matrix multiplication, some redundant calculations can be skipped by setting a mask to reduce computational complexity.

[0061] In some implementations, the input sequence of the attention layer of the initial student model may include a query vector sequence Q, a key vector sequence K, and a value vector sequence V. The input sequence Q / K is divided into multiple tiles along the sequence dimension, referred to herein as blocks. For example, based on a preset sequence length B, each B tokens constitute a block.

[0062] Furthermore, for each attention head of each attention layer in the initial student model, a target block can be selected from multiple blocks based on the distillation feature loss increment corresponding to each block, and sparse attention operations are performed only on the target block, so that each token interacts only with a portion of the tokens in the sequence, thereby effectively reducing computational overhead and memory usage.

[0063] In order to enable the initial student model to learn the teacher model's ability to capture key information, we define a distillation feature loss to reflect the difference between the output of the attention layer of the initial student model and the output of the attention layer of the teacher model, i.e., the first output mentioned above.

[0064] In some optional implementations, by determining the distillation feature loss increment corresponding to each block, the most important block is selected from multiple blocks of the input sequence as the target block for sparse attention operation. Here, the distillation feature loss increment corresponding to each block refers to the difference between the distillation feature loss when the block is removed and when it is retained; that is, the difference between the first distillation feature loss between the output of the attention layer of the initial student model and the first output when the block is included, and the second distillation feature loss between the output of the attention layer of the initial student model and the first output when the block is not included.

[0065] Furthermore, based on the determined target blocks, the final output of the attention layer of the initial student model is determined by performing sparse attention operations only on the target blocks. This output is also referred to as the second output in this paper.

[0066] Step 203: Update the initial student model based on the difference between the first output and the second output to obtain the target student model.

[0067] In some implementations, the target distillation feature loss can be determined based on the distance between the first output and the second output, and then the parameters of the initial student model can be updated based on this target distillation feature loss. This target distillation feature loss can be understood as the output feature loss of the attention layers of the student model and the teacher model, used to optimize the depth and accuracy of feature learning by the attention layer of the initial student model.

[0068] In some exemplary embodiments, the target distillation feature loss can be determined by calculating the L2 (Euclidean distance) loss between the first output of the teacher model and the second output of the initial student model. It is understood that the target distillation feature loss can also be determined using other differences or distances between the first and second outputs, and this disclosure does not impose any specific limitations on this.

[0069] In some exemplary implementations, during the update of the initial student model, the sample data used may come from a calibration set, which helps to accelerate the training process and enables the final target student model to accurately mimic the output of the teacher model on specific inputs.

[0070] The model distillation method provided in the above embodiments of this disclosure divides the input sequence of the initial student model into blocks and determines the distillation feature loss increment corresponding to each block. Since the distillation feature loss increment can reflect the real impact of removing a certain block on the final output, it can make the sparse selection decision sensitive to the final output. Then, based on the distillation loss feature increment corresponding to each block, the most important block is selected from multiple blocks for sparse attention operation, which can significantly reduce the amount of computation and memory usage while retaining the ability of the attention layer to capture key information.

[0071] According to some embodiments of this disclosure, for Figure 2 In step 202 above, after dividing the input sequence of the attention layer of the initial student model into multiple blocks, the distillation feature loss increment corresponding to each block is further determined, and the target block is selected from multiple blocks according to the distillation feature loss increment corresponding to each block. This process can be specifically implemented as follows: steps 2021-2023: Step 2021: For each attention head in each attention layer, determine the corresponding set of candidate key vector blocks for each query vector block of the query vector sequence.

[0072] As mentioned earlier, the attention mechanism of the standard Transformer architecture is defined as a function of three vectors: Q, K, and V. Given an input sequence, the model transforms it to obtain the Q / K / V matrix. In the computational calculations related to the attention layer, the computational complexity mainly stems from... Matrix multiplication has a time complexity of O(N). 2 ·d). When the sequence length N is much larger than the dimension d, O(N) 2 ) thus became the dominant factor.

[0073] To reduce computational load and complexity, according to the embodiments of this disclosure, for each head of each attention layer, the query vector sequence Q is divided into multiple query vector blocks, denoted as q-blocks, and the key vector sequence K is divided into multiple key vector blocks, denoted as k-blocks. Then, a corresponding set of candidate key vector blocks, denoted as C(q), is determined for each q-block.

[0074] In some alternative implementations, the candidate key vector block set C(q) may include some key vector blocks from the attention layer. For example, based on a preset local window, only tokens within the window can participate in the computation, because in data processing such as natural language and images, the correlation between adjacent elements is usually stronger than that between elements that are far apart. Alternatively, based on preset global blocks, some special tokens or regions in the sequence can be preset as anchor points, and sparse attention operations can be performed with these anchor points as the query center.

[0075] However, the sparse selection modes based on local windows or preset anchor points described above cannot reflect the true impact of removing a certain block on the final output of the model's attention layer, and therefore are prone to local quality degradation when generating class models. To avoid this problem, in some optional implementations, for each q-block, the candidate key vector block set C(q) can also be determined to include all other k-blocks in the attention layer.

[0076] Step 2022: For each candidate key vector block in the candidate key vector block set, determine the distillation feature loss increment corresponding to the candidate key vector block based on the difference between the first distillation feature loss and the second distillation feature loss when the candidate key vector block is included and not included.

[0077] First, the distillation characteristic loss is defined as follows: : ; in, For the dense attention output of the attention layer of the teacher model, This is the sparse attention output of the attention layer of the initial student model based on target segmentation. x This refers to the input sample data.

[0078] Furthermore, for each candidate k-block in the candidate key vector block set C(q), the distillation feature loss increment Δ is determined as follows: L ( b ): ; in, The characteristic loss of the first distillation, This is a characteristic loss of the second distillation process. To preserve the output of the attention layer of the initial student model when this block is divided, This is the output of the attention layer of the initial student model when removing this block.

[0079] The increment of distillation characteristic loss Δ L ( b The larger the value, the more critical the candidate k-block is, and the more it should be retained in the attention operation.

[0080] Step 2023: Determine the target key vector block corresponding to the query vector block from the set of candidate key vector blocks based on the distillation feature loss increment corresponding to each candidate key vector block, and use it as the target block.

[0081] In some alternative implementations, the distillation feature loss increment Δ corresponding to each candidate k-block in C(q) can be used. L ( b The importance score of each candidate k-block is determined, and the candidate k-blocks in C(q) are ranked according to their importance scores. Then, based on the set number of sparse constraints, such as top-K, K k-blocks corresponding to each q-block are determined from C(q) in descending order of importance as the final target blocks to be retained. Here, K is the maximum number of k-blocks to be retained for each q-block, and the value of K can be a positive integer.

[0082] In some alternative implementations, the distillation feature loss increment Δ corresponding to each candidate k-block in C(q) can be used. L ( b The importance score of each candidate k-block is determined by averaging the calibration set.

[0083] It should be understood that, without departing from the teachings of this disclosure, other alternative implementation methods may be used to determine the maximum number of k-blocks retained per q-block, for example, based on a preset full-layer sparsity r, etc., and this disclosure does not impose any specific limitations on this.

[0084] According to the above-described embodiments of this disclosure, by combining the input sequence of the attention layer with sparse selection based on importance ranking, each token interacts only with a portion of the more important tokens in the sequence. Compared with the fully connected attention graph mode of the teacher model, this can significantly reduce computation and memory usage. Furthermore, by adopting a sparse selection mode based on distilled feature loss increment, the ability of the attention layer to capture key information is preserved as much as possible while reducing computation and complexity.

[0085] Furthermore, according to some embodiments of this disclosure, for Figure 2 Step 203 above, which updates the initial student model based on the target distillation feature loss to obtain the target student model, can be specifically implemented as follows: steps 2031-2034: Step 2031: Determine the block sparsity pattern table based on the target block.

[0086] Specifically, the PatternTable (block sparse pattern table) is determined based on the selected final retained target blocks. The PatternTable records the target k-block corresponding to each query vector block q-block.

[0087] As an exemplary implementation, the index list of the target k-block corresponding to each q-block of each attention head in each attention layer can be stored in the PatternTable. For example, a compressed coded segmented index or a compressed sparse row (CSR) can be used, where CSR is an efficient sparse matrix storage format.

[0088] Step 2032: Update the quantization parameters of the initial student model according to the block sparse pattern table.

[0089] According to the embodiments of this disclosure, attention quantization refers to the process of converting the computation and storage involved in the attention mechanism from high precision to low precision, so as to reduce the computational overhead and memory usage of the attention layer while minimizing the loss of precision.

[0090] In some exemplary implementations, the initial quantization parameters of the student model may include scale, and optionally, zero-point, clip, or the amax (Absolute Maximum) required by FP8. The initial values ​​of the quantization parameters may be determined based on statistics or k-means clustering of the calibration set. For example, the initial quantization parameters can be obtained by initially quantizing the model using a small amount of calibration set data. Alternatively, the initial quantization parameters may be determined based on preset default values.

[0091] Step 2033: Determine the second output based on the target block and the updated quantization parameters, and determine the target distillation feature loss based on the difference between the first output and the second output.

[0092] The distillation training process for the initial student model can be divided into a forward propagation phase and a backpropagation phase. In the forward propagation phase, the dense attention output of the teacher model based on a full attention mechanism and the sparse attention output of the initial student model based on a sparse attention mechanism are determined, and the distillation feature loss is determined based on the difference between the two. In the backpropagation phase, the initial quantization parameters of the initial student model are updated based on the determined distillation feature loss, causing the distillation feature loss to gradually converge to a lower level.

[0093] Specifically, for the initial student model, attention operations are performed on each q-block and which k-blocks are selected based on the determined PatternTable to determine the sparse attention output. The quantization parameters of the initial student model are continuously adjusted through iterative training so that the target distillation feature loss between the first output and the second output is reduced to meet the preset accuracy threshold.

[0094] In some alternative implementations, when updating the quantization parameters of the initial student model according to the PatternTable, it can be determined which blocks are ignored or skipped. This way, during the update of the initial quantization parameters, only the loss values ​​of important tokens retained in sparse mode need to be considered; for unimportant tokens or regions, the weights can be reset to zero or given lower weights. For example, based on the determined PatternTable, for each q-block, blocks other than the target k-block corresponding to that q-block can be skipped, and only the quantization parameters of the region corresponding to the target k-block can be updated.

[0095] Step 2034: Iteratively train the initial student model based on the target distillation feature loss. If the target distillation feature loss meets the preset accuracy threshold, obtain the target student model and output the block sparse pattern table and the quantization parameter table.

[0096] As an exemplary implementation, a preset accuracy threshold can be used as a metric to evaluate the learning quality of the initial student model. After each training round, the target distillation feature loss between the outputs of the attention layers of the initial student model and the teacher model is calculated. If the target distillation feature loss value does not meet the preset accuracy threshold, the initial student model continues to be iteratively updated. Training can be stopped when the loss value converges to less than the preset accuracy threshold, resulting in the trained target student model. The current PatternTable and QuantTable (quantization parameter table) can then be output. The quantization parameter table includes the updated quantization parameters.

[0097] In some optional implementations, if the target distillation feature loss does not meet the preset accuracy threshold, the target block can be re-determined from multiple blocks of the input sequence. The number of target blocks retained can also be changed by adjusting the value of K. Based on the re-determined target block, the above steps 2031-2033 are returned to be executed again to redetermine the target distillation loss and iteratively update the initial student model.

[0098] According to the above embodiments of this disclosure, the initial student model can be updated based on the determined distillation feature loss. By comparing the distillation feature differences between the second output and the first output, effective gradient information can be provided to optimize the block sparsity mode of the attention layer and the low-bit quantization strategy of the initial student model, thereby improving the inference performance and efficiency of the target student model.

[0099] To facilitate understanding and explanation, and to highlight the effects of the model distillation method provided in this disclosure in practical application scenarios as much as possible, the model distillation method provided in this disclosure will be described in detail below in conjunction with specific application scenarios.

[0100] See also Figure 3 This illustrates a flowchart 300 of a model distillation method for a specific application scenario. Taking an application scenario for generating images as an example, this method can be performed by, for example... Figure 1 The server 105 shown is used to execute the process. The processing flow 300 of this model distillation method can be divided into the following stages: Phase S1: Data and model preparation phase.

[0101] Step 301: Prepare the calibration set.

[0102] The calibration set D can come from random sampling of real requests from the teacher model, or it can be offline sampled data. The calibration set D can be used to observe the distribution of activation values ​​in the model, thereby determining appropriate quantization parameters, such as scale and zero-point.

[0103] Step 302: Determine the initial student model.

[0104] For example, the teacher model is a standard Transformer structure without any compression, based on a dense attention mechanism. This means that each token needs to calculate its correlation with all other tokens, and attention computation uses raw precision such as FP16. The high-precision teacher model's final attention layer output during forward inference is shown below. It can be used to guide the training or calibration of an initial student model that has undergone block sparse / quantized processing.

[0105] The initial student model is determined based on the teacher model, and the attention layer is replaced with a block sparse-low bit quantization fusion operator to support block-level sparse masking and block quantization.

[0106] Step 303: Initialize the quantization parameter table.

[0107] Initial quantization parameters, such as scale and zero-point, are determined for each attention layer / attention head / block of the initial student model and stored in the quantization parameter table QuantTable. The initialization of the quantization parameter table can be based on statistics of the calibration set, k-means clustering, or default values.

[0108] Phase S2: Block sparse mode selection phase.

[0109] Step 304: Determine the candidate k-block set.

[0110] For each head in each attention layer, determine a set of candidate k-blocks C(q) for each q_block of the query vector sequence Q.

[0111] For example, a small amount of sample data from the calibration set is input into a pre-trained, parameter-frozen teacher model to determine attention statistics. For each q-block, the top-n most concerned blocks are selected, such as the n k-blocks with the highest weights in the teacher model, as candidate k-blocks, forming a candidate k-block set C(q). The value of n can be selected from positive integers.

[0112] Step 305: Evaluate the importance of candidate k-blocks.

[0113] For each sample data x ∈D, based on the output of the attention layer of the teacher model and the output of the initial student model Determine the distillation characteristic loss function : ; Where M is the block sparse attention operator of the attention layer of the initial student model, and Q is the Q sequence of the attention layer of the initial student model.

[0114] For each candidate k-block of q-block in C(q), calculate the distillation loss. and To determine the increment of distillation characteristic loss Δ L ( b ), and based on this distillation characteristic loss increment Δ L ( b The importance of candidate k-blocks is evaluated to determine the target k-block corresponding to each q-block from C(q). This process can be specifically described in steps 2022 and 2023 above, and will not be repeated here.

[0115] Step 306: Determine the block sparse pattern table.

[0116] Based on the target k-block corresponding to each determined q-block, a PatternTable is generated, and the index list of the target k-block corresponding to each q-block is recorded in the PatternTable, including information such as the layer number, head number, q-block index, and the list of retained target k-blocks for each attention layer.

[0117] Phase S3: Calibration phase.

[0118] Step 307: Perform joint optimization of the Mask and quantization parameters.

[0119] Step 308: Determine whether the preset accuracy threshold is met. If yes, proceed to step 309; otherwise, return to step 304 or 305 to iterate again.

[0120] During the calibration phase, the Mask and quantization parameters are alternately optimized to minimize the distillation feature loss: First, the PatternTable is fixed based on the target k-block, and the quantization parameters such as scale and zero point in the QuantTable are updated to minimize the quantization error of the model's weights / activations, resulting in updated quantization parameters such as scale and zero point. Then, the distillation feature loss is determined based on the target block and the updated quantization parameters. If the distillation feature loss has converged to meet a preset accuracy threshold, training can stop; otherwise, the current quantization parameters in the QuantTable are fixed, and the process returns to step 304 or 305 to continue adjusting the target k-block or candidate k-block set C(q) and adjusting the PatternTable. This iterative training continues until the distillation feature loss meets the accuracy threshold or the maximum number of iterations is reached.

[0121] Stage S4: Curing stage.

[0122] Step 309 yields the target student model.

[0123] If the training has converged to meet the preset accuracy threshold, the training is complete, the target student model is obtained, and the current PatternTable and QuantTable can be exported for quick loading in the subsequent inference stage.

[0124] The model distillation methods described above illustrate from various aspects how to distill to obtain the target student model. Furthermore, this disclosure also specifically provides a scheme for using the trained target student model for inference.

[0125] Figure 4 A flowchart of a model inference method provided in an embodiment of this disclosure is shown. See also Figure 4 The process 400 specifically includes the following steps: Step 401: Input the data to be processed into the target student model.

[0126] The target student model is trained using the model distillation method described in any of the above embodiments. The data to be processed may include a fusion of various data or information, such as text information, image data, etc.

[0127] During the online inference phase, the required input parameters include: model parameters, a fixed PatternTable, and a QuantTable. The PatternTable may include a list of indices of the target k-block corresponding to each attention layer / attention head / q-block, and the QuantTable may include fixed quantization parameters, such as scale and zero point.

[0128] Step 402: Based on the input sequence of the attention layer of the target student model, perform attention operations only on the target block in the input sequence to obtain the sparse attention output result.

[0129] During the inference phase, for each attention layer, the block sparse-low-bit quantization fusion operator is invoked. This involves selecting the target block to participate in the computation based on the PatternTable, and simultaneously quantizing the input sequence into blocks based on the QuantTable. Specifically, this is achieved by performing matrix multiplication on each q-block and its corresponding k-block, and by quantizing the input sequence into a low-precision data type, thus quantizing the attention operations. Multiplication is transformed into low-bit matrix multiplication of (q-block, k-block) pairs, thereby significantly reducing the amount of computation and computational complexity.

[0130] In some optional implementations, the block size of block quantization can be consistent with the block size of the input sequence. Specifically, based on the determined PatternTable, for each attention head in each attention layer, the target k-block corresponding to each q-block of the Q-sequence is determined, and blocks other than the target k-block are skipped; the input sequence is block quantized according to the quantization parameters in the determined QuantTable, and low-bit quantization is performed on the retained target k-block; based on the block quantized input sequence, each q-block and its corresponding k-block are... Matrix multiplication and aggregation output yield the sparse attention output results for each attention layer. By aligning block quantization with the block size of the input sequence, the sparse block selection is perfectly aligned with the granularity of the block quantization attention operator. This allows block skipping, low-bit matrix multiplication, and aggregation output to be completed within a single operator, avoiding the memory read / write and scheduling overhead caused by multiple kernel executions, reducing GPU memory usage, and improving inference computational throughput.

[0131] Step 403: Generate target content based on the sparse attention output.

[0132] Based on the sparse attention output of the attention layer, a two-dimensional feature map is restored and mapped back to the pixel space to generate target content, such as generating a target image or target video that matches the text information input by the user.

[0133] The model inference method provided in the above embodiments of this disclosure, since the input sequence of the attention layer of the target student model is divided into multiple blocks during the training process, and the target block is determined from multiple blocks, and sparse attention output is obtained by performing attention operation based on the target block, fundamentally reduces the amount of computation and complexity of model inference, thereby significantly improving the inference speed and reducing the computational cost. In addition, the block sparse selection mechanism based on feature distillation loss increment can enhance the learning ability of key information and make the target student model have better generalization ability. Therefore, it effectively solves the core bottleneck problem of large-scale inference model deployment, making it possible for high-quality image or video generation tasks to run in real time on edge devices with limited computing resources.

[0134] Next reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model distillation apparatus, which corresponds to the aforementioned model distillation method embodiment, and the apparatus can be specifically applied to various electronic devices.

[0135] like Figure 5 As shown, the model distillation apparatus 500 of this embodiment includes: a model determination module 501, a block sparsity selection module 502, and an update module 503.

[0136] In this embodiment, the model determination module 501 is configured to determine the initial student model based on a pre-trained teacher model, wherein the teacher model includes an attention layer.

[0137] The block sparse selection module 502 is configured to input sample data into a teacher model and determine the first output of the attention layer of the teacher model, and to input sample data into an initial student model and divide the input sequence of the attention layer of the initial student model into multiple blocks, and select a target block from the multiple blocks according to the distillation feature loss increment corresponding to each block, and determine the second output of the attention layer of the initial student model based on the target block; wherein, the distillation feature loss increment corresponding to each block is the difference between the first distillation feature loss between the output of the attention layer of the initial student model and the first output when the block is included, and the second distillation feature loss between the output of the attention layer of the initial student model and the first output when the block is not included.

[0138] Update 503 is configured to update the initial student model based on the difference between the first output and the second output to obtain the target student model.

[0139] In some optional implementations of this embodiment, the input sequence of the attention layer of the initial student model includes a query vector sequence and a key vector sequence; the block sparse selection module 502 is further configured to: when the initial student model includes multiple attention layers, select a target block from multiple blocks according to the distillation feature loss increment corresponding to each block, including: for each attention head in each attention layer, determine a corresponding set of candidate key vector blocks for each query vector block of the query vector sequence; for each candidate key vector block in the set of candidate key vector blocks, determine the distillation feature loss increment corresponding to the candidate key vector block according to the difference between the first distillation feature loss and the second distillation feature loss when the candidate key vector block is included and not included; determine the target key vector block corresponding to the query vector block from the set of candidate key vector blocks according to the distillation feature loss increment corresponding to each candidate key vector block, and use it as the target block.

[0140] In some optional implementations of this embodiment, the block sparsity selection module 502 is further configured to: determine the importance score of each candidate key vector block based on the distillation feature loss increment corresponding to each candidate key vector block; sort the candidate key vector blocks in the candidate key vector block set according to the importance score corresponding to each candidate key vector block; and determine the target key vector block corresponding to the query vector block from the candidate key vector block set in descending order of importance based on the set number of sparsity constraints or the full layer sparsity rate.

[0141] In some optional implementations of this embodiment, the block sparsity selection module 502 is further configured to: determine a block sparsity pattern table based on the target block; wherein the block sparsity pattern table records the target key vector block corresponding to each query vector block. The update module 503 is further configured to update the initial student model according to the difference between the first output and the second output, as follows, to obtain the target student model: determine a block sparse pattern table based on the target block; wherein the block sparse pattern table records the target key vector block corresponding to each query vector block; update the quantization parameters of the initial student model according to the block sparse pattern table; determine the second output based on the target block and the updated quantization parameters, and determine the target distillation feature loss according to the difference between the first output and the second output; in response to the determined target distillation feature loss not meeting the preset accuracy threshold, redetermine the target block from multiple blocks, and perform the step of determining the block sparse pattern table again based on the redetermined target block to redetermine the target distillation feature loss, until the target distillation feature loss meets the preset accuracy threshold, to obtain the target student model, and output the block sparse pattern table and the quantization parameter table; wherein the quantization parameter table includes the updated quantization parameters.

[0142] In some optional implementations of this embodiment, the update module 503 is further configured to update the quantization parameters of the initial student model according to the block sparse pattern table as follows: according to the block sparse pattern table, skip blocks other than the target block among multiple blocks, and update the quantization parameters of the target block.

[0143] In some optional implementations of this embodiment, the update module 503 is configured to determine the target distillation feature loss based on the Euclidean distance between the first output and the second output.

[0144] In some optional implementations of this embodiment, the model distillation apparatus 500 may further include a data preparation module (not shown) configured to determine the initial values ​​of the quantization parameters based on statistics or clustering of the calibration set.

[0145] In this embodiment, more specific processing and effects of the model determination module 501, block sparsity selection module 502, update module 503, and data preparation module of the model distillation apparatus 500 can be found in [reference needed]. Figure 2 or Figure 3 The relevant descriptions in the aforementioned method embodiments are not repeated here.

[0146] Continue to refer to Figure 6 This disclosure provides an embodiment of a reasoning model. This reasoning model can be a target student model obtained according to the model distillation method of the foregoing embodiments. For example... Figure 6 As shown, the inference model 600 may include an attention operation module 601 and a content generation module 602. The attention operation module 601 is configured to perform attention operation only on the target blocks in the input sequence according to the input sequence of the attention layer to obtain sparse attention output results. The content generation module 602 is configured to generate target content based on the sparse attention output results. The target content may include images or videos.

[0147] In some optional implementations of this embodiment, the inference model 600 further includes a quantization module (not shown), which is configured to perform block quantization on the input sequence according to the quantization parameters in a determined quantization parameter table; wherein the block size of the block quantization is consistent with the block size of the input sequence. The attention operation module 601 is further configured to: determine the target key vector block corresponding to each query vector block of the query vector sequence for each attention head in each attention layer of the target student model according to a determined block sparsity pattern table; and, based on the block quantized input sequence, perform matrix multiplication and aggregation output on each query vector block and the corresponding target key vector block to obtain the sparse attention output result of each attention layer.

[0148] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0149] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0150] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in process 200, 300, or 400.

[0151] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in process 200, 300, or 400.

[0152] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0153] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0154] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0155] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the model distillation method. For example, in some embodiments, the model distillation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the model distillation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the model distillation method by any other suitable means (e.g., by means of firmware).

[0156] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. This disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this disclosure can be implemented, in whole or in part, by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. This application can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code. A computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to embodiments of this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model distillation method, characterized in that, include: Sample data, including at least one of text data and image data, is input into a pre-trained teacher model, and the first output of the attention layer of the teacher model is determined. The sample data is input into an initial student model, and the input sequence of the attention layer of the initial student model is divided into multiple blocks. Based on the distillation feature loss increment corresponding to each block, the most important block among the multiple blocks is selected as the target block. The initial student model employs a sparse attention operator. The distillation feature loss increment corresponding to each block is the difference between the first distillation feature loss between the output of the attention layer of the initial student model containing the block and the first output, and the second distillation feature loss between the output of the attention layer of the initial student model not containing the block and the first output. The importance of the block is proportional to the magnitude of the difference. The second output of the attention layer of the initial student model is determined based on the target block. The initial student model is updated based on the difference between the first output and the second output to obtain the target student model.

2. The method according to claim 1, characterized in that, The input sequence of the attention layer of the initial student model includes a query vector sequence and a key vector sequence; In the case where the initial student model includes multiple attention layers, the most important portion of the multiple blocks is selected as the target block based on the distillation feature loss increment corresponding to each block, in the following manner: For each attention head in each attention layer, a set of candidate key vector blocks is determined for each query vector block of the query vector sequence. For each candidate key vector block in the candidate key vector block set, the distillation feature loss increment corresponding to the candidate key vector block is determined based on the difference between the first distillation feature loss and the second distillation feature loss when the candidate key vector block is included and when it is not included. The target key vector block corresponding to the query vector block is determined from the set of candidate key vector blocks based on the distillation feature loss increment corresponding to each candidate key vector block, and is used as the target block.

3. The method according to claim 2, characterized in that, The step of determining the target key vector block corresponding to the query vector block from the set of candidate key vector blocks based on the distillation feature loss increment corresponding to each candidate key vector block includes: The importance score of each candidate key vector block is determined based on the distillation feature loss increment corresponding to each candidate key vector block. The candidate key vector blocks in the candidate key vector block set are sorted by importance score according to their importance scores. Based on the set number of sparse constraints or the full layer sparsity, the target key vector block corresponding to the query vector block is determined from the candidate key vector block set in descending order of importance.

4. The method according to claim 2 or 3, characterized in that, The step of updating the initial student model based on the difference between the first output and the second output to obtain the target student model includes: A block sparse pattern table is determined based on the target block; wherein, the block sparse pattern table records the target key vector block corresponding to each query vector block; The quantization parameters of the initial student model are updated according to the block sparse pattern table; The second output is determined based on the target block and the updated quantization parameters, and the target distillation feature loss is determined based on the difference between the first output and the second output. In response to the determined target distillation feature loss not meeting the preset accuracy threshold, a new target block is determined from the plurality of blocks, and the step of determining the block sparse pattern table is executed again based on the newly determined target block to redetermine the target distillation feature loss until the target distillation feature loss meets the preset accuracy threshold, thereby obtaining the target student model, and outputting the block sparse pattern table and the quantization parameter table; wherein, the quantization parameter table includes the updated quantization parameters.

5. The method according to claim 4, characterized in that, The step of updating the quantization parameters of the initial student model according to the block sparse pattern table includes: Based on the block sparse pattern table, skip blocks other than the target block among multiple blocks, and update the quantization parameters of the target block.

6. The method according to claim 4, characterized in that, The sample data comes from the calibration set; the initial value of the quantization parameter is determined based on the statistics or clustering of the calibration set.

7. The method according to claim 4, characterized in that, The step of determining the target distillation feature loss based on the difference between the first output and the second output includes: The target distillation feature loss is determined based on the Euclidean distance between the first output and the second output.

8. The method according to claim 1, characterized in that, The teacher model includes a converter model or a diffusion-converter model for generating images or videos; the initial student model is obtained based on the teacher model.

9. A model reasoning method, characterized in that, include: The data to be processed is input into the target student model; wherein the target student model is obtained by the model distillation method according to any one of claims 1-8; Based on the input sequence of the attention layer of the target student model, attention operations are performed only on the target blocks in the input sequence to obtain sparse attention output results; The target content is generated based on the sparse attention output; wherein, the target content includes images or videos.

10. The method according to claim 9, characterized in that, The step of performing attention operations only on the target block in the input sequence to obtain a sparse attention output result includes: Based on the determined block sparse pattern table, for each attention head in each attention layer of the target student model, determine the target key vector block corresponding to each query vector block of the query vector sequence; The input sequence is divided into blocks for quantization according to the quantization parameters in the determined quantization parameter table; wherein the block size of the block quantization is the same as the block size of the input sequence. Based on the input sequence after block quantization, matrix multiplication and aggregation are performed on each block of query vector and the corresponding block of target key vector to obtain the sparse attention output result of each attention layer.

11. A model distillation apparatus, characterized in that, include: The model determination module is configured to determine the initial student model based on a pre-trained teacher model, wherein the teacher model includes an attention layer; A block sparse selection module is configured to input sample data, including at least one of text data and image data, into the teacher model and determine a first output of the attention layer of the teacher model; and input the sample data into the initial student model and divide the input sequence of the attention layer of the initial student model into multiple blocks, and select the most important part of the multiple blocks as a target block according to the distillation feature loss increment corresponding to each block, wherein the initial student model employs a sparse attention operator, and wherein the distillation feature loss increment corresponding to each block is the difference between a first distillation feature loss between the output of the attention layer of the initial student model containing the block and the first output, and a second distillation feature loss between the output of the attention layer of the initial student model not containing the block and the first output, the importance of the block being proportional to the magnitude of the difference; and determine a second output of the attention layer of the initial student model based on the target block; The update module is configured to update the initial student model based on the difference between the first output and the second output to obtain the target student model.

12. A reasoning model, characterized in that, The inference model is the target student model obtained by the model distillation method according to any one of claims 1-8, and the inference model includes: The attention operation module is configured to perform attention operations only on the target blocks in the input sequence based on the input sequence of the attention layer, so as to obtain sparse attention output results. The content generation module is configured to generate target content based on the sparse attention output, wherein the target content includes images or videos.

13. The reasoning model according to claim 12, characterized in that, Also includes: The quantization module is configured to perform block quantization on the input sequence according to the quantization parameters in a determined quantization parameter table; wherein the block size of the block quantization is the same as the block size of the input sequence. The attention operation module is further configured to, based on a determined block sparse pattern table, determine the target key vector block corresponding to each query vector block of the query vector sequence for each attention head in each attention layer of the inference model; and, based on the block-quantized input sequence, perform matrix multiplication and aggregation output on each query vector block and the corresponding target key vector block to obtain the sparse attention output result of each attention layer.

14. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more computer programs are stored, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the model distillation method as described in any one of claims 1-8, or the model inference method as described in claim 9 or 10.

15. A computer-readable storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the model distillation method as described in any one of claims 1-8, or the model inference method as described in claim 9 or 10.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the model distillation method as described in any one of claims 1-8, or the model inference method as described in claim 9 or 10.

Images