Attention structure compilation optimization method, device, equipment, medium and program

Abstract

The embodiment of the application discloses a kind of attention structure's compilation optimization method, device, equipment, medium and program, method includes: obtaining the attention event monitoring data generated by real-time monitoring of current running model by performance monitoring unit, and according to attention event monitoring data generation attention execution configuration file;Performance monitoring unit includes event sampling module, event sampling module includes a group of first register, each first register is used to record and store attention event monitoring data in real time;According to attention execution configuration file, the parallel optimization branch of current attention structure branch in current running model is generated;According to the branch flow data of the parallel optimization branch of current attention structure branch, the pre-fetching processing of current attention structure branch is carried out.The technical scheme of the embodiment of the application can improve the efficiency of instruction-level parallel and pipeline of attention structure, and then improve the hardware resource utilization rate of the execution of attention structure in model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of chip and artificial intelligence technology, and in particular to a compilation optimization method, apparatus, electronic device, storage medium and program for attention structure. Background Technology

[0002] With the continuous evolution of artificial intelligence technologies such as Large Language Model (LLM), they have been widely applied in various industries.

[0003] Currently, attention structures are a core component in large-scale model inference. The attention structure within a network model can mimic the human cognitive approach of selectively focusing on important information. When processing sequences (such as text), it does not treat all elements equally, but dynamically assigns different weights to different parts based on the context. Attention structures include dense attention and sparse attention. Dense attention calculates attention for each query and all keys in the sequence, forming a complete attention matrix, with a complexity of O(n^2). 2 ), where n is the sequence length. Sparse attention only calculates attention for some key query-key pairs, while the remaining positions are masked or skipped. The attention matrix is ​​sparse, and the complexity can be reduced to O(n) or O(n log n). When the context length of a large model exceeds 128K or even higher, the sparse attention architecture becomes the key to balancing model performance and effectiveness.

[0004] Dense attention structures suffer from poor runtime performance due to the need to compute the complete attention matrix. If sparsity processing is only applied at the algorithm level without local runtime performance optimization, issues such as random memory access, discontinuous memory access, fragmented key-value (KV) caches, incomplete computational pipelines, and idle computing power on the device side can arise, potentially making them slower and more memory-intensive than ordinary dense attention structures. Summary of the Invention

[0005] This invention provides a compilation optimization method, apparatus, electronic device, storage medium, and program for attention structures, which can reduce the latency of attention structure inference runtime in a model, improve the efficiency of instruction-level parallelism and pipeline of attention structures, and thus improve the hardware resource utilization of attention structure execution in a model.

[0006] According to one aspect of the present invention, a compilation optimization method for attention structures is provided, comprising: The system acquires attention event monitoring data generated in real time by the performance monitoring unit for the currently running model, and generates an attention execution configuration file based on the attention event monitoring data. The performance monitoring unit includes an event sampling module, which includes a set of first registers, each of which is used to record and store the attention event monitoring data in real time. Generate a parallel optimization branch for the current attention structure branch in the currently running model based on the attention execution configuration file; The branch flow data of the current attention structure branch is prefetched based on the parallel optimization branch of the current attention structure branch.

[0007] According to another aspect of the present invention, a compiler optimization apparatus for attention structures is provided, comprising: The attention execution configuration file generation module is used to obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the attention event monitoring data in real time; The parallel optimization branch generation module is used to generate a parallel optimization branch for the current attention structure branch in the currently running model based on the attention execution configuration file. The branch process data prefetching module is used to prefetch the branch process data of the current attention structure branch according to the parallel optimization branch of the current attention structure branch.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the compilation optimization method for the attention structure described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the compilation optimization method of the attention structure described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the compilation optimization method for the attention structure described in any embodiment of the present invention.

[0011] This invention, through an embodiment of the present invention, acquires attention event monitoring data generated in real-time by a performance monitoring unit monitoring the current running model. Based on this data, an attention execution configuration file is generated, which in turn generates a parallel optimization branch for the current attention structure branch within the current running model. Furthermore, the branch flow data of the current attention structure branch is prefetched based on this parallel optimization branch. The performance monitoring unit configures an event sampling module using a set of first registers. Each first register within this module is used to record and store attention event monitoring data in real-time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire attention event-related data in real-time, assisting in adjusting and optimizing the prefetching operation of branch flow data within the attention structure network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient compilation optimization strategies for the attention structure network. This significantly reduces memory access latency of the attention structure, thereby reducing runtime latency during attention structure inference, improving the efficiency of instruction-level parallelism and pipelines, and ultimately increasing the hardware resource utilization of attention structure execution.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a compilation optimization method for attention structures provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the internal structure of an LLM provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the interaction between a performance monitoring unit and the software layer according to Embodiment 1 of the present invention; Figure 4 This is a flowchart of a compilation optimization method for attention structures provided in Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of a process for prefetching KVCache data of a sparse attention structure branch in parallel optimization, as provided in Embodiment 2 of the present invention. Figure 6 This is a flowchart of a compilation optimization method for a sparse attention structure provided in Embodiment 3 of the present invention; Figure 7 This is a schematic diagram of the branch compilation optimization of a sparse attention structure in the convergence barrier mode provided in Embodiment 3 of the present invention; Figure 8 This is a schematic diagram of the branch compilation optimization of a sparse attention structure in a no-convergence barrier mode provided in Embodiment 3 of the present invention; Figure 9 This is a schematic diagram of a high-frequency execution branch instruction prefetching process inside a sparse attention structure provided in Embodiment 3 of the present invention; Figure 10 This is a schematic diagram of a compiler optimization device for an attention structure provided in Embodiment 4 of the present invention; Figure 11 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1This is a flowchart of a compilation optimization method for attention structures provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where attention structures are compiled and optimized using a software-hardware co-processing approach. This method can be executed by an attention structure compilation optimization device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a terminal device or a server device, as long as it can execute the attention structure compilation optimization method. The present invention does not limit the specific type of electronic device. Correspondingly, as... Figure 1 As shown, the method includes the following operations: S110. Obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data; wherein, the performance monitoring unit includes an event sampling module, the event sampling module includes a set of first registers, each of the first registers being used to record and store the attention event monitoring data in real time.

[0018] The Performance Monitoring Unit (PMU) can be a hardware module running within a device unit in a parallel computing context. This device unit can be a processor, for example, including but not limited to a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), or NPU (Neural Processing Unit), as long as it can work collaboratively with the host CPU (Central Processing Unit). This embodiment of the invention does not limit the specific type of device unit that works collaboratively with the host CPU. The currently running model can be the model currently running in the device unit (or device side), such as a large model, various machine learning models, or other types of network models. This embodiment of the invention does not limit the model type of the currently running model. The attention event monitoring data can be data obtained by the Performance Monitoring Unit monitoring the attention behavior of the attention structure in the currently running model. Optionally, the attention structure can be a dense attention structure or a sparse attention structure. This embodiment of the invention does not limit the specific type of attention structure. The attention execution configuration file can be a configuration file used as a reference for compiling and optimizing the attention structure. The event sampling module can be a hardware module used to collect information about the dynamic behavior of the attention structure. The first register can be a register that constitutes the event sampling module.

[0019] Large language models, in particular, refer to deep learning models trained with massive amounts of relevant data (such as text, speech, or combined text and image data) capable of processing sequences (such as text, speech, images, audio / video, and combined text and image sequences). These models can generate natural language text or understand the meaning of spoken text, and typically have billions of parameters. Large language models can handle various natural language tasks, such as text classification, translation, question answering, text-image analysis, and dialogue, and have wide applications. The input to a large language model is data, such as text, speech, or combined text and image data. The large language model encodes the input data to obtain corresponding word vectors, and then decodes these word vectors to automatically process the input data and obtain the corresponding output data. For example, text can be input into a large language model, which then processes and predicts the input text, outputting the corresponding response text. Therefore, large models can perform a wide range of tasks, including but not limited to text summarization, translation, and sentiment analysis.

[0020] For large models or other network models, attention structures are one of their core components, used to dynamically allocate computational resources and focus on the most relevant information in the input, thereby improving the model's ability to model long-distance dependencies and complex semantic relationships. Dense attention structures are widely used in Transformer models, large models, recurrent neural networks, convolutional neural networks, graph neural networks, and generative adversarial networks. Sparse attention structures are widely used in large models, hybrid expert models, brain-like / efficient reasoning models, and hardware co-optimization models.

[0021] Figure 2 This is a schematic diagram of the internal structure of an LLM provided in Embodiment 1 of the present invention. In a specific example, such as Figure 2 As shown, dense attention structures such as multi-head attention and sparse attention structures are typically applied to the computation graph of the attention layer. Dense attention structures perform fully connected computations on the relationships between all tokens in the sequence, meaning each token calculates its attention weight with every other token in the sequence, thus classifying it as dense attention. This design comprehensively captures multi-dimensional dependencies, including syntactic, semantic, and logical ones. Sparse attention structures, as the core sparse architecture of the model, control which tokens (lexical units) can communicate with each other through masks, routing, or gating mechanisms, belonging to the underlying information flow topology of the model.

[0022] The performance monitoring unit, as a hardware module within a device unit, is dedicated to monitoring, statistically analyzing, and sampling performance events of the processor or system for hardware-level performance analysis. Its core functions include monitoring microarchitectural events such as instruction execution, cache access, branch prediction, and TLB (Translation Lookaside Buffer) hits / misses. It also provides capabilities such as cycle counting, event counting, and sampling analysis, supporting fine-grained filtering, such as data filtering mechanisms based on privilege level or security domain. In this embodiment, the performance monitoring unit may be configured with an event sampling module, which can consist of a set of first registers. Each first register can be used to record and store attention event monitoring data obtained by sampling the dynamic behavior of the attention structure in real time. For example, attention event monitoring data may include, but is not limited to, index events of the indexer and Gather events of the KV Cache (Key-Value Cache). (The Gather operation refers to retrieving data from memory to the computing unit.) The index event of the indexer can include, for example, the index events of the Top K token KV Cache inside the sparse attention structure, which is applicable to sparse attention structures; the Gather event of the KV Cache is applicable to both sparse and dense attention structures.

[0023] Optionally, the event sampling module can be configured to count specific microarchitectural events. Count types may include, but are not limited to, the number of executed instructions, cache misses, branch prediction errors, and floating-point operations. This is accomplished by setting a monitoring counter as the first register and triggering recording. The event sampling module configures monitoring parameters, specifying the events to be monitored and the sampling period. For example, monitoring the KV Cache Gather event and setting the counter overflow value to 500. When the number of monitored events reaches the threshold, the counter overflows. The event sampling module automatically writes the precise machine state to a specific memory area, a hardware-driven process that ensures a precise correspondence between the record and the triggering instruction.

[0024] In this embodiment of the invention, the device-side performance monitoring unit is responsible for real-time monitoring of the operation of the attention structure of the current running model, generating attention event monitoring data based on the real-time monitoring results, and then sending the attention event monitoring data generated by the real-time monitoring to the host side, such as the CPU side. The host side can receive the attention event monitoring data generated by the performance monitoring unit for real-time monitoring of the current running model, and perform in-depth analysis based on the received attention event monitoring data. Optionally, the host side can export the attention event monitoring data to memory, and generate a software-compilable executable attention execution configuration file from the exported attention event monitoring data.

[0025] S120. Generate a parallel optimization branch of the current attention structure branch in the currently running model according to the attention execution configuration file.

[0026] The current attention structure branch can be the original code execution path of the attention structure, and can be treated as an independent execution branch. The parallel optimization branch can be another independent execution branch created from the current attention structure branch, used to optimize the code instructions of the current attention structure branch in a parallel manner.

[0027] S130. Perform prefetching processing on the branch flow data of the current attention structure branch according to the parallel optimization branch of the current attention structure branch.

[0028] The branch process data can be the data processed within the current attention structure branch. For example, it could be token data.

[0029] After the performance monitoring unit on the device side generates attention event monitoring data by monitoring the operation of the attention structure of the current running model in real time, and generates an attention execution configuration file on the host side based on the attention event monitoring data, the host side can parse the attention execution configuration file through the compiler. Based on the parsing results, a parallel optimization branch is constructed to run in parallel with the current attention structure branch. This parallel optimization branch is used to predict the type of the current attention structure and the subsequent data loading or processing of the current running model. Specifically, the parallel optimization branch can predict the specific content of the branch process data that the current attention structure branch may process, thereby prefetching the predicted branch process data. The core of prefetching is to improve system efficiency by predicting future behavior and performing resource loading, instruction fetching, or data preparation in advance.

[0030] Optionally, parallel optimization branches can predict and prefetch data for the next branch process based on the historical attention behavior, access patterns, or control flow features of the current attention structure branch. By prefetching the data that the attention structure branch may process during runtime, compilation optimization of the attention structure branch can be achieved, improving the cache hit rate of the attention structure branch, significantly reducing the latency of memory access to the attention structure, thereby improving the performance of model locality and hiding memory latency.

[0031] Figure 3 This is a schematic diagram illustrating the interaction between a performance monitoring unit and the software layer according to Embodiment 1 of the present invention. In a specific example, such as... Figure 3 As shown, the host side can receive sampling data (attention event monitoring data) sent by the performance monitoring unit through a performance sampling tool, and generate an attention execution configuration file based on the received attention event monitoring data. The compiler running on the host side can perform compilation optimization through the attention execution configuration file, such as creating a parallel optimization branch for the current attention structure branch of the currently running model, and prefetching the branch flow data of the current attention structure branch through the parallel optimization branch to achieve compilation optimization of the current attention structure branch. The optimized attention structure code is processed by the compiler and finally transformed into a new kernel file that can be executed on the device side. The new kernel file is the kernel code file after the attention structure branch is optimized. In this way, when the device side runs the model based on the newly compiled and optimized kernel file, the currently running model can preload the branch flow data that needs to be processed by the current attention structure branch during the execution of the current attention structure branch. It can be seen that the embodiments of the present invention can significantly reduce the memory access latency of the attention structure by optimizing the execution mode of the attention structure code of the model, thereby reducing the latency of the attention structure inference runtime, improving the efficiency of instruction-level parallelism and pipeline of the attention structure, and thus improving the hardware resource utilization of the attention structure execution.

[0032] Understandably, optimizing the local performance of dense attention structures can significantly improve processor computational and inference efficiency while maintaining high accuracy, and reduce memory usage. Optimizing the local performance of sparse attention structures, in addition to significantly improving processor computational and inference efficiency and reducing memory usage, can also translate the theoretical advantages of algorithm sparsity into real-world throughput, latency, and memory savings, directly determining model training convergence efficiency, long text inference throughput, and model deployment costs.

[0033] This invention, through an embodiment of the present invention, acquires attention event monitoring data generated in real-time by a performance monitoring unit monitoring the current running model. Based on this data, an attention execution configuration file is generated, which in turn generates a parallel optimization branch for the current attention structure branch within the current running model. Furthermore, the branch flow data of the current attention structure branch is prefetched based on this parallel optimization branch. The performance monitoring unit configures an event sampling module using a set of first registers. Each first register within this module is used to record and store attention event monitoring data in real-time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire attention event-related data in real-time, assisting in adjusting and optimizing the prefetching operation of branch flow data within the attention structure network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient compilation optimization strategies for the attention structure network. This significantly reduces memory access latency of the attention structure, thereby reducing runtime latency during attention structure inference, improving the efficiency of instruction-level parallelism and pipelines, and ultimately increasing the hardware resource utilization of attention structure execution.

[0034] Example 2 Figure 4 This is a flowchart of a compilation optimization method for attention structures provided in Embodiment 2 of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, various specific optional implementation methods are given for generating parallel optimization branches of the current attention structure branches in the current running model and for prefetching the branch process data of the current attention structure branches. Correspondingly, as Figure 4 As shown, the method in this embodiment may include: S410. Obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data.

[0035] The performance monitoring unit includes an event sampling module, which includes a set of first registers, each of which is used to record and store the attention event monitoring data in real time.

[0036] S420. Determine the current Token query vector currently being processed by the current attention structure branch based on the attention execution configuration file.

[0037] The current token query vector can be the query vector corresponding to the current token.

[0038] During the processing of the large model, each input token undergoes a linear transformation to generate three vectors: Query (Q), Key (K), and Value (V). Here, Q represents what the current token is "looking for," i.e., what information the current token hopes to obtain from other tokens; K represents what other tokens "can provide," which are their exposed features; and V represents the "actual information content" of other tokens, which is the data source for the final aggregated output. In other words, each generated token corresponds to a set of K / V vectors, which can be stored in a pre-allocated cache pool. The indices of the KV cache are the offset addresses or slot IDs in this cache pool.

[0039] S430. Generate a parallel optimization branch for the current token query vector currently being processed by the current attention structure branch.

[0040] Understandably, both dense and sparse attention structures require calculating attention between the query vector and the key values ​​in the sequence. Therefore, for the current attention structure, the process of processing the attention for each token query vector can be considered as a branch of the current attention structure. Correspondingly, a parallel optimization branch can be generated for each attention structure branch that processes each token query vector. That is, each parallel optimization branch can prefetch the branch process data involved in the attention process of processing each token query vector within the attention structure.

[0041] Specifically, if the current attention structure branch is a dense attention structure branch, since dense attention structures need to calculate all key-value (KV) data, and the amount of KV data is large, the parallel optimization branch of the current attention structure branch can monitor the capacity of the cached data in real time according to the set KV data caching strategy. For example, in the initial stage, if the cached data is 0, the parallel optimization branch can prefetch a set number of KV data from the memory data and move them to the cache for the dense attention structure to perform attention calculations. During the attention calculation process of the dense attention structure, the parallel optimization branch can monitor the remaining capacity of the KV data in the cache in real time, and when it is determined that the remaining capacity of the KV data is less than a set threshold, it continues to prefetch a set number of KV data from the memory data according to the mark of the previous KV data prefetching and move them to the cache for the dense attention structure to perform attention calculations, until the attention calculation process of all KV data is completed.

[0042] Specifically, if the current attention structure branch is a sparse attention structure branch, since the sparse attention structure only requires a portion of the key-value data and the amount of key-value data is small, the parallel optimization branch of the current attention structure branch can prefetch all the key-value data that needs to be calculated into the cache at once, so that the sparse attention structure can perform attention calculation.

[0043] Optionally, if the current attention structure branch is a sparse attention structure branch, the above method may further include the following operations: S440. Generate the historical co-occurrence information of the current token query vector based on the attention execution configuration file.

[0044] Among them, the token historical co-occurrence information can be the information of the tokens corresponding to the KV Cache indices selected by the token query vector during the historical computation attention process of the sparse attention structure.

[0045] Since the attention execution profile can monitor all attention events and obtain attention event monitoring data, the parallel optimization branch can analyze the co-occurring tokens selected by the current token query vector during the historical attention computation process of the sparse attention structure based on the attention event monitoring data included in the attention execution profile, thereby obtaining the token historical co-occurrence information of the current token query vector. It is understandable that for different tokens, the token historical co-occurrence information corresponding to the token query vector may be partially or completely the same, or it may be different.

[0046] For example, direct statistical methods can be used to compile the historical co-occurrence information of each token query vector. Alternatively, other models can be used to predict which token key-value caches a given token query vector in a certain transformer layer might select, thereby obtaining the historical co-occurrence information of the token query vector.

[0047] In an optional embodiment of the present invention, generating the historical co-occurrence information of the current token query vector based on the attention execution configuration file may include: determining the historical loaded KV Cache of the historical token query vector based on the attention execution configuration file; counting token co-occurrence pairs based on the historical loaded KV Cache of the historical token query vector; filtering target token co-occurrence pairs from the token co-occurrence pairs whose co-occurrence frequency exceeds a set frequency threshold, and generating a token co-occurrence table based on the target token co-occurrence pairs; and matching the current token query vector with the token co-occurrence table to obtain the historical co-occurrence information of the current token query vector.

[0048] The historical KV Cache can be the KV Cache corresponding to the tokens historically selected and hit by the current Token query vector. A token co-occurrence pair can be a token pair consisting of the current Token and its co-occurring tokens. Co-occurring tokens are those historically selected and hit by the current Token query vector. Co-occurrence frequency can be the frequency at which the current Token hits co-occurring tokens. A target token co-occurrence pair can be a token co-occurrence pair with a co-occurrence frequency greater than a certain threshold. The frequency threshold can be configured according to actual needs; this embodiment of the invention does not limit the specific value of the frequency threshold.

[0049] Specifically, the KV Cache loaded into the historical attention behavior of the sparse attention structure can be determined based on the attention execution configuration file; this is known as the historical loaded KV Cache. Further, the historical loaded KV Cache of the historical token query vectors can be used to statistically analyze the token co-occurrence pairs formed by each historical token query vector and its corresponding co-occurring tokens. Then, target token co-occurrence pairs with a co-occurrence frequency exceeding a set frequency threshold can be selected from these pairs, and a token co-occurrence table can be generated based on these target pairs. Thus, the token co-occurrence table records information about the tokens most likely to be hit by the historical token query vector. The token co-occurrence tables generated by different corresponding historical token query vectors may be the same or different. Based on this, the token co-occurrence table corresponding to the current token query vector can be determined, and by matching the current token query vector with its corresponding token co-occurrence table, the more important historical token co-occurrence information of the current token query vector can be obtained. That is, the historical token co-occurrence information is not the complete token co-occurrence information of the current token query vector, which can reduce the computational load of the attention behavior and improve computational efficiency.

[0050] S450. Calculate the KV Cache prefetch address of the associated co-occurring tokens of the current token query vector based on the token historical co-occurrence information of the current token query vector.

[0051] S460. Prefetch the predicted KV Cache data of the current Token query vector according to the KV Cache prefetch address of the associated co-occurring Tokens of the current Token query vector.

[0052] Among them, the associated co-occurring token can be the token that the current token query vector may hit. The predicted KVCache data can be the KV Cache data that needs to be selected based on the prediction of the current token query vector.

[0053] Optionally, the historical co-occurrence information of tokens can be sorted in descending order of hit count to reflect the co-occurrence frequency with the current token query vector. Further, the associated co-occurring tokens of the current token query vector can be calculated based on the historical co-occurrence information of the current token query vector. For example, the top-ranked co-occurring tokens in the historical co-occurrence information can be used as the associated co-occurring tokens of the current token query vector. Alternatively, all co-occurring tokens in the historical co-occurrence information can be used as the associated co-occurring tokens of the current token query vector. This embodiment of the invention does not limit the method for determining the associated co-occurring tokens. After determining the associated co-occurring tokens, the KV Cache of each associated co-occurring token can be used as the branch process data to be prefetched. The KV Cache storage address of the associated co-occurring tokens of the current token query vector is then calculated as the KV Cache prefetch address. Finally, based on the KV Cache prefetch address of the associated co-occurring tokens of the current token query vector, the KV Cache of the corresponding associated co-occurring token is prefetched as the predicted KV Cache data for the current token query vector.

[0054] In an optional embodiment of the present invention, the step of prefetching the predicted KV Cache data of the current Token query vector according to the KV Cache prefetch address of the associated co-occurring tokens of the current Token query vector may include: initiating a prefetch instruction according to the KV Cache prefetch address of the associated co-occurring tokens of the current Token query vector; and, in response to the prefetch instruction, preloading the predicted KV Cache data of the current Token query vector into caches of different levels according to model architecture parameters and hardware architecture parameters.

[0055] Optionally, the prefetch instruction can be a data prefetch instruction, used to prefetch data and store it in a cache.

[0056] Specifically, after determining the predicted KV Cache data for the current Token query vector, a prefetch instruction can be initiated based on the KV Cache prefetch address where the predicted KV Cache data for the current Token query vector resides. Correspondingly, the host side can respond to the prefetch instruction by preloading the predicted KV Cache data for the current Token query vector from main memory into different cache levels based on model architecture parameters and hardware architecture parameters. It is understandable that different model architecture parameters and hardware architecture parameters will result in different cache levels where the predicted KV Cache data is loaded.

[0057] In an optional embodiment of the present invention, the step of preloading the predicted KV Cache data of the current Token query vector into different levels of cache according to model architecture parameters and hardware architecture parameters may include: calculating the cache capacity of the predicted KV Cache data; preloading the predicted KV Cache data into the first-level cache when the cache capacity of the predicted KV Cache data is less than or equal to the first-level cache capacity; preloading the predicted KV Cache data into the second-level cache when the cache capacity of the predicted KV Cache data is greater than the first-level cache capacity and less than or equal to the second-level cache capacity; and preloading the predicted KV Cache data into the third-level cache when the cache capacity of the predicted KV Cache data is greater than the second-level cache capacity.

[0058] The first-level cache is the L1 cache. The second-level cache is the L2 cache. The third-level cache is the L3 cache.

[0059] Understandably, the model architecture parameters determine the size of the KV Cache. The hardware architecture parameters, in turn, define the size of each cache level. Therefore, when preloading the predicted KV Cache data of the current Token query vector into different cache levels based on the model and hardware architecture parameters, the capacity of the predicted KV Cache data can be calculated first. The size of the KV Cache data is related to both the model structure and the data type. Optionally, the cache capacity of the predicted KV Cache data is bsh. BPE. Here, b is the batch size, s is the sequence length, h is the hidden size (number of neurons or vector dimension in the hidden layer), and BPE stands for Bytes Per Element, representing the number of bytes occupied by each data element in programming or data processing. The length of the KVCache to be prefetched, i.e., s, can be obtained from the Token co-occurrence table.

[0060] In multi-level caching, L1 / L2 level caches are suitable for frequently accessed and performance-critical high-frequency tokens; L3 level cache is suitable as a buffer before prefetching. Specifically, we can first determine whether the cache size of the predicted KV Cache data (KVCache Size) is less than or equal to the capacity of the L1 level cache; if so, the predicted KV Cache data is preloaded into the L1 level cache; otherwise, we determine whether the cache size of the predicted KV Cache data is greater than the L1 level cache capacity and less than or equal to the L2 level cache capacity. If so, the predicted KV Cache data is preloaded into the L2 level cache; otherwise, if the cache size of the predicted KV Cache data is greater than the L2 level cache capacity, the predicted KV Cache data is preloaded into the L3 level cache.

[0061] Figure 5 This is a schematic diagram illustrating the process of prefetching KVCache data for a sparse attention structure branch in parallel optimization, as provided in Embodiment 2 of the present invention. In a specific example, such as... Figure 5 As shown, in a sparse attention structure, the Top-K selection and indexer can be adjusted through parallel optimization branches. Specifically, KV cache data is predicted and loaded in advance based on attention event monitoring data from the hardware performance monitoring unit, thereby transforming the random gathering process of KV cache data into a cache hit process, significantly reducing memory access latency. Figure 5 As shown, the process of prefetching KV Cache data of the sparse attention structure branch through parallel optimization is as follows: a) Upon detecting a query for the current Token Q, a parallel optimization branch is generated for the current Token Q. Figure 5 The right branch in the middle), and run the left and right branches at the same time. The left branch is the path branch of steps b~e, and the right branch is the path branch of steps f~j. The left branch includes the following specific instruction operations: b) FP8 (8-bit Floating Point) matrix multiplication calculation QK T Relevance score; c) ReLU (Rectified Linear Unit) activation can be achieved using the MAX (Maximum) instruction; d) Select the KV Cache indices for the Top K tokens; e) Output the indices tensor (usually referring to an integer type tensor used for indexing operations or identifying a specific position).

[0062] It is understood that the instruction operations included in the above left branch are conventional attention behavior operations of sparse attention, and the details therein will not be described further in this embodiment of the invention.

[0063] The instructions and operations included in the right branch are as follows: f) Find the co-occurrence table of the tokens corresponding to the current token Q based on the current token ID; g) Predict the set of tokens selected to co-occur with the current token based on the token co-occurrence table; h) Calculate the KV Cache address for each Token in the predicted Token set; i) Initiate a data prefetch command; j) Preload the KV cache into the L1 / L2 / L3 cache.

[0064] Correspondingly, such as Figure 5 As shown, after the left and right branches have been executed, the following operations will continue: k) Merge indices with prefetch operations; The indices tensor output by the left branch is the index of the KVCache obtained from the actual operation of the model's attention structure. If the indices output by the left branch have been prefetched into the cache through the right branch, the cached indices can be read directly from the cache; if some are not prefetched into the cache, they can be read from memory.

[0065] l) The Gather operation is performed, and the prefetched KV Cache enjoys a low-latency hit processing process; The Gather operation refers to retrieving data from memory to a computing unit, which can be done at the byte level. However, due to the non-contiguous nature of the data addresses, it is usually inefficient.

[0066] m), perform attention calculations on K KV values, and iterate through them; n) Output the final attention result.

[0067] The above technical solution monitors the dynamic behavior of the attention structure through hardware, and the compiler creates parallel optimization branches based on this dynamic behavior. For sparse attention structure types, the parallel optimization branches can generate historical co-occurrence information of tokens in the token query vector based on the attention execution configuration file. Then, based on this information, the key-value cache (KV Cache) prefetch address of the associated co-occurring tokens in the token query vector is calculated. This allows for the prefetching of predicted KV Cache data of the token query vector based on these prefetch addresses, thus optimizing the compilation of the attention structure. This technical solution employs a hardware-software co-design, combining hardware monitoring data with efficient compilation optimization strategies for the attention structure network. This significantly reduces memory access latency, thereby reducing runtime latency during inference, improving instruction-level parallelism and pipeline efficiency, and ultimately increasing hardware resource utilization during attention structure execution.

[0068] Example 3 For network models, branches can be understood as different paths executed during runtime. Large models often encounter branching scenarios during training or inference, exhibiting dynamic branching behavior. Some large models employ novel Attention Residuals architectures, integrating attention mechanisms into the residual structure and introducing more branching path choices.

[0069] Meanwhile, for sparse attention structures, the kernel performs numerous comparison, sorting, and selection operations when retrieving the corresponding key-value information for the current token query vector. The processing path of each thread varies greatly depending on the data, and the preset K value of sparse attention also plays a role. All these factors contribute to severe warp divergence within the attention structure. Furthermore, the threshold-based dynamic selection strategy of sparse attention structures, such as retaining only tokens with a probability exceeding 0.9 after softmax (normalization), also introduces implicit branching. Different threads within a warp may choose to retain different numbers of tokens due to different data, leading to divergent execution paths.

[0070] Branch divergence leads to inefficiencies in the SIMT (Single Instruction Multiple Threads) pipeline, thus affecting the overall model runtime. Furthermore, severe branch divergence in the model's branching behavior will significantly slow down and reduce overall model performance.

[0071] Figure 6This is a flowchart of a compilation optimization method for a sparse attention structure provided in Embodiment 3 of the present invention. This embodiment is based on the above embodiment and is further specified. In this embodiment, various specific implementation methods for compiling and optimizing the internal branch structure of the sparse attention structure network are given. Accordingly, as... Figure 6 As shown, the method in this embodiment may include: S610. Obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data.

[0072] The performance monitoring unit includes an event sampling module, which includes a set of first registers, each of which is used to record and store the attention event monitoring data in real time.

[0073] S620. Generate a parallel optimization branch of the current attention structure branch in the currently running model according to the attention execution configuration file.

[0074] Optionally, the type of the current attention structure branch in this embodiment of the invention can be a sparse attention structure branch. A sparse attention structure branch is a branch path formed by a sparse attention structure.

[0075] S630. Perform prefetching processing on the branch flow data of the current attention structure branch according to the parallel optimization branch of the current attention structure branch.

[0076] S640. During the operation of the current attention structure branch, obtain the model branch monitoring data generated by the performance monitoring unit in real time monitoring the current attention structure branch.

[0077] The performance monitoring unit further includes a branch recording module, which includes a set of second registers. Each second register uses a circular buffer to record and store the model branch monitoring data in real time.

[0078] The model branch monitoring data can be data obtained by the performance monitoring unit monitoring the branch status of the currently running model. The branch recording module can be a hardware module used to generate model branch monitoring data in real time. The second register can be a register that constitutes the branch recording module.

[0079] Understandably, the implementation of an attention structure is essentially a trainable neural network module. This structure can be stacked in multiple layers to form a complete "attention network." During the execution of numerous comparison, sorting, and selection operations, sparse attention structures also encounter various branching scenarios. That is, sparse attention structures may execute different paths during operation. To further improve the performance of sparse attention structures and the overall model, the branching behavior within the sparse attention structure can be monitored in real time during its operation, allowing for compilation optimization of the branch code.

[0080] In embodiments of the present invention, such as Figure 3 As shown, the performance monitoring unit can also be configured with a branch recording module. This module can consist of a set of second registers, each using a circular buffer to record and store model branch monitoring data in real time. For example, model branch monitoring data may include, but is not limited to, the branch instruction source address, branch instruction target address, and additional information for each branch entry. The additional information may include, but is not limited to, the branch type and execution cycle. Branch types can be, for example, CALL (function call), RET (function return), B conditional jump, and unconditional jump. The circular buffer configuration within the branch recording module effectively avoids frequent memory allocation and deallocation operations, resulting in low data write time complexity and suitability for real-time recording of high-frequency events.

[0081] S650. Generate a branch execution configuration file based on the model branch monitoring data, and determine the high-frequency execution branches among the branches to be executed in the current attention structure branch based on the branch execution configuration file.

[0082] The branch execution configuration file can be a configuration file used as a reference for compiling and optimizing branches. Optionally, the attention network execution configuration file and the branch execution configuration file can be the same configuration file, that is, the attention execution configuration information and the branch execution configuration information can be stored in the same configuration file. Alternatively, the attention execution configuration file and the branch execution configuration file can be two different configuration files, and this embodiment of the invention does not limit this. The branch to be executed can be a code branch that needs to be executed. A high-frequency execution branch can be a target execution branch, which can be a code branch that is frequently executed during program execution. That is, a high-frequency execution branch can be a code branch with a high execution frequency. The execution frequency threshold of the high-frequency execution branch can be configured according to actual needs. That is, a code branch with an execution frequency greater than a set frequency threshold can be identified as a high-frequency execution branch.

[0083] Specifically, the host side can export model branch monitoring data into memory and generate a branch execution configuration file that can be compiled and executed by the software. Correspondingly, the host side can use the compiler to parse the branch execution configuration file to predict and judge the branch execution frequency of each branch to be executed within the current attention structure, thereby selecting the high-frequency execution branches with higher execution frequency from the branches to be executed.

[0084] S660. Adjust the instruction execution order of the high-frequency execution branch so that the branch instructions of the high-frequency execution branch are executed first.

[0085] Among them, the branch instructions of the high-frequency execution branch can be each instruction on the high-frequency execution branch path, and the branch instructions can also be called path instructions.

[0086] In a specific example, such as Figure 3 As shown, during the execution of the current attention structure branch (a sparse attention structure branch), the host side can receive sampled data (model branch monitoring data) sent by the performance monitoring unit after real-time monitoring of the internal operation of the current attention structure branch through a performance sampling tool. Based on the received model branch monitoring data, a branch execution configuration file is generated. The compiler running on the host side can perform compilation optimization using the branch execution configuration file, such as predicting and selecting high-frequency execution branches from each branch to be executed, and further adjusting the instruction execution order of the high-frequency execution branches. This allows priority execution of the branch instructions of the high-frequency execution branches within the current attention structure branch, thus achieving compilation optimization of the high-frequency execution branches within the sparse attention structure. The optimized branch code within the sparse attention structure is processed by the compiler and finally transformed into a new kernel file that can be executed on the device side. The new kernel file is the branch-optimized kernel code file. Therefore, when the device side runs the model based on the newly compiled and optimized kernel file, the instructions of the high-frequency execution branches within the current attention structure branch can be executed preferentially. Therefore, the embodiments of the present invention can shift from instruction branch prediction within the sparse attention structure network to data access pattern prediction and computation path inference, so as to optimize the layout of the running code of the sparse attention structure, reduce the dynamic branches within the sparse attention structure, eliminate SIMT divergence through vectorization, and achieve near-peak computing power utilization.

[0087] Currently, for network models, the device side typically manages warp branches through a SIMT stack, serializing branch execution using predicate registers and execution masks. The execution time is the sum of multiple branch paths, leading to low computational efficiency. For the device side, Diverge / Converge is a method for implementing thread-level branch management using masks and synchronization stacks. For example, using IPDom (Immediate Post Dominator) to implement convergence limits the stack depth to nested levels, typically 8-32 levels. While the Diverge / Converge mechanism allows the SIMT architecture to efficiently handle conditional branches, severe branch divergence can significantly degrade performance. During model deployment, operator scheduling strategies, memory access optimizations, and hardware adaptability are bottlenecks restricting inference throughput and latency.

[0088] The above technical solution proposes a hardware branch prediction method. This method combines hardware and software co-design to predict and execute SIMT during the operation of the sparse attention structure. By predicting and executing high-frequency branches within the attention structure through hardware and software collaboration, it can reduce or eliminate branches within the sparse attention structure. This avoids the latency caused by branches waiting to be parsed and re-fetching instructions after branch jumps, and can smooth out pipeline bubbles caused by branch jumps within the sparse attention structure. It can achieve efficient acceleration of the sparse attention structure operation process, thereby further reducing redundant operations and memory access power consumption, and optimizing processor energy efficiency while improving performance.

[0089] In an optional embodiment of the present invention, adjusting the instruction execution order of the high-frequency execution branch may include: determining the branch compilation execution mode based on the number of instruction entries in the high-frequency execution branch; and adjusting the instruction arrangement order of the high-frequency execution branch based on the type of the branch compilation execution mode.

[0090] After identifying the high-frequency execution branches (hereinafter referred to as high-frequency execution branches) within the current attention structure branch, the branch compilation execution mode can be determined based on the number of instruction entries in the high-frequency execution branches. Different branch compilation execution modes result in different methods of compiling and optimizing the branches. In this embodiment of the invention, optionally, the branch compilation execution mode may include two compilation execution modes: one with a Converge Barrier and one without a Converge Barrier.

[0091] Optionally, if the number of instruction entries in frequently executed branches is small, a convergence barrier-free mode without Converge Barrier can be selected; if the number of instruction entries in frequently executed branches is large, a convergence barrier mode with Converge Barrier can be selected. In the convergence barrier-free mode without Converge Barrier, there is no need to explicitly insert a convergence point. When an exit instruction is encountered, it checks if there are any unexecuted branches in the stack. If so, the unexecuted branch is popped and executed, achieving automatic divergence and convergence. The converged instructions need to be executed multiple times according to different branch masks. The convergence barrier mode with Converge Barrier explicitly defines the branch convergence point through independent convergence barriers and participation masks for each thread bundle, supporting independent thread scheduling and arbitrary depth of branch nesting.

[0092] Different branch compilation execution modes require different methods for adjusting and optimizing the instruction order of frequently executed branches. Regardless of the branch compilation execution mode used, corresponding compilation optimization strategies can be employed to adjust the instruction order of frequently executed branches, ensuring that branch instructions in these branches are executed first.

[0093] In an optional embodiment of the present invention, adjusting the instruction arrangement order of the high-frequency execution branches according to the type of the branch compilation execution mode may include: when the branch compilation execution mode is determined to be a convergence barrier mode, inserting a first convergence barrier instruction before each of the branches to be executed, inserting a second convergence barrier instruction before the original convergence instruction of each of the branches to be executed, and inserting an unconditional jump instruction at the end of the non-high-frequency execution branches; writing the currently active thread mask into the target thread mask of the convergence barrier, and clearing the completed thread mask of the convergence barrier to zero; during instruction execution, prioritizing the execution of the path instructions of the high-frequency execution branches, and setting the program counter (PC) of the first instruction of the non-high-frequency execution branches to zero. The program counter (PC) and active thread mask are pushed onto the stack; after the instructions of the high-frequency execution branch are executed, the active thread mask of the instructions of the high-frequency execution branch is subjected to convergence barrier processing according to the second convergence barrier instruction; based on the convergence barrier processing result, the PC and active thread mask of the first instruction of the non-high-frequency execution branch are popped from the stack to execute the instructions of the non-high-frequency execution branch path; after the instructions of the non-high-frequency execution branch are executed, convergence barrier processing is performed according to the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

[0094] The first convergence barrier instruction can be a convergege instruction used to implement a Join operation. The original convergence instruction can be the instruction executed after the branches merge. The second convergence barrier instruction can be a convergege instruction used to implement a Sync operation. A non-high-frequency execution branch can be a branch of another execution path corresponding to a high-frequency execution branch. An unconditional jump instruction can be an instruction that jumps without any conditional constraints. The currently active thread mask can be the thread mask of the currently executing instruction. Path instructions can refer to all instructions involved in a branch path. The first instruction is also the first instruction of the branch.

[0095] Figure 7 This is a schematic diagram illustrating the branch compilation optimization of a sparse attention structure under a convergence barrier mode, as provided in Embodiment 3 of the present invention. In a specific example, such as... Figure 7 As shown, A through H each represent an instruction within the sparse attention structure. Here, we'll use a thread warp value of 4 as an example. Since instruction B is a branch decision instruction (the instruction is B.EQ, i.e., the EQ mode of instruction B), in convergence barrier mode, the compiler inserts a first convergence barrier instruction before the branch to be executed, a second convergence barrier instruction before the original convergence instruction of each branch to be executed, and an unconditional jump instruction at the end of non-frequently executed branches. That is, instruction A (CB.Join) is inserted before instruction B, instruction F is inserted before instruction G, and instruction D is inserted at the end of instruction C. Instructions A and F are specifically convergence instructions with two different operations; that is, the first convergence barrier instruction can be of the CB.Join instruction type, and the second convergence barrier instruction can be of the CB.Sync instruction type. Instruction D can be an unconditional jump instruction. Specifically, instruction D is B.EQ.F, indicating an unconditional jump to instruction F.

[0096] Converge Barrier can be a barrier counter that occupies a continuous 64-bit address space and is stored in a scalar register file. It is used to optimize the merging behavior of warp branches caused by conditional jumps and can realize thread bundle merging at the beginning and end of thread bundle branches. Converge Barrier consists of two WARP_SIZE (thread bundle size) bits of Thread BitMask, namely the Target Mask representing the target convergence Bit Mask and the Arrive Mask representing the Bit Mask of the threads that have reached the convergence point. It is stored in two consecutive scalar registers with IDs of 2N and 2N+1 using SRT (Scalar Register). There are three operations of Converge Barrier: (1) Join operation: used for the current warp branch. Before branch divergence occurs, Converge Barrier is initialized. The Active Thread Mask of the current warp branch is written to the Target Thread Mask, and the Arrive Mask is cleared. (2) Sync operation: Used for the currently running warp branch, performing an OR operation between the current Active Thread Mask and the ArriveMask of the Converge Barrier. If the ArriveMask and the Target Mask are equal after the operation, the Active Thread Mask of the current warp branch is rewritten as the Target Mask, the convergence is complete, and the current warp branch continues execution. Otherwise, the SIMT Stack is popped, and the warp branch held at the top of the SIMT Stack is executed instead. (3) Break operation: Used to remove the Active Thread Mask of the current warp branch from the Target Mask of the Converge Barrier, and the current warp branch continues execution.

[0097] Table 1. List of components of the converge instruction Table 2 List of operation types for the converge command Table 1 is a list of the structural components of a converge instruction provided in Embodiment 3 of the present invention, and Table 2 is a list of the operation types of a converge instruction provided in Embodiment 3 of the present invention. In a specific example, as shown in Tables 1 and 2, the converge instruction is a 32-bit instruction. The first 7 bits of this convergence barrier instruction are identified by 0x20 as a dedicated convergence barrier instruction. This dedicated convergence barrier instruction is used to read the Target Mask and the Arrived Mask respectively based on two consecutive scalar registers, 2N and 2N+1, represented by the SRT register, a scalar register. TYPE represents a convergence barrier instruction type described by 2 bits of data: 0 represents JOIN, 1 represents SYNC, and 2 represents BREAK. NA indicates that the data bits are meaningless. Bits [10:8] of the instruction are assigned the constant 0x1.

[0098] The Diverge instruction, often abbreviated as B instruction, determines whether a branch exists during a conditional check and executes that branch. The Diverge instruction reads the Vector registers RS (Source Register) and RT (Target Register), performs a 32-bit signed comparison between RS and RT based on the comparison condition COND, and determines which branch to follow based on the comparison result. It then updates the Program Counter (PC) of each thread (Thread) based on the comparison result and an 8-bit signed immediate value. For all active threads in the current warp branch, it determines whether to jump to "current PC (address of the current instruction) + IMM (immediate value)" based on the vector comparison result. The address is 4". When some threads choose to jump and others choose to execute sequentially, the current warp branch will branch; otherwise, no branch will occur. RS / RT can be a vector register or a scalar register. When a branch occurs, the number of threads in each of the two branches is counted after the active threads branch. The branch with fewer threads can be taken as the new current warp branch, i.e., the high-frequency execution branch. If the number of active threads in each of the two branches is equal, the branch that executes sequentially is selected. After determining the branch direction, the current PC is updated, and the active thread mask of the current warp is updated with the set of threads that selected the branch. The target PC and the mask corresponding to the set of threads of the other unselected branch are pushed onto the SIMT Stack. They are not executed until all threads of the selected branch have finished executing or a Converge instruction is encountered, at which point the selected branch is popped from the SIMT Stack and executed as the current warp branch.

[0099] Table 3 is a list of the composition structure of a Diverge instruction provided in Embodiment 3 of the present invention, Table 4 is a list of the composition structure of a Diverge instruction provided in Embodiment 3 of the present invention, and Table 5 is a list of the operation types of a Diverge instruction provided in Embodiment 3 of the present invention. As shown in Tables 3 and 4, the instruction bits [31:0] in the first row of Tables 3 and 4 indicate that the branch judgment instruction is a 32-bit instruction. The second row refers to the physical meaning of the data filled in different instruction bits. For example, "0x10" filled in the [6:2] position indicates that bits 2 to 6 of the instruction are the instruction identifier of the branch judgment instruction. SRS[6:0] represents a source operand register number described using 7 bits of data; this source operand is a scalar register. VRS[7:0] represents a source operand register described using 8 bits of data; this source operand register is a vector register. VRT[7:0] represents a destination operand register described using 8 bits of data; this destination operand register is a vector register. SRT[6:0] represents a destination operand register described using 7 bits of data; this destination operand register is a scalar register. NA indicates that the data bits are meaningless. COND represents a comparison condition described using 2 bits of data: EQ for equal to, LT for less than, and GE for greater than or equal to. IMM represents an immediate value described using 8 bits of data. Unconditional jump instructions are a special instruction mode of branch judgment instructions.

[0100] Table 3. List of the components of the Diverge command Table 4. List of the components of the Diverge command Table 5. List of operation types for the Diverge command For example, such as Figure 7 As shown, after the insertion of the two convergence barrier instructions, A and F, the thread mask for instruction A is 1111, and four threads execute instruction A in parallel. The active thread mask (ActiveThread Mask) is written to the convergence barrier's Target Mask (1111), and the convergence barrier's Arrive Mask (completed mask) is cleared. Simultaneously, instruction F (CB.Sync instruction) is inserted before instruction G at the convergence point of instructions C and E. After instruction B is executed, two branches appear: C (thread mask 0111) and E (thread mask 1000), two ADD instructions. If the high-frequency execution branch is predicted based on the number of threads occupied by the branch, the branch with fewer active threads is selected as the predicted high-frequency execution branch. In the thread mask, 1 represents an active thread, and 0 represents an inactive thread. Therefore, instruction E on the right side can be predicted as the high-frequency execution branch and executed first. At the same time, the PC corresponding to instruction C and the active thread mask (Active Mask 0111) are pushed onto the SIMT stack. After executing instruction E, the second convergence barrier instruction, instruction F, is executed to perform convergence barrier processing on the active thread mask of instructions in high-frequency execution branches.

[0101] In an optional embodiment of the present invention, the step of performing convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction may include: performing an OR operation on the active thread mask of the instructions of the high-frequency execution branch and the completed thread mask according to the second convergence barrier instruction to obtain an updated completed thread mask; comparing the updated completed thread mask with the target thread mask to obtain the convergence barrier processing result.

[0102] Among them, the updated completed thread mask is the updated completed thread mask.

[0103] For example, such as Figure 7As shown, when performing convergence barrier processing on the active thread mask of the E instruction in a high-frequency execution branch according to the second convergence barrier instruction CB.Sync, the CB.Sync instruction can either the current active thread mask ActiveMask (1000) or the completed thread mask Arrive Mask of the convergence barrier, thus updating Arrive Mask to 1000. Further, it checks whether Arrive Mask is equal to Target Mask, obtaining the convergence barrier processing result if they are not equal. At this point, based on this convergence barrier processing result, the non-high-frequency execution branch in the SIMT stack is popped ( Figure 7 The first instruction in the left branch (the C instruction) corresponds to the PC and Active Mask, and execution of the C instruction begins. After executing the C instruction, execution continues with the D instruction, which unconditionally transfers to the F instruction.

[0104] Correspondingly, after the instructions in the non-high-frequency execution branch have been executed, an unconditional jump instruction can be used to unconditionally transfer to the second convergence barrier instruction. Based on the second convergence barrier instruction, the active thread mask of the instructions in the non-high-frequency execution branch is ORed with the completed thread mask to obtain an updated completed thread mask. The updated completed thread mask is then compared with the target thread mask to obtain the convergence barrier processing result. For example, as shown... Figure 7 As shown, after the instructions of the non-high-frequency execution branch path on the left are executed, the Arrive Mask of the convergence barrier is updated again. The current Active Mask (0111) is ORed with the Arrive Mask (1000) to obtain the updated Active Mask (1111), which is equal to the Target Mask (1111), thus completing the convergence. After all threads perform convergence, they continue to execute G and H instructions until they exit.

[0105] In summary, based on the model branch monitoring data obtained by the performance monitoring unit, the high-frequency execution branches within the sparse attention structure can be identified. For example, if the ADD2 branch executes before the ADD1 branch, then the ADD2 branch is determined to be a high-frequency execution branch. Furthermore, the compiler rearranges the instruction code of the high-frequency execution branches within the sparse attention structure, placing the ADD2 and converge instructions after the branch decision instructions, thereby reducing pipeline bubbles caused by instruction jumps within the network. When rearranging the instruction code of the high-frequency execution branches, the compiler can inline or rearrange the actual execution order of the branch paths, thus ensuring continuous hot paths and reducing the number of branches.

[0106] In an optional embodiment of the present invention, adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode may include: when it is determined that the branch compilation execution mode is a non-convergence barrier mode, inserting an unconditional jump instruction at the end of the non-high-frequency execution branch, prioritizing the execution of the path instructions of the high-frequency execution branch, and pushing the PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after it is determined that the path instructions of the high-frequency execution branch have been executed, popping the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack to execute the path instructions of the non-high-frequency execution branch.

[0107] Figure 8 This is a flowchart illustrating branch compilation optimization of a sparse attention structure in a no-convergence barrier mode, as provided in Embodiment 3 of the present invention. In a specific example, such as... Figure 8 As shown, A through F each represent an instruction within the sparse attention structure. Here, we take an example with a thread warp value of 4. Since A is a branch decision instruction (instruction B.EQ, i.e., the EQ mode of instruction B), the thread mask for instruction A is 1111, and four threads execute instruction A in parallel. Because it's a no-convergence-barrier mode, the compiler doesn't insert Converge instructions, but it inserts an unconditional jump instruction C (B.EQ.E, indicating an unconditional jump to instruction E) at the end of non-high-frequency execution branches. After instruction A is executed, two branches appear: B (thread mask 0111) and D (thread mask 1000), two ADD instructions. If we predict the high-frequency execution branch based on the number of threads occupied by the branch, the branch with fewer active threads is chosen as the predicted high-frequency execution branch. That is, instruction D can be predicted as a high-frequency execution branch and executed first. Simultaneously, the PC and Active Mask (0111) corresponding to instruction B are pushed onto the SIMT stack. After executing instruction D, instruction E is executed, followed by instruction F (exit). Instructions D, E, and F are the path instructions for the high-frequency execution branches on the right. Further, it checks if the SIMT stack is empty. If not, the PC and Active Mask corresponding to instruction B are popped from the SIMT stack, and instruction B is executed. After executing instruction B, the unconditional jump instruction C is executed, unconditionally transferring to instruction E. Instruction F (exit) is then executed, and it checks if the SIMT stack is empty. If it is, the program exits. Therefore, in the no-convergence-barrier mode, rearranging the branch path code based on the model branch monitoring data obtained by the performance monitoring unit can reduce instruction jumps and latency caused by cache misses.

[0108] Optionally, each SM (Streaming Multiprocessor) subcore can use scalar registers to build convergence barriers and SIMT stacks, without occupying stack storage space, allowing for flexible execution granularity, and enabling automatic push and pop operations via hardware.

[0109] In an optional embodiment of the present invention, the method may further include: inserting a prefetch instruction before executing the high-frequency execution branch; during instruction execution, if it is determined that the current instruction is the prefetch instruction, parsing the prefetch instruction to obtain the address of the target prefetch instruction; in response to the prefetch instruction, initiating an instruction prefetch request and loading the target prefetch instruction into the instruction cache; wherein the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; if it is determined that the current instruction is a high-frequency execution branch instruction, reading the cached target prefetch instruction from the instruction cache and directly executing the target prefetch instruction.

[0110] In this context, a prefetch instruction can be used to prefetch data or instructions. The target prefetch instruction address can be the memory address used to fetch the target prefetch instruction. An instruction prefetch request can be used to request prefetch instructions. The target prefetch instruction is the instruction that needs to be prefetched; it can be some or all of the instructions in a frequently executed branch.

[0111] To further optimize the compilation of the internal branches of the sparse attention structure, after adjusting the instruction arrangement order of the high-frequency execution branches inside the attention structure according to the type of branch compilation execution mode, the high-frequency execution branch path to be executed inside the sparse attention structure can be determined according to the actual execution order of the branch path instructions. Then, the compiler inserts prefetch instructions before the instructions of the current high-frequency execution branch path to explicitly notify the hardware to prefetch the high-frequency execution branch instructions, thereby improving hardware efficiency.

[0112] Figure 9 This is a schematic diagram of a high-frequency execution branch instruction prefetching process within a sparse attention structure according to Embodiment 3 of the present invention. In a specific example, such as Figure 9As shown, during the execution of instructions within the sparse attention structure by the kernel after compilation and optimization on the device side, it can be determined whether the current instruction is a prefetch instruction. If so, the prefetch instruction is parsed to obtain the address of the target prefetch instruction, and an instruction prefetch request is initiated to the memory subsystem in response to the prefetch instruction. This requests the memory subsystem to load the cache line containing the target prefetch instruction into the instruction cache (I Cache), and then the execution of subsequent instructions continues. It is understood that the target prefetch instruction can be part or all of the instructions in a high-frequency execution branch within the sparse attention structure. The number of instruction entries included in the target prefetch instruction can be configured according to specific hardware parameters, and this embodiment of the invention does not limit this. If the current instruction is not a prefetch instruction, it can be further determined whether the current instruction is a high-frequency execution branch instruction. If so, it can be further checked whether the target prefetch instruction is cached in the I Cache. If a match is found, the target prefetch instruction is executed directly; if it is missing, an instruction loading request is initiated to load the missing target prefetch instruction from memory into the I Cache. If the current instruction is not a high-frequency execution branch instruction, the execution of subsequent instructions continues.

[0113] It is evident that the above scheme improves pipeline efficiency by inserting prefetch instructions to prefetch instructions for high-frequency execution branches, thereby enhancing the model's running efficiency and hardware utilization.

[0114] Prefetch instructions, used to prefetch both data and instructions, are called prefetch instructions. Their data cache prefetch function reads a contiguous file of vector registers with IDs 2N and 2N+1 specified by the Vector register VRT (destination operand register in vector form), forming a 64-bit vector base address. This base address is then added to a 32-bit offset address in the register specified by the SRS (source operand register number in scalar form) to form a 64-bit memory access address. The TYPE field defines which cache level (L1, L2, or L3) the data is prefetched into. Their instruction cache prefetch function reads a contiguous file of vector registers with IDs 2N and 2N+1 specified by the Vector register VRT, forming a 64-bit vector base address. This base address is then added to an IMM offset address to form a 64-bit memory access address PC. NUM instructions are prefetched into the I Cache.

[0115] Table 6. List of components of the data prefetch command Table 7. List of components of the prefetch command Table 8. List of possible values ​​for the TYPE field in the prefetch command. Table 9. List of possible values ​​for the SIZE field in the prefetch command. Table 10 List of NUM field values ​​for the prefetch command Table 6 is a list of the components of a prefetch instruction provided in Embodiment 3 of the present invention; Table 7 is a list of the components of a prefetch instruction provided in Embodiment 3 of the present invention; Table 8 is a list of values ​​for the TYPE field in a prefetch instruction provided in Embodiment 3 of the present invention; Table 9 is a list of values ​​for the SIZE field in a prefetch instruction provided in Embodiment 3 of the present invention; and Table 10 is a list of values ​​for the NUM field in a prefetch instruction provided in Embodiment 3 of the present invention. That is, prefetch instructions can include two types: data prefetch instructions and instruction prefetch instructions. Data prefetch instructions can be used to prefetch data, and instruction prefetch instructions can be used to prefetch instructions. In Tables 6 and 7, the instruction bits [31:0] in the first row represent that the prefetch instruction is a 32-bit instruction. The second row refers to the physical meaning of the data filled under different instruction bits. For example, "0x30" filled at position [6:2] represents that bits 2 to 6 of the instruction are the instruction identifier of the prefetch instruction. VRT[7:1] represents a source operand register described using 8 bits of data. This source operand register is a vector register used to fetch 2N and 2N+1 consecutive vector registers to form a 64-bit base address. SRS[6:0] represents a source operand register number described using 7 bits of data. This source operand is a scalar register representing the offset address. NA indicates that the data bits are meaningless. As shown in Table 8, TYPE represents a cache type described using 2 bits of data, where 0 indicates reading from L1 Cache, 1 indicates reading from L2 Cache, and 2 indicates reading from L3 Cache. As shown in Table 9, SIZE represents a prefetch data type described using 2 bits of data. 0 indicates a 32-bit Word (W), 1 indicates a 16-bit Half Word (H), 2 indicates an 8-bit signed Byte (B), and 3 indicates an 8-bit unsigned Byte (BU). NUM represents the number of prefetch instructions described using 2 bits: 0 for 1 instruction, 1 for 2 instructions, 2 for 4 instructions, and 3 for 8 instructions. IMM represents the immediate value.

[0116] This invention proposes a hardware and software architecture for accelerating inference of in-model attention structures, which can be integrated into existing deep learning frameworks. This architecture uses a performance monitoring unit to monitor the attention structure in real time and generate model branch monitoring data. The compiler, based on the acquired model branch monitoring data and combining efficient Diverge / Converge / Prefetch instructions with compilation optimization strategies, performs targeted optimization of the model branch code of the attention structure. It predicts high-frequency execution branches within the attention structure based on branch execution probabilities and, combined with prefetch instructions, prefetches and executes instructions for high-frequency execution branch paths, thereby shifting from instruction branch prediction to data access pattern prediction and computation path inference. Prefetching and vectorization eliminate SIMT divergence, optimize the internal code layout of the attention structure, and reduce dynamic branches within the attention structure, thus reducing runtime latency during inference, improving the efficiency of instruction-level parallelism and pipelines, maximizing computational and memory access efficiency, and ultimately improving the operational efficiency of the attention structure. This, in turn, improves the execution efficiency and hardware resource utilization of the attention structure, achieving near-peak computing power utilization.

[0117] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0118] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0119] Example 4 Figure 10 This is a schematic diagram of a compiler optimization device for an attention structure provided in Embodiment 4 of the present invention, as shown below. Figure 10 As shown, the device includes: an attention execution configuration file generation module 1010, a parallel optimization branch generation module 1020, and a branch process data prefetching module 1030, wherein: The attention execution configuration file generation module 1010 is used to obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the attention event monitoring data in real time; The parallel optimization branch generation module 1020 is used to generate a parallel optimization branch of the current attention structure branch in the currently running model according to the attention execution configuration file; The branch process data prefetching module 1030 is used to prefetch the branch process data of the current attention structure branch according to the parallel optimization branch of the current attention structure branch.

[0120] This invention, through an embodiment of the present invention, acquires attention event monitoring data generated in real-time by a performance monitoring unit monitoring the current running model. Based on this data, an attention execution configuration file is generated, which in turn generates a parallel optimization branch for the current attention structure branch within the current running model. Furthermore, the branch flow data of the current attention structure branch is prefetched based on this parallel optimization branch. The performance monitoring unit configures an event sampling module using a set of first registers. Each first register within this module is used to record and store attention event monitoring data in real-time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire attention event-related data in real-time, assisting in adjusting and optimizing the prefetching operation of branch flow data within the attention structure network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient compilation optimization strategies for the attention structure network. This significantly reduces memory access latency of the attention structure, thereby reducing runtime latency during attention structure inference, improving the efficiency of instruction-level parallelism and pipelines, and ultimately increasing the hardware resource utilization of attention structure execution.

[0121] Optionally, the parallel optimization branch generation module 1020 is further configured to: determine the current token query vector currently being processed by the current attention structure branch according to the attention execution configuration file; and generate a parallel optimization branch of the current token query vector currently being processed by the current attention structure branch.

[0122] Optionally, the type of the current attention structure branch is a sparse attention structure branch, and the branch process data prefetching module 1030 is further configured to: generate the token historical co-occurrence information of the current token query vector according to the attention execution configuration file; calculate the key-value cache (KV Cache) prefetch address of the associated co-occurring tokens of the current token query vector according to the token historical co-occurrence information of the current token query vector; and prefetch the predicted KV Cache data of the current token query vector according to the KV Cache prefetch address of the associated co-occurring tokens of the current token query vector.

[0123] Optionally, the branch process data prefetching module 1030 is further configured to: determine the historical loading KV Cache of the historical Token query vector according to the attention execution configuration file; count Token co-occurrence pairs according to the historical loading KV Cache of the historical Token query vector; filter target Token co-occurrence pairs from the Token co-occurrence pairs whose co-occurrence frequency exceeds a set frequency threshold, and generate a Token co-occurrence table according to the target Token co-occurrence pairs; match the current Token query vector with the Token co-occurrence table to obtain the historical Token co-occurrence information of the current Token query vector.

[0124] Optionally, the branch process data prefetching module 1030 is further configured to: initiate a prefetching instruction based on the KV Cache prefetching address of the associated co-occurring token of the current token query vector; and, in response to the prefetching instruction, preload the predicted KV Cache data of the current token query vector into different levels of cache based on the model architecture parameters and hardware architecture parameters.

[0125] Optionally, the branch process data prefetching module 1030 is further configured to: calculate the cache capacity of the predicted KV Cache data; preload the predicted KV Cache data into the first-level cache if the cache capacity of the predicted KV Cache data is less than or equal to the first-level cache capacity; preload the predicted KV Cache data into the second-level cache if the cache capacity of the predicted KV Cache data is greater than the first-level cache capacity and less than or equal to the second-level cache capacity; and preload the predicted KV Cache data into the third-level cache if the cache capacity of the predicted KV Cache data is greater than the second-level cache capacity.

[0126] Optionally, the above device further includes an attention structure internal branch optimization module, used for: during the execution of the current attention structure branch, acquiring model branch monitoring data generated by the performance monitoring unit in real time monitoring the current attention structure branch; wherein, the performance monitoring unit further includes a branch recording module, the branch recording module includes a set of second registers, each of the second registers using a circular buffer to record and store the model branch monitoring data in real time; generating a branch execution configuration file based on the model branch monitoring data; determining the high-frequency execution branches among the branches to be executed in the current attention structure branch based on the branch execution configuration file; and adjusting the instruction execution order of the high-frequency execution branches to prioritize the execution of the branch instructions of the high-frequency execution branches.

[0127] Optionally, the attention structure internal branch optimization module is further configured to: determine the branch compilation execution mode based on the number of instruction entries in the high-frequency execution branch; and adjust the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode.

[0128] Optionally, the attention structure internal branch optimization module is further configured to: when the branch compilation and execution mode is determined to be a convergence barrier mode, insert a first convergence barrier instruction before each of the branches to be executed, insert a second convergence barrier instruction before the original convergence instruction of each of the branches to be executed, and insert an unconditional jump instruction at the end of the non-high-frequency execution branch; write the currently active thread mask into the target thread mask of the convergence barrier, and clear the completed thread mask of the convergence barrier; during instruction execution, prioritize the execution of the path instructions of the high-frequency execution branch, and push the program counter PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after the instructions of the high-frequency execution branch are executed, perform convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction; pop the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack according to the convergence barrier processing result, so as to execute the instructions of the non-high-frequency execution branch path; after the instructions of the non-high-frequency execution branch are executed, perform convergence barrier processing according to the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

[0129] Optionally, the attention structure internal branch optimization module is further configured to: perform an OR operation on the active thread mask of the instruction of the high-frequency execution branch and the completed thread mask according to the second convergence barrier instruction to obtain an updated completed thread mask; and compare the updated completed thread mask with the target thread mask to obtain the convergence barrier processing result.

[0130] Optionally, the attention structure internal branch optimization module is further configured to: when it is determined that the branch compilation execution mode is a non-convergence barrier mode, insert an unconditional jump instruction at the end of the non-high-frequency execution branch, prioritize the execution of the path instructions of the high-frequency execution branch, and push the PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after it is determined that the path instructions of the high-frequency execution branch have been executed, pop the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack to execute the path instructions of the non-high-frequency execution branch.

[0131] Optionally, the attention structure internal branch optimization module is further configured to: insert a prefetch instruction before executing the high-frequency execution branch; during instruction execution, if it is determined that the current instruction is the prefetch instruction, parse the prefetch instruction to obtain the address of the target prefetch instruction; in response to the prefetch instruction, initiate an instruction prefetch request and load the target prefetch instruction into the instruction cache; wherein, the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; if it is determined that the current instruction is a high-frequency execution branch instruction, read the cached target prefetch instruction from the instruction cache and directly execute the target prefetch instruction.

[0132] The above-described attention structure compilation optimization apparatus can execute the attention structure compilation optimization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the attention structure compilation optimization method provided in any embodiment of the present invention.

[0133] Since the attention structure compilation optimization apparatus described above is an apparatus capable of executing the attention structure compilation optimization method in the embodiments of the present invention, those skilled in the art can understand the specific implementation and various variations of the attention structure compilation optimization apparatus of this embodiment based on the attention structure compilation optimization method described in the embodiments of the present invention. Therefore, how the attention structure compilation optimization apparatus implements the attention structure compilation optimization method in the embodiments of the present invention will not be described in detail here. Any apparatus used by those skilled in the art to implement the attention structure compilation optimization method in the embodiments of the present invention falls within the scope of protection of this application.

[0134] Example 5 Figure 11 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0135] like Figure 11As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0136] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as compiler optimization methods for attention structures.

[0138] In some embodiments, the compilation optimization method for attention structures can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the compilation optimization method for attention structures described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the compilation optimization method for attention structures by any other suitable means (e.g., by means of firmware).

[0139] Optionally, the compilation optimization method for the attention structure may include: acquiring attention event monitoring data generated in real time by a performance monitoring unit monitoring the current running model, and generating an attention execution configuration file based on the attention event monitoring data; wherein the performance monitoring unit includes an event sampling module, the event sampling module includes a set of first registers, each of the first registers being used to record and store the attention event monitoring data in real time; generating a parallel optimization branch of the current attention structure branch in the current running model based on the attention execution configuration file; and prefetching the branch flow data of the current attention structure branch based on the parallel optimization branch of the current attention structure branch.

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0145] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0146] This invention also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the compilation optimization method for the attention structure provided in any embodiment of this invention. This program product shares the same inventive concept as the compilation optimization method for the attention structure disclosed in the embodiments of this invention, and therefore will not be described further here.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the embodiments of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in the embodiments of the present invention can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A compilation optimization method for attention structures, characterized in that, include: The system acquires attention event monitoring data generated in real time by the performance monitoring unit for the currently running model, and generates an attention execution configuration file based on the attention event monitoring data. The performance monitoring unit includes an event sampling module, which includes a set of first registers, each of which is used to record and store the attention event monitoring data in real time. Generate a parallel optimization branch for the current attention structure branch in the currently running model based on the attention execution configuration file; The branch flow data of the current attention structure branch is prefetched based on the parallel optimization branch of the current attention structure branch.

2. The method according to claim 1, characterized in that, The step of generating a parallel optimization branch for the current attention structure branch in the currently running model based on the attention execution configuration file includes: The current token query vector for the current attention structure branch is determined based on the attention execution configuration file. Generate a parallel optimization branch for the current token query vector that is currently being processed by the current attention structure branch.

3. The method according to claim 1 or 2, characterized in that, The current attention structure branch is of the sparse attention structure branch type. The prefetching process of the branch flow data of the current attention structure branch based on the parallel optimization branch of the current attention structure branch includes: Based on the attention execution configuration file, the historical co-occurrence information of the tokens in the current token query vector is generated; Calculate the key-value cache (KV Cache) prefetch address of the associated co-occurring tokens of the current token query vector based on the token historical co-occurrence information of the current token query vector; Prefetch the predicted KV Cache data of the current Token query vector based on the KV Cache prefetch address of the associated co-occurring Tokens of the current Token query vector.

4. The method according to claim 3, characterized in that, The step of generating the historical co-occurrence information of the current token query vector based on the attention execution configuration file includes: Based on the attention execution configuration file, the historical Token query vector is determined and loaded into the KV Cache. Based on the historical KV Cache loaded with the historical token query vector, token co-occurrence pairs are counted. Target token co-occurrence pairs with a co-occurrence frequency exceeding a set frequency threshold are selected from the token co-occurrence pairs, and a token co-occurrence table is generated based on the target token co-occurrence pairs; The current token query vector is matched with the token co-occurrence table to obtain the token historical co-occurrence information of the current token query vector.

5. The method according to claim 3, characterized in that, The step of prefetching the predicted KV Cache data of the current Token query vector based on the KV Cache prefetch address of the associated co-occurring Tokens of the current Token query vector includes: A prefetch instruction is initiated based on the KV Cache prefetch address of the co-occurring token associated with the current token query vector; In response to the prefetch instruction, the predicted KV Cache data of the current Token query vector is preloaded into different levels of cache according to the model architecture parameters and hardware architecture parameters.

6. The method according to claim 3, characterized in that, The method further includes: During the execution of the current attention structure branch, model branch monitoring data generated by the performance monitoring unit in real time is acquired; wherein, the performance monitoring unit further includes a branch recording module, the branch recording module includes a set of second registers, and each second register uses a circular buffer to record and store the model branch monitoring data in real time; Generate a branch execution configuration file based on the model branch monitoring data; The high-frequency execution branches among the branches to be executed in the current attention structure branch are determined according to the branch execution configuration file; Adjust the instruction execution order of the high-frequency execution branch so that the branch instructions of the high-frequency execution branch are executed first.

7. The method according to claim 6, characterized in that, The adjustment of the instruction execution order of the high-frequency execution branch includes: The branch compilation and execution mode is determined based on the number of instruction entries in the high-frequency execution branch; Adjust the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation and execution mode.

8. The method according to claim 7, characterized in that, The step of adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode includes: When the branch compilation execution mode is determined to be the convergence barrier mode, a first convergence barrier instruction is inserted before each of the branches to be executed, a second convergence barrier instruction is inserted before the original convergence instruction of each of the branches to be executed, and an unconditional jump instruction is inserted at the end of the non-high frequency execution branch. Write the currently active thread mask into the target thread mask of the convergence barrier, and clear the completed thread mask of the convergence barrier to zero; During instruction execution, the path instructions of the high-frequency execution branches are executed first, and the program counter PC and active thread mask of the first instruction of the low-frequency execution branches are pushed onto the stack. After the instructions of the high-frequency execution branch are executed, the active thread mask of the instructions of the high-frequency execution branch is subjected to convergence barrier processing according to the second convergence barrier instruction; Based on the convergence barrier processing result, pop the PC and active thread mask of the first instruction of the non-high frequency execution branch from the stack to execute the instructions of the non-high frequency execution branch path; After the instructions of the non-high-frequency execution branch are executed, a convergence barrier is performed based on the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

9. The method according to claim 8, characterized in that, The step of performing convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction includes: According to the second convergence barrier instruction, the active thread mask of the instruction of the high-frequency execution branch is ORed with the completed thread mask to obtain the updated completed thread mask; The updated thread mask is compared with the target thread mask to obtain the convergence barrier processing result.

10. The method according to claim 7, characterized in that, The step of adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode includes: If the branch compilation execution mode is determined to be a non-convergence barrier mode, an unconditional jump instruction is inserted at the end of the non-high-frequency execution branch, the path instruction of the high-frequency execution branch is executed first, and the PC and active thread mask of the first instruction of the non-high-frequency execution branch are pushed onto the stack. After the path instructions of the high-frequency execution branch have been executed, the PC and active thread mask of the first instruction of the low-frequency execution branch are popped from the stack to execute the path instructions of the low-frequency execution branch.

11. The method according to any one of claims 7-10, characterized in that, Also includes: Insert a prefetch instruction before executing the high-frequency execution branch; During instruction execution, if it is determined that the current instruction is the prefetch instruction, the prefetch instruction is parsed to obtain the target prefetch instruction address; In response to the prefetch instruction, an instruction prefetch request is initiated to load the target prefetch instruction into the instruction cache; wherein, the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; If it is determined that the current instruction is a high-frequency branch instruction, the target prefetch instruction is read from the instruction cache and the target prefetch instruction is executed directly.

12. A compiler optimization apparatus for an attention structure, characterized in that, include: The attention execution configuration file generation module is used to obtain attention event monitoring data generated by the performance monitoring unit in real time monitoring the current running model, and generate an attention execution configuration file based on the attention event monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the attention event monitoring data in real time; The parallel optimization branch generation module is used to generate a parallel optimization branch for the current attention structure branch in the currently running model based on the attention execution configuration file. The branch process data prefetching module is used to prefetch the branch process data of the current attention structure branch according to the parallel optimization branch of the current attention structure branch.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor, such that the at least one processor is able to perform the compilation optimization method for the attention structure according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the compilation optimization method for the attention structure described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the compilation optimization method of the attention structure according to any one of claims 1-11.

Images