Branch compilation optimization method, device and equipment of expert network, medium and program

Abstract

Embodiments of the present application disclose a branch compilation optimization method, device, equipment, medium and program of an expert network, wherein the method comprises: obtaining expert call monitoring data generated by a performance monitoring unit in real time monitoring of expert call conditions of a current running model, and generating an expert network execution configuration file according to the expert call monitoring data; wherein the performance monitoring unit comprises an event sampling module, the event sampling module comprises a group of first registers, and each first register is used for real-time recording and storing expert call monitoring data; determining a high-frequency expert combination according to the expert network execution configuration file; and adjusting the code layout order of each expert network in the current running model according to the high-frequency expert combination. The technical solution of the embodiments of the present application can reduce the delay of expert network reasoning running time, improve the efficiency of expert network instruction level parallelism and pipeline, and further improve the hardware resource utilization rate of expert network execution.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of chip and artificial intelligence technology, and in particular to a branch compilation optimization method, apparatus, electronic device, storage medium and program for expert networks. Background Technology

[0002] With the continuous evolution of artificial intelligence technologies such as Large Language Models (LLMs), they have been widely applied in various industries. During the inference process, various large models generally exhibit dynamic branching behavior.

[0003] Currently, in large-scale model inference, when the context length of a large model exceeds 128K or even higher, sparse architecture becomes crucial for balancing model performance and effectiveness. MoE (Mixture of Experts) is the mainstream sparse architecture for large models. Its core principle is to separate the total number of parameters from the actual computational cost by using a parallel expert ensemble dynamic routing approach, achieving a training / inference process with large model capacity and moderate computational cost. The sparse architecture characteristics of MoE involve dynamic branching behavior. Whether it's the branching behavior of other structures within the model or the dynamic branching behavior involved in the sparse architecture of MoE, severe branch divergence can lead to inefficiencies in the SIMT (Single Instruction Multiple Threads) pipeline, significantly dragging down and reducing the overall model performance, thus affecting the overall model runtime. Summary of the Invention

[0004] This invention provides a branch compilation optimization method, apparatus, electronic device, storage medium, and program for expert networks, which can reduce the latency of expert network inference runtime, improve the efficiency of expert network instruction-level parallelism and pipeline, and thus improve the hardware resource utilization of expert network execution.

[0005] According to one aspect of the present invention, a branch compilation optimization method for expert networks is provided, comprising: The performance monitoring unit acquires expert call monitoring data generated in real time by monitoring the expert call status of the current running model, and generates an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; The high-frequency expert combination is determined based on the configuration file executed by the expert network. Adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0006] According to another aspect of the present invention, a branch compilation optimization apparatus for expert networks is provided, comprising: An expert network execution configuration file generation module is used to obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and to generate an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; A high-frequency expert portfolio determination module is used to determine a high-frequency expert portfolio based on the expert network execution configuration file. The code layout order adjustment module is used to adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the branch compilation optimization method for expert networks according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the branch compilation optimization method for expert networks according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the branch compilation optimization method for expert networks as described in any embodiment of the present invention.

[0010] This invention, through an embodiment of the present invention, acquires expert call monitoring data generated in real time by a performance monitoring unit that monitors the expert call status of the current running model. Based on this data, an expert network execution configuration file is generated, which in turn determines high-frequency expert combinations. Furthermore, the code layout order of each expert network in the current running model is adjusted according to these high-frequency expert combinations. The performance monitoring unit configures an event sampling module through a set of first registers. Each first register within this module is used to record and store expert call monitoring data in real time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire expert call-related data in real time, assisting in adjusting the code layout order of each expert network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient expert network branch compilation optimization strategies. This significantly reduces expert network branch divergence, thereby reducing latency during expert network inference runtime, improving the efficiency of expert network instruction-level parallelism and pipelines, and ultimately increasing the utilization rate of hardware resources for expert network execution.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a branch compilation optimization method for expert networks provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the internal structure of an LLM provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of an unoptimized expert network code sequence layout in an existing model, provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the sequential layout execution flow of unoptimized expert network code in an existing model, provided in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the interaction between a performance monitoring unit and the software layer according to Embodiment 1 of the present invention; Figure 6 This is a flowchart of a branch compilation optimization method for expert networks provided in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of an optimized expert network code sequence layout provided in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of the optimized expert network code sequential layout execution flow in a model provided in Embodiment 2 of the present invention; Figure 9 This is a flowchart of a branch compilation optimization method for expert networks provided in Embodiment 3 of the present invention; Figure 10 This is a schematic diagram of the branch compilation optimization process of an expert network in convergence barrier mode provided in Embodiment 3 of the present invention; Figure 11 This is a schematic diagram of the branch compilation optimization process of an expert network in a convergence barrier-free mode, provided in Embodiment 3 of the present invention; Figure 12 This is a schematic diagram of a high-frequency execution branch instruction prefetching process provided in Embodiment 3 of the present invention; Figure 13 This is a schematic diagram of a branch compilation optimization device for an expert network provided in Embodiment 4 of the present invention; Figure 14 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1This is a flowchart of a branch compilation optimization method for expert networks provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where expert networks are optimized using a software-hardware co-processing approach. This method can be executed by an expert network branch compilation optimization device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. This electronic device can be a terminal device or a server device, as long as it can execute the expert network branch compilation optimization method. The present invention does not limit the specific type of electronic device. Correspondingly, as... Figure 1 As shown, the method includes the following operations: S110. Obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and generate an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time.

[0017] The Performance Monitoring Unit (PMU) can be a hardware module running within a device unit in a parallel computing context. This device unit can be a processor, for example, including but not limited to a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), or NPU (Neural Processing Unit), as long as it can work collaboratively with the host CPU (Central Processing Unit). This embodiment of the invention does not limit the specific type of device unit that works with the host CPU. The currently running model can be the model currently running in the device unit (or device side), such as a Large Language Model (LLM), various machine learning models, or other types of network models. This embodiment of the invention does not limit the model type of the currently running model. Expert call monitoring data can be data obtained by the Performance Monitoring Unit monitoring the branch calls of the expert network (which can be simply referred to as the expert) in the currently running model. The expert network execution configuration file can be a configuration file used as a reference for compiling and optimizing the expert network branches. The event sampling module can be a hardware module used to collect expert call events. The first register can be a register that constitutes the event sampling module.

[0018] Large language models, in particular, refer to deep learning models trained with massive amounts of relevant data (such as text, speech, or combined text and image data) capable of processing sequences (such as text, speech, images, audio / video, and combined text and image sequences). These models can generate natural language text or understand the meaning of spoken text, and typically have billions of parameters. Large language models can handle various natural language tasks, such as text classification, translation, question answering, text-image analysis, and dialogue, and have wide applications. The input to a large language model is data, such as text, speech, or combined text and image data. The large language model encodes the input data to obtain corresponding word vectors, and then decodes these word vectors to automatically process the input data and obtain the corresponding output data. For example, text can be input into a large language model, which then processes and predicts the input text, outputting the corresponding response text. Therefore, large models can perform a wide range of tasks, including but not limited to text summarization, translation, and sentiment analysis. During the training or inference of large models, branching scenarios are often encountered. The branching problem leads to low efficiency of the SIMT (Single Instruction Multiple Threads) pipeline, thus affecting the overall running time of the model.

[0019] For network models, branching can be understood as different paths being executed during runtime. Branching scenarios are commonly encountered during the training or inference of large models, such as sparse attention and MoE (Mixture of Experts). The selection and invocation of expert networks within the model can be considered a branching situation. Branch divergence can lead to inefficiencies in the SIMT pipeline, affecting the overall runtime of the model.

[0020] The performance monitoring unit, as a hardware module within a device unit, is dedicated to monitoring, statistically analyzing, and sampling performance events of the processor or system for hardware-level performance analysis. Its core functions include monitoring microarchitectural events such as instruction execution, cache access, branch prediction, and TLB (Translation Lookaside Buffer) hits / misses. It also provides capabilities such as cycle counting, event counting, and sampling analysis, supporting fine-grained filtering, such as data filtering mechanisms based on privilege level or security domain. In this embodiment, the performance monitoring unit may be configured with an event sampling module, which can consist of a set of first registers. Each first register can be used to record and store sampled expert call monitoring data in real time. For example, expert call monitoring data may include, but is not limited to, expert routing distribution data, which may include a snapshot of the complete expert call context.

[0021] Optionally, the event sampling module can be configured to count specific microarchitectural events. Count types may include, but are not limited to, the number of executed instructions, cache misses, branch prediction errors, and floating-point operations. This is accomplished by setting a monitoring counter as the first register and triggering recording. The event sampling module configures monitoring parameters, specifying the events to be monitored and the sampling period. For example, monitoring expert routing events and setting the counter overflow value to 500. When the number of monitored events reaches the threshold, the counter overflows. The event sampling module automatically writes the precise machine state to a specific memory area, a hardware-driven process that ensures a precise correspondence between the records and the triggering instructions.

[0022] In this embodiment of the invention, the device-side performance monitoring unit is responsible for real-time monitoring of expert calls in the current operating model, generating expert call monitoring data based on the real-time monitoring results, and then sending the generated expert call monitoring data to the host side, such as the CPU side. The host side can receive the expert call monitoring data generated by the performance monitoring unit in real-time monitoring of the current operating model, and perform in-depth analysis based on the received expert call monitoring data. Optionally, the host side can export the expert call monitoring data to memory, and generate a software-compilable executable expert network execution configuration file from the exported expert call monitoring data.

[0023] S120. Determine the high-frequency expert combination based on the configuration file of the expert network.

[0024] The high-frequency expert group can be a combination of expert networks that are frequently invoked and interconnected during model execution. Optionally, the high-frequency expert group can be a combination of expert networks that are invoked sequentially at a high frequency during model execution.

[0025] S130. Adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0026] Figure 2 This is a schematic diagram of the internal structure of an LLM provided in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the unoptimized expert network code sequence layout in an existing model, provided in Embodiment 1 of the present invention. Figure 4 This is a schematic diagram illustrating the unoptimized sequential layout execution flow of expert network code in an existing model, as provided in Embodiment 1 of the present invention. In a specific example, such as... Figure 2 , Figure 3 and Figure 4 As shown, in the existing model, the unoptimized expert network code is laid out sequentially. The execution flow of the expert network in the LLM is as follows: (1) Start processing tokens; (2) Routing and selecting experts, such as selecting expert 1, expert 3 and expert 5; (3) Calling expert 1 and returning; (4) Calling expert 3 and returning; (5) Calling expert 5 and returning; (6) Ending. It can be seen that during the operation of the model network, in specific scenarios involving the calling of expert networks, some experts will be frequently called. If these frequently called experts are not optimized, due to the sequential layout of the expert network code in the existing large model, there will be a high frequency of indirect expert network calls, which will lead to the dynamic branching problem of expert network calls, such as calling expert 1 and returning, and then calling expert 3, etc., thereby increasing the overhead of dynamic indirect calls and the missing rate of instruction cache (I Cache).

[0027] To address the aforementioned technical challenges, the device-side performance monitoring unit generates expert call monitoring data by real-time monitoring of expert calls in the current running model. Based on this data, the host-side compiler parses the expert network execution configuration file to predict subsequent calls to each expert network in the current running model. This allows for the selection of frequently called and interconnected expert networks into high-frequency expert combinations. The code layout order of each expert network in the current running model is then adjusted according to these combinations to reduce indirect calls between expert networks involved in high-frequency expert combinations. This reduces dynamic branches in expert calls, enabling the most frequently and consecutively called expert networks to be called directly and sequentially. This achieves branch compilation optimization of expert networks, reducing dynamic branches in expert calls, improving model locality performance, and hiding memory latency.

[0028] Figure 5This is a schematic diagram illustrating the interaction between a performance monitoring unit and the software layer according to Embodiment 1 of the present invention. In a specific example, such as... Figure 5 As shown, the host side can receive sampling data (expert call monitoring data) sent by the performance monitoring unit through a performance sampling tool, and generate an expert network execution configuration file based on the received expert call monitoring data. The compiler running on the host side can perform compilation optimization through the expert network execution configuration file, such as predicting and selecting high-frequency expert combinations from various expert networks, and further adjusting the code layout order of each expert network in the current running model according to the selected high-frequency expert combinations, so as to realize the compilation optimization of dynamic call branches of the expert network. The optimized expert code is processed by the compiler and finally transformed into a new kernel file that can be executed on the device side. The new kernel file is the kernel code file after the expert network branches are optimized. In this way, when the device side runs the model based on the new kernel file after compilation optimization, the current running model can prioritize the execution of various instructions of the high-frequency expert networks, avoiding repeated indirect calls between different expert networks. It can be seen that the embodiment of the present invention can reduce the problem of expert network branch divergence by optimizing the call method of expert network code in model operation, thereby reducing the latency of expert network inference runtime, improving the efficiency of expert network instruction-level parallelism and pipeline, and thus improving the hardware resource utilization of expert network execution.

[0029] Therefore, the above technical solution proposes a method based on dynamic branch prediction using hardware expert networks. This method combines hardware and software co-design to predict and execute SIMT during model operation. By predicting high-frequency expert networks through hardware and software co-design and directly calling and executing them, dynamic branches caused by expert network calls can be reduced or eliminated. This avoids the delay caused by re-fetching instructions after dynamic branch jumps caused by expert network calls, and can smooth out pipeline bubbles caused by dynamic branch jumps. This can achieve efficient acceleration of model operation, reduce redundant operations and memory access power consumption, and optimize processor energy efficiency while improving performance.

[0030] This invention, through an embodiment of the present invention, acquires expert call monitoring data generated in real time by a performance monitoring unit that monitors the expert call status of the current running model. Based on this data, an expert network execution configuration file is generated, which in turn determines high-frequency expert combinations. Furthermore, the code layout order of each expert network in the current running model is adjusted according to these high-frequency expert combinations. The performance monitoring unit configures an event sampling module through a set of first registers. Each first register within this module is used to record and store expert call monitoring data in real time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire expert call-related data in real time, assisting in adjusting the code layout order of each expert network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient expert network branch compilation optimization strategies. This significantly reduces expert network branch divergence, thereby reducing latency during expert network inference runtime, improving the efficiency of expert network instruction-level parallelism and pipelines, and ultimately increasing the utilization rate of hardware resources for expert network execution.

[0031] Example 2 Figure 6 This is a flowchart of a branch compilation optimization method for expert networks provided in Embodiment 2 of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, various specific optional implementation methods are given for determining the high-frequency expert combination according to the expert network execution configuration file and adjusting the code layout order of each expert network in the current running model according to the high-frequency expert combination. Correspondingly, as Figure 6 As shown, the method in this embodiment may include: S610. Obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and generate an expert network execution configuration file based on the expert call monitoring data.

[0032] The performance monitoring unit includes an event sampling module, which includes a set of first registers, each of which is used to record and store the expert call monitoring data in real time.

[0033] S620. Generate an expert network call topology diagram based on the expert call monitoring data in the expert network execution configuration file.

[0034] The expert network call topology graph can be a type of topology graph that records information related to expert network calls, such as the call relationships and call frequencies between expert networks. The expert network call topology graph can also be called an expert call chain graph.

[0035] Since the expert call monitoring data included in the expert network execution configuration file records a snapshot of the complete expert call context, it can serve as the raw material for constructing the expert network call topology. Specifically, the expert call monitoring data in the expert network execution configuration file can be parsed to obtain the call relationships between different expert networks and the call frequency of each expert network. Based on the expert network identifiers and the parsed call relationships and call frequencies of different expert networks, an expert network call topology can be constructed and generated.

[0036] S630. Calculate the frequency of occurrence of each relation edge in the expert network call topology graph, and determine the high-frequency nodes and high-frequency relation edges based on the frequency of occurrence of each relation edge in the expert network call topology graph.

[0037] In the expert network call topology graph, the relationship edges can reflect the call relationship between two expert networks, such as the call order of expert 1 calling expert 2. Nodes in the expert network call topology graph can serve as identifiers for the expert networks. High-frequency nodes are those with a high call frequency, and high-frequency relationship edges are those with a high call frequency.

[0038] S640. Determine the high-frequency expert combination based on the high-frequency nodes and high-frequency relationship edges in the expert network topology graph.

[0039] Optionally, nodes and relationship edges in the expert network call topology graph can be statistically analyzed and labeled with information such as call frequency or number of calls based on historical expert call monitoring data of the expert network. Optionally, the weight of each node can be generated based on the call frequency, with higher call frequency resulting in higher node weight. The weight of each relationship edge can also be generated based on the call frequency, with higher call frequency resulting in higher relationship edge weight. Therefore, the most frequently called expert networks and those frequently called sequentially can be filtered out based on the weights of each node and relationship edge in the expert network call topology graph, resulting in high-frequency nodes and high-frequency relationship edges. Sequential calling can be understood as different expert networks being called consecutively, such as expert 1 frequently calling expert 2 after completing its call. Furthermore, high-frequency expert combinations can be determined based on the high-frequency nodes and high-frequency relationship edges resolved in the expert network call topology graph. That is, high-frequency expert combinations can be hotspot expert combinations, and the experts within a hotspot expert combination can be the most frequently called experts and experts frequently called "connected" together.

[0040] S650. Adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0041] In an optional embodiment of the present invention, adjusting the code layout order of each expert network in the current running model according to the high-frequency expert combination may include: rearranging the expert function code of the high-frequency expert network in the memory address space according to the high-frequency expert combination; or, adjusting the insertion code execution order of the target high-frequency expert network according to the high-frequency expert combination; wherein the code execution order adjustment instruction includes a jump instruction or a prefetch instruction.

[0042] Among these, the high-frequency expert network can be an expert network with a high execution frequency. The code execution order adjustment instruction can be an instruction that adjusts the execution order of the code. The target high-frequency expert network can be a predicted high-frequency execution expert network.

[0043] Optionally, several methods can be used to adjust the code layout order of each expert network in the current running model based on the high-frequency expert combination. One option is to directly rearrange the expert function code of the high-frequency expert network in the memory address space based on the high-frequency expert combination. That is, by using the high-frequency expert combination determined by the expert network execution configuration file, the compiler can be guided to optimize the code layout of the frequently co-occurring expert code, physically arranging the frequently called and related expert function codes together in memory, thereby transforming dynamic expert branch jumps into sequential execution and maximizing the utilization of the instruction cache.

[0044] In an optional embodiment of the present invention, the step of rearranging the expert function code of the high-frequency expert network in the memory address space according to the high-frequency expert combination may include: determining the calling order of each high-frequency expert network according to the high-frequency expert combination; determining the code concatenation order of the expert function code of each high-frequency expert network in the memory address space according to the calling order of each high-frequency expert network; and concatenating and storing the expert function code of each high-frequency expert network in the memory address space according to the code concatenation order of the expert function code of each high-frequency expert network in the memory address space.

[0045] Optionally, the high-frequency expert group can reflect information such as the hot path and co-occurrence pattern of expert calls. Therefore, the calling order of each high-frequency expert network can be determined based on the arrangement order of each expert network within the high-frequency expert group, and then the code concatenation order of the expert function code of each high-frequency expert network in the memory address space can be determined according to the calling order of each high-frequency expert network. Optionally, the calling order of the high-frequency expert networks can be directly used as the code concatenation order of the expert function code of each high-frequency expert network in the memory address space. Furthermore, according to the code concatenation order of the expert function code of each high-frequency expert network in the memory address space, the expert function code of each high-frequency expert network can be concatenated and stored sequentially in the memory address space, so that the expert function code of the high-frequency expert networks is physically arranged together in memory.

[0046] Alternatively, based on the predicted high-frequency expert combinations and the current running state of the current model, the expert networks to be executed next can be predicted and selected as target high-frequency expert networks. Then, instructions such as jump instructions or prefetch instructions are inserted into these target high-frequency expert networks to adjust the code execution order. These instructions arrange frequently called, interconnected expert function codes together in the logical space, transforming dynamic expert branch jumps into sequential execution and maximizing the utilization of the instruction cache. Jump instructions can directly jump to the corresponding expert function code of the expert network to execute it. Prefetch instructions can prefetch the expert function code of the expert network into the cache before running the expert network, thus preloading the various instructions for the expert network's runtime. This method of inserting jump instructions or prefetch instructions to adjust the code execution order does not require changing the arrangement of the expert network code in the physical memory space, thus converting the existing indirect expert network calling method into a direct calling method, making the optimization process simpler.

[0047] Figure 7 This is a schematic diagram of the optimized expert network code sequence layout provided in Embodiment 2 of the present invention. Figure 8 This is a schematic diagram of the optimized expert network code sequential layout execution flow in a model provided in Embodiment 2 of the present invention. In a specific example, such as Figure 7 and Figure 8As shown, the expert route distribution event is analyzed based on the expert call monitoring data obtained by the performance monitoring unit, and the high-frequency expert combination is determined. Assuming that the high-frequency expert combination is expert 1, expert 3 and expert 5, the compiler can be guided to optimize the code layout of the expert network, and the expert function code of expert 1, expert 3 and expert 5 is rearranged in the memory address space so that the expert function code of expert 1, expert 3 and expert 5 is physically arranged together in memory. Accordingly, during the model operation, the execution flow of the expert network in LLM is as follows: (1) Start processing token; (2) Route and select experts, such as selecting expert 1, expert 3 and expert 5; (3) Enter expert 1 in sequence; (4) Automatically fall into expert 3 without jumping; (5) Automatically fall into expert 5 without jumping; (6) End.

[0048] In summary, by utilizing the hot path and co-occurrence pattern information of expert calls collected by the performance monitoring unit hardware to generate an expert network execution configuration file, and then guiding the compiler to parse and determine high-frequency expert combinations through the expert network execution configuration file, the code layout of high-frequency expert combinations is optimized. Frequently called and related expert function codes are physically arranged together in memory or direct calls to expert network code are implemented, generating more efficient code. This significantly reduces the branch overhead of dynamic indirect calls in the expert network and the instruction cache miss rate, greatly improves the instruction cache hit rate, and reduces the possibility of instruction page faults. This achieves the effects of reducing dynamic branches, improving locality of execution performance, and hiding memory latency.

[0049] Example 3 Figure 9 This is a flowchart of a branch compilation optimization method for an expert network provided in Embodiment 3 of the present invention. This embodiment is based on the above embodiment and is further specified. In this embodiment, several specific optional implementation methods for compiling and optimizing the internal branch structure of the expert network are given. Accordingly, such as Figure 9 As shown, the method in this embodiment may include: S910. Obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and generate an expert network execution configuration file based on the expert call monitoring data.

[0050] S920. Determine the high-frequency expert combination according to the expert network execution configuration file, and adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0051] S930. During the operation of the current expert network, obtain the model branch monitoring data generated by the performance monitoring unit in real time monitoring the current expert network.

[0052] The performance monitoring unit further includes a branch recording module, which includes a set of second registers. Each second register uses a circular buffer to record and store the model branch monitoring data in real time.

[0053] The model branch monitoring data can be data obtained by the performance monitoring unit monitoring the branch status of the currently running model. The branch recording module can be a hardware module used to generate model branch monitoring data in real time. The second register can be a register that constitutes the branch recording module.

[0054] Understandably, each expert network in MoE is a complete small feedforward network with its own weights and the ability to independently perform feature transformations. Expert networks are not simply a single neuron within a larger model, but rather miniature networks with their own parameters, nonlinear transformations, and representational capabilities. Therefore, during operation, expert networks may branch, meaning they may execute different paths. To further improve model performance, the branching behavior within expert networks can be monitored in real-time during operation, allowing for compilation and optimization of the branch code.

[0055] In embodiments of the present invention, such as Figure 5 As shown, the performance monitoring unit can also be configured with a branch recording module. This module can consist of a set of second registers, each using a circular buffer to record and store model branch monitoring data in real time. For example, model branch monitoring data may include, but is not limited to, the branch instruction source address, branch instruction target address, and additional information for each branch entry. The additional information may include, but is not limited to, the branch type and execution cycle. Branch types can be, for example, CALL (function call), RET (function return), B conditional jump, and unconditional jump. The circular buffer configuration within the branch recording module effectively avoids frequent memory allocation and deallocation operations, resulting in low data write time complexity and suitability for real-time recording of high-frequency events.

[0056] S940. Generate a branch execution configuration file based on the model branch monitoring data, and determine the high-frequency execution branches among the branches to be executed in the currently running model based on the branch execution configuration file.

[0057] The branch execution configuration file can be a configuration file used as a reference for compiling and optimizing branches. Optionally, the expert network execution configuration file and the branch execution configuration file can be the same configuration file, that is, the expert network execution configuration information and the branch execution configuration information can be stored in the same configuration file. Alternatively, the expert network execution configuration file and the branch execution configuration file can also be two different configuration files, which is not limited in this embodiment of the invention. The branch to be executed can be a code branch that needs to be executed. A high-frequency execution branch can be a target execution branch, which can be a code branch that is frequently executed during program execution. That is, a high-frequency execution branch can be a code branch with a high execution frequency. The execution frequency threshold of the high-frequency execution branch can be configured according to actual needs. That is, a code branch with an execution frequency greater than a set frequency threshold can be identified as a high-frequency execution branch.

[0058] Specifically, the host side can export model branch monitoring data into memory and generate a branch execution configuration file that can be compiled and executed by the software. Correspondingly, the host side can use the compiler to parse the branch execution configuration file to predict and judge the branch execution frequency of each branch to be executed within the current expert network, thereby selecting the high-frequency execution branches from among the branches to be executed.

[0059] S950. Adjust the instruction execution order of the high-frequency execution branch so as to prioritize the execution of the branch instructions of the high-frequency execution branch in the current expert network.

[0060] Among them, the branch instructions of the high-frequency execution branch can be each instruction on the high-frequency execution branch path, and the branch instructions can also be called path instructions.

[0061] In a specific example, such as Figure 5As shown, during the operation of the current expert network, the host side can receive sampled data (model branch monitoring data) sent by the performance monitoring unit after real-time monitoring of the internal operation of the current expert network through a performance sampling tool. Based on the received model branch monitoring data, a branch execution configuration file is generated. The compiler running on the host side can perform compilation optimization using the branch execution configuration file, such as predicting and selecting high-frequency execution branches from each branch to be executed, and further adjusting the instruction execution order of the high-frequency execution branches. This allows the branch instructions of the high-frequency execution branches within the current expert network to be executed first, thus achieving compilation optimization of the high-frequency execution branches within the expert network. The optimized branch code within the expert network is processed by the compiler and finally transformed into a new kernel file that can be executed on the device side. The new kernel file is the branch-optimized kernel code file. Therefore, when the device side runs the model based on the newly compiled and optimized kernel file, the current expert network can prioritize the execution of each instruction of the high-frequency execution branches within the network. Therefore, the embodiments of the present invention can realize the shift from instruction branch prediction within the network to data access pattern prediction and computation path inference, so as to optimize the layout of the expert network's running code, reduce dynamic branches within the network, eliminate SIMT divergence through vectorization, and achieve near-peak computing power utilization.

[0062] Currently, for network models, the device side typically manages warp branches through a SIMT stack, serializing branch execution using predicate registers and execution masks. The execution time is the sum of multiple branch paths, leading to low computational efficiency. For the device side, Diverge / Converge is a method for implementing thread-level branch management using masks and synchronization stacks. For example, using IPDom (Immediate Post Dominator) to implement convergence limits the stack depth to nested levels, typically 8-32 levels. While the Diverge / Converge mechanism allows the SIMT architecture to efficiently handle conditional branches, severe branch divergence can significantly degrade performance. During model deployment, operator scheduling strategies, memory access optimizations, and hardware adaptability are bottlenecks restricting inference throughput and latency.

[0063] The above technical solution proposes a hardware branch prediction method. This method combines hardware and software co-design to predict and execute SIMT during the operation of the expert network. By predicting and executing high-frequency branches within the expert network through hardware and software collaboration, it can reduce or eliminate branches within the expert network. This avoids the latency caused by waiting for branch parsing and re-fetching instructions after branch jumps, and can smooth out pipeline bubbles caused by branch jumps within the expert network. It can achieve efficient acceleration of the expert network operation process, thereby further reducing redundant computation and memory access power consumption, and optimizing processor energy efficiency while improving performance.

[0064] In an optional embodiment of the present invention, adjusting the instruction execution order of the high-frequency execution branch may include: determining the branch compilation execution mode based on the number of instruction entries in the high-frequency execution branch; and adjusting the instruction arrangement order of the high-frequency execution branch based on the type of the branch compilation execution mode.

[0065] After identifying the high-frequency execution branches (hereinafter referred to as high-frequency execution branches) within the current expert network, the branch compilation and execution mode can be determined based on the number of instruction entries in the high-frequency execution branches. Different branch compilation and execution modes result in different methods for compiling and optimizing the branches. In this embodiment of the invention, optionally, the branch compilation and execution mode may include two compilation and execution modes: one with a Converge Barrier and one without a Converge Barrier.

[0066] Optionally, if the number of instruction entries in frequently executed branches is small, a convergence barrier-free mode without Converge Barrier can be selected; if the number of instruction entries in frequently executed branches is large, a convergence barrier mode with Converge Barrier can be selected. In the convergence barrier-free mode without Converge Barrier, there is no need to explicitly insert a convergence point. When an exit instruction is encountered, it checks if there are any unexecuted branches in the stack. If so, the unexecuted branch is popped and executed, achieving automatic divergence and convergence. The converged instructions need to be executed multiple times according to different branch masks. The convergence barrier mode with Converge Barrier explicitly defines the branch convergence point through independent convergence barriers and participation masks for each thread bundle, supporting independent thread scheduling and arbitrary depth of branch nesting.

[0067] Different branch compilation execution modes require different methods for adjusting and optimizing the instruction order of frequently executed branches. Regardless of the branch compilation execution mode used, corresponding compilation optimization strategies can be employed to adjust the instruction order of frequently executed branches, ensuring that branch instructions in these branches are executed first.

[0068] In an optional embodiment of the present invention, adjusting the instruction arrangement order of the high-frequency execution branches according to the type of the branch compilation execution mode may include: when the branch compilation execution mode is determined to be a convergence barrier mode, inserting a first convergence barrier instruction before each of the branches to be executed, inserting a second convergence barrier instruction before the original convergence instruction of each of the branches to be executed, and inserting an unconditional jump instruction at the end of the non-high-frequency execution branches; writing the currently active thread mask into the target thread mask of the convergence barrier, and clearing the completed thread mask of the convergence barrier to zero; during instruction execution, prioritizing the execution of the path instructions of the high-frequency execution branches, and setting the program counter (PC) of the first instruction of the non-high-frequency execution branches to zero. The program counter (PC) and active thread mask are pushed onto the stack; after the instructions of the high-frequency execution branch are executed, the active thread mask of the instructions of the high-frequency execution branch is subjected to convergence barrier processing according to the second convergence barrier instruction; based on the convergence barrier processing result, the PC and active thread mask of the first instruction of the non-high-frequency execution branch are popped from the stack to execute the instructions of the non-high-frequency execution branch path; after the instructions of the non-high-frequency execution branch are executed, convergence barrier processing is performed according to the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

[0069] The first convergence barrier instruction can be a convergege instruction used to implement a Join operation. The original convergence instruction can be the instruction executed after the branches merge. The second convergence barrier instruction can be a convergege instruction used to implement a Sync operation. A non-high-frequency execution branch can be a branch of another execution path corresponding to a high-frequency execution branch. An unconditional jump instruction can be an instruction that jumps without any conditional constraints. The currently active thread mask can be the thread mask of the currently executing instruction. Path instructions can refer to all instructions involved in a branch path. The first instruction is also the first instruction of the branch.

[0070] Figure 10 This is a flowchart illustrating branch compilation optimization in an expert network under a convergence barrier mode, as provided in Embodiment 3 of the present invention. In a specific example, such as... Figure 10As shown, A through H each represent an instruction within the expert network. Here, we'll use a thread warp value of 4 as an example. Since instruction B is a branch decision instruction (the instruction is B.EQ, i.e., the EQ mode of instruction B), in convergence barrier mode, the compiler inserts a first convergence barrier instruction before the branch to be executed, a second convergence barrier instruction before the original convergence instruction of each branch to be executed, and an unconditional jump instruction at the end of non-frequently executed branches. That is, instruction A (CB.Join) is inserted before instruction B, instruction F is inserted before instruction G, and instruction D is inserted at the end of instruction C. Instructions A and F are specifically convergence instructions with two different operations; that is, the first convergence barrier instruction can be of the CB.Join instruction type, and the second convergence barrier instruction can be of the CB.Sync instruction type. Instruction D can be an unconditional jump instruction. Specifically, instruction D is B.EQ.F, indicating an unconditional jump to instruction F.

[0071] Converge Barrier can be a barrier counter that occupies a continuous 64-bit address space and is stored in a scalar register file. It is used to optimize the merging behavior of warp branches caused by conditional jumps and can realize thread bundle merging at the beginning and end of thread bundle branches. Converge Barrier consists of two WARP_SIZE (thread bundle size) bits of Thread BitMask, namely the Target Mask representing the target convergence Bit Mask and the Arrive Mask representing the Bit Mask of the threads that have reached the convergence point. It is stored in two consecutive scalar registers with IDs of 2N and 2N+1 using SRT (Scalar Register). There are three operations of Converge Barrier: (1) Join operation: used for the current warp branch. Before branch divergence occurs, Converge Barrier is initialized. The Active Thread Mask of the current warp branch is written to the Target Thread Mask, and the Arrive Mask is cleared. (2) Sync operation: Used for the currently running warp branch, performing an OR operation between the current Active Thread Mask and the ArriveMask of the Converge Barrier. If the ArriveMask and the Target Mask are equal after the operation, the Active Thread Mask of the current warp branch is rewritten as the Target Mask, the convergence is complete, and the current warp branch continues execution. Otherwise, the SIMT Stack is popped, and the warp branch held at the top of the SIMT Stack is executed instead. (3) Break operation: Used to remove the Active Thread Mask of the current warp branch from the Target Mask of the Converge Barrier, and the current warp branch continues execution.

[0072] Table 1. List of components of the converge instruction Table 2 List of operation types for the converge command Table 1 is a list of the structural components of a converge instruction provided in Embodiment 3 of the present invention, and Table 2 is a list of the operation types of a converge instruction provided in Embodiment 3 of the present invention. In a specific example, as shown in Tables 1 and 2, the converge instruction is a 32-bit instruction. The first 7 bits of this convergence barrier instruction are identified by 0x20 as a dedicated convergence barrier instruction. This dedicated convergence barrier instruction is used to read the Target Mask and the Arrived Mask respectively based on two consecutive scalar registers, 2N and 2N+1, represented by the SRT register, a scalar register. TYPE represents a convergence barrier instruction type described by 2 bits of data: 0 represents JOIN, 1 represents SYNC, and 2 represents BREAK. NA indicates that the data bits are meaningless. Bits [10:8] of the instruction are assigned the constant 0x1.

[0073] The Diverge instruction, often abbreviated as B instruction, determines whether a branch exists during a conditional check and executes that branch. The Diverge instruction reads the Vector registers RS (Source Register) and RT (Target Register), performs a 32-bit signed comparison between RS and RT based on the comparison condition COND, and determines which branch to follow based on the comparison result. It then updates the Program Counter (PC) of each thread (Thread) based on the comparison result and an 8-bit signed immediate value. For all active threads in the current warp branch, it determines whether to jump to "current PC (address of the current instruction) + IMM (immediate value)" based on the vector comparison result. The address is 4". When some threads choose to jump and others choose to execute sequentially, the current warp branch will branch; otherwise, no branch will occur. RS / RT can be a vector register or a scalar register. When a branch occurs, the number of threads in each of the two branches is counted after the active threads branch. The branch with fewer threads can be taken as the new current warp branch, i.e., the high-frequency execution branch. If the number of active threads in each of the two branches is equal, the branch that executes sequentially is selected. After determining the branch direction, the current PC is updated, and the active thread mask of the current warp is updated with the set of threads that selected the branch. The target PC and the mask corresponding to the set of threads of the other unselected branch are pushed onto the SIMT Stack. They are not executed until all threads of the selected branch have finished executing or a Converge instruction is encountered, at which point the selected branch is popped from the SIMT Stack and executed as the current warp branch.

[0074] Table 3 is a list of the composition structure of a Diverge instruction provided in Embodiment 3 of the present invention, Table 4 is a list of the composition structure of a Diverge instruction provided in Embodiment 3 of the present invention, and Table 5 is a list of the operation types of a Diverge instruction provided in Embodiment 3 of the present invention. As shown in Tables 3 and 4, the instruction bits [31:0] in the first row of Tables 3 and 4 indicate that the branch judgment instruction is a 32-bit instruction. The second row refers to the physical meaning of the data filled in different instruction bits. For example, "0x10" filled in the [6:2] position indicates that bits 2 to 6 of the instruction are the instruction identifier of the branch judgment instruction. SRS[6:0] represents a source operand register number described using 7 bits of data; this source operand is a scalar register. VRS[7:0] represents a source operand register described using 8 bits of data; this source operand register is a vector register. VRT[7:0] represents a destination operand register described using 8 bits of data; this destination operand register is a vector register. SRT[6:0] represents a destination operand register described using 7 bits of data; this destination operand register is a scalar register. NA indicates that the data bits are meaningless. COND represents a comparison condition described using 2 bits of data: EQ for equal to, LT for less than, and GE for greater than or equal to. IMM represents an immediate value described using 8 bits of data. Unconditional jump instructions are a special instruction mode of branch judgment instructions.

[0075] Table 3. List of the components of the Diverge command Table 4. List of the components of the Diverge command Table 5. List of operation types for the Diverge command For example, such as Figure 10 As shown, after the insertion of the two convergence barrier instructions, A and F, the thread mask for instruction A is 1111, and four threads execute instruction A in parallel. The active thread mask is written to the convergence barrier's target mask (1111), and the convergence barrier's arrive mask (completed mask) is cleared. Simultaneously, instruction F (CB.Sync instruction) is inserted before instruction G at the convergence point of instructions C and E. After instruction B is executed, two branches appear: C (thread mask 0111) and E (thread mask 1000), two ADD instructions. If the high-frequency execution branch is predicted based on the number of threads occupied by the branch, the branch with fewer active threads is selected as the predicted high-frequency execution branch. In the thread mask, 1 represents an active thread, and 0 represents an inactive thread. Therefore, instruction E on the right side can be predicted as the high-frequency execution branch and executed first. At the same time, the PC corresponding to instruction C and the active thread mask (0111) are pushed onto the SIMT stack. After executing instruction E, the second convergence barrier instruction, instruction F, is executed to perform convergence barrier processing on the active thread mask of instructions in high-frequency execution branches.

[0076] In an optional embodiment of the present invention, the step of performing convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction may include: performing an OR operation on the active thread mask of the instructions of the high-frequency execution branch and the completed thread mask according to the second convergence barrier instruction to obtain an updated completed thread mask; comparing the updated completed thread mask with the target thread mask to obtain the convergence barrier processing result.

[0077] Among them, the updated completed thread mask is the updated completed thread mask.

[0078] For example, such as Figure 10As shown, when performing convergence barrier processing on the active thread mask of the E instruction in a high-frequency execution branch according to the second convergence barrier instruction CB.Sync, the CB.Sync instruction can either the current active thread mask ActiveMask (1000) or the completed thread mask Arrive Mask of the convergence barrier, thus updating Arrive Mask to 1000. Further, it checks whether Arrive Mask is equal to Target Mask, obtaining the convergence barrier processing result if they are not equal. At this point, based on this convergence barrier processing result, the non-high-frequency execution branch in the SIMT stack is popped ( Figure 10 The first instruction (the C instruction) on the left branch (corresponding to the PC and Active Mask) begins execution of the C instruction. After executing the C instruction, the D instruction continues execution, unconditionally transferring to the F instruction via the D instruction.

[0079] Correspondingly, after the instructions in the non-high-frequency execution branch have been executed, an unconditional jump instruction can be used to unconditionally transfer to the second convergence barrier instruction. Based on the second convergence barrier instruction, the active thread mask of the instructions in the non-high-frequency execution branch is ORed with the completed thread mask to obtain an updated completed thread mask. The updated completed thread mask is then compared with the target thread mask to obtain the convergence barrier processing result. For example, as shown... Figure 10 As shown, after the instructions of the non-high-frequency execution branch path on the left are executed, the Arrive Mask of the convergence barrier is updated again. The current Active Mask (0111) is ORed with the Arrive Mask (1000) to obtain the updated Active Mask (1111), which is equal to the Target Mask (1111), thus completing the convergence. After all threads perform convergence, they continue to execute G and H instructions until they exit.

[0080] In summary, based on the model branch monitoring data obtained by the performance monitoring unit, the high-frequency execution branches within the expert network can be identified. For example, if the ADD2 branch executes before the ADD1 branch, then the ADD2 branch is determined to be a high-frequency execution branch. Furthermore, the compiler rearranges the instruction code of the high-frequency execution branches within the expert network, placing the ADD2 and converge instructions after the branch decision instructions, thereby reducing pipeline bubbles caused by instruction jumps within the network. When rearranging the instruction code of the high-frequency execution branches, the compiler can inline or rearrange the actual execution order of the branch paths, thus ensuring continuous hot paths and reducing the number of branches.

[0081] In an optional embodiment of the present invention, adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode may include: when it is determined that the branch compilation execution mode is a non-convergence barrier mode, inserting an unconditional jump instruction at the end of the non-high-frequency execution branch, prioritizing the execution of the path instructions of the high-frequency execution branch, and pushing the PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after it is determined that the path instructions of the high-frequency execution branch have been executed, popping the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack to execute the path instructions of the non-high-frequency execution branch.

[0082] Figure 11 This is a flowchart illustrating branch compilation optimization of an expert network in a convergence barrier-free mode, as provided in Embodiment 3 of the present invention. In a specific example, such as... Figure 11 As shown, A through F each represent an instruction within the expert network. Here, we take a thread warp value of 4 as an example. Since A is a branch decision instruction (instruction B.EQ, i.e., the EQ mode of instruction B), the thread mask for instruction A is 1111, and four threads execute instruction A in parallel. Because it's a no-convergence-barrier mode, the compiler won't insert a Converge instruction, but it will insert an unconditional jump instruction C (B.EQ.E, indicating an unconditional jump to instruction E) at the end of non-high-frequency execution branches. After instruction A is executed, two branches appear: B (thread mask 0111) and D (thread mask 1000), two ADD instructions. If we predict the high-frequency execution branch based on the number of threads occupied by the branch, the branch with fewer active threads is chosen as the predicted high-frequency execution branch. That is, instruction D can be predicted as a high-frequency execution branch and executed first. Simultaneously, the PC and Active Mask (0111) corresponding to instruction B are pushed onto the SIMT stack. After executing instruction D, instruction E is executed, followed by instruction F (exit). Instructions D, E, and F are the path instructions for the frequently executed branches on the right. Further, it checks if the SIMT stack is empty. If not, the PC and Active Mask corresponding to instruction B are popped from the SIMT stack, and instruction B is executed. After executing instruction B, the unconditional jump instruction C is executed, unconditionally transferring to instruction E. Instruction F (exit) is then executed, and it checks if the SIMT stack is empty. If it is, the program exits. Therefore, in the no-convergence-barrier mode, rearranging the branch path code based on the model branch monitoring data obtained by the performance monitoring unit can reduce instruction jumps and latency caused by cache misses.

[0083] Optionally, each SM (Streaming Multiprocessor) subcore can use scalar registers to build convergence barriers and SIMT stacks, without occupying stack storage space, allowing for flexible execution granularity, and enabling automatic push and pop operations via hardware.

[0084] In an optional embodiment of the present invention, the method may further include: inserting a prefetch instruction before executing the high-frequency execution branch; during instruction execution, if it is determined that the current instruction is the prefetch instruction, parsing the prefetch instruction to obtain the address of the target prefetch instruction; in response to the prefetch instruction, initiating an instruction prefetch request and loading the target prefetch instruction into the instruction cache; wherein the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; if it is determined that the current instruction is a high-frequency execution branch instruction, reading the cached target prefetch instruction from the instruction cache and directly executing the target prefetch instruction.

[0085] In this context, a prefetch instruction can be used to prefetch data or instructions. The target prefetch instruction address can be the memory address used to fetch the target prefetch instruction. An instruction prefetch request can be used to request prefetch instructions. The target prefetch instruction is the instruction that needs to be prefetched; it can be some or all of the instructions in a frequently executed branch.

[0086] To further optimize the compilation of branches running within the expert network, after adjusting the instruction arrangement order of high-frequency execution branches within the expert network according to the type of branch compilation execution mode, the high-frequency execution branch path to be executed within the expert network can be determined based on the actual execution order of the branch path instructions. Then, the compiler inserts prefetch instructions before the instructions of the current high-frequency execution branch path to explicitly notify the hardware to prefetch the high-frequency execution branch instructions, thereby improving hardware efficiency.

[0087] Figure 12 This is a schematic diagram of a high-frequency execution branch instruction prefetching process provided in Embodiment 3 of the present invention. In a specific example, such as Figure 12As shown, during the process of executing expert network instructions by the kernel after compilation and optimization on the device side, it can be determined whether the current instruction is a prefetch instruction. If so, the prefetch instruction is parsed to obtain the address of the target prefetch instruction, and an instruction prefetch request is initiated to the memory subsystem in response to the prefetch instruction. This requests the memory subsystem to load the cache line containing the target prefetch instruction into the instruction cache (I Cache), and then the execution of subsequent instructions continues. It is understood that the target prefetch instruction can be part or all of the instructions in a high-frequency execution branch. The number of instruction entries included in the target prefetch instruction can be configured according to specific hardware parameters, and this embodiment of the invention does not limit this. If the current instruction is not a prefetch instruction, it can be further determined whether the current instruction is a high-frequency execution branch instruction. If so, it can be further checked whether the target prefetch instruction is cached in the I Cache. If a match is found, the target prefetch instruction is executed directly; if it is missing, an instruction loading request is initiated to load the missing target prefetch instruction from memory into the I Cache. If the current instruction is not a high-frequency execution branch instruction, the execution of subsequent instructions continues.

[0088] It is evident that the above scheme improves pipeline efficiency by inserting prefetch instructions to prefetch instructions for high-frequency execution branches, thereby enhancing the model's running efficiency and hardware utilization.

[0089] Prefetch instructions, used to prefetch both data and instructions, are called prefetch instructions. Their data cache prefetch function reads a contiguous file of vector registers with IDs 2N and 2N+1 specified by the Vector register VRT (destination operand register in vector form), forming a 64-bit vector base address. This base address is then added to a 32-bit offset address in the register specified by the SRS (source operand register number in scalar form) to form a 64-bit memory access address. The TYPE field defines which cache level (L1, L2, or L3) the data is prefetched into. Their instruction cache prefetch function reads a contiguous file of vector registers with IDs 2N and 2N+1 specified by the Vector register VRT, forming a 64-bit vector base address. This base address is then added to an IMM offset address to form a 64-bit memory access address PC. NUM instructions are prefetched into the I Cache.

[0090] Table 6. List of components of the data prefetch command Table 7. List of components of the prefetch command Table 8. List of possible values ​​for the TYPE field in the prefetch command. Table 9. List of possible values ​​for the SIZE field in the prefetch command. Table 10 List of NUM field values ​​for the prefetch command Table 6 is a list of the components of a prefetch instruction provided in Embodiment 3 of the present invention; Table 7 is a list of the components of a prefetch instruction provided in Embodiment 3 of the present invention; Table 8 is a list of values ​​for the TYPE field in a prefetch instruction provided in Embodiment 3 of the present invention; Table 9 is a list of values ​​for the SIZE field in a prefetch instruction provided in Embodiment 3 of the present invention; and Table 10 is a list of values ​​for the NUM field in a prefetch instruction provided in Embodiment 3 of the present invention. That is, prefetch instructions can include two types: data prefetch instructions and instruction prefetch instructions. Data prefetch instructions can be used to prefetch data, and instruction prefetch instructions can be used to prefetch instructions. In Tables 6 and 7, the instruction bits [31:0] in the first row represent that the prefetch instruction is a 32-bit instruction. The second row refers to the physical meaning of the data filled under different instruction bits. For example, "0x30" filled at position [6:2] represents that bits 2 to 6 of the instruction are the instruction identifier of the prefetch instruction. VRT[7:1] represents a source operand register described using 8 bits of data. This source operand register is a vector register used to fetch 2N and 2N+1 consecutive vector registers to form a 64-bit base address. SRS[6:0] represents a source operand register number described using 7 bits of data. This source operand is a scalar register representing the offset address. NA indicates that the data bits are meaningless. As shown in Table 8, TYPE represents a cache type described using 2 bits of data, where 0 indicates reading from L1 Cache, 1 indicates reading from L2 Cache, and 2 indicates reading from L3 Cache. As shown in Table 9, SIZE represents a prefetch data type described using 2 bits of data. 0 indicates a 32-bit Word (W), 1 indicates a 16-bit Half Word (H), 2 indicates an 8-bit signed Byte (B), and 3 indicates an 8-bit unsigned Byte (BU). NUM represents the number of prefetch instructions described using 2 bits: 0 for 1 instruction, 1 for 2 instructions, 2 for 4 instructions, and 3 for 8 instructions. IMM represents the immediate value.

[0091] This invention proposes a hardware and software architecture for accelerating inference in expert network models, which can be integrated into existing deep learning frameworks. This architecture uses a performance monitoring unit to monitor the expert network in real time and generate model branch monitoring data. The compiler, based on the acquired model branch monitoring data and combining efficient Diverge / Converge / Prefetch instructions with compilation optimization strategies, performs targeted optimization on the expert network's model branch code. It predicts high-frequency execution branches within the expert network based on branch execution probabilities and, combined with prefetch instructions, prefetches and executes instructions for high-frequency execution branch paths. This shifts the focus from instruction branch prediction to data access pattern prediction and computation path inference. Prefetching and vectorization eliminate SIMT divergence, optimize the code layout within the expert network, and reduce dynamic branches, thereby reducing latency during expert network inference runtime. This improves the efficiency of instruction-level parallelism and pipelines, maximizes computational and memory access efficiency, and ultimately enhances the expert network's execution efficiency and hardware resource utilization, achieving near-peak computing power utilization.

[0092] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0093] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0094] Example 4 Figure 13 This is a schematic diagram of a branch compilation optimization device for an expert network provided in Embodiment 4 of the present invention, as shown below. Figure 13 As shown, the device includes: an expert network execution configuration file generation module 1310, a high-frequency expert combination determination module 1320, and a code layout order adjustment module 1330, wherein: The expert network execution configuration file generation module 1310 is used to obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and to generate an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; The high-frequency expert combination determination module 1320 is used to determine the high-frequency expert combination based on the expert network execution configuration file. The code layout order adjustment module 1330 is used to adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination.

[0095] This invention, through an embodiment of the present invention, acquires expert call monitoring data generated in real time by a performance monitoring unit that monitors the expert call status of the current running model. Based on this data, an expert network execution configuration file is generated, which in turn determines high-frequency expert combinations. Furthermore, the code layout order of each expert network in the current running model is adjusted according to these high-frequency expert combinations. The performance monitoring unit configures an event sampling module through a set of first registers. Each first register within this module is used to record and store expert call monitoring data in real time. Therefore, the above technical solution utilizes a hardware-based performance monitoring unit to monitor and acquire expert call-related data in real time, assisting in adjusting the code layout order of each expert network in the model. This achieves hardware-software co-design, combining hardware monitoring data with efficient expert network branch compilation optimization strategies. This significantly reduces expert network branch divergence, thereby reducing latency during expert network inference runtime, improving the efficiency of expert network instruction-level parallelism and pipelines, and ultimately increasing the utilization rate of hardware resources for expert network execution.

[0096] Optionally, the high-frequency expert combination determination module 1320 is further configured to: generate an expert network call topology graph based on the expert call monitoring data in the expert network execution configuration file; count the occurrence frequency of each relation edge in the expert network call topology graph; determine high-frequency nodes and high-frequency relation edges based on the occurrence frequency of each relation edge in the expert network call topology graph; and determine the high-frequency expert combination based on the high-frequency nodes and high-frequency relation edges in the expert network call topology graph.

[0097] Optionally, the code layout order adjustment module 1330 is further configured to: rearrange the expert function code of the high-frequency expert network in the memory address space according to the high-frequency expert combination; or, execute an order adjustment instruction on the insertion code of the target high-frequency expert network according to the high-frequency expert combination; wherein the code execution order adjustment instruction includes a jump instruction or a prefetch instruction.

[0098] Optionally, the code layout order adjustment module 1330 is further configured to: determine the calling order of each high-frequency expert network according to the high-frequency expert combination; determine the code concatenation order of the expert function code of each high-frequency expert network in the memory address space according to the calling order of each high-frequency expert network; and concatenate and store the expert function code of each high-frequency expert network in the memory address space according to the code concatenation order of the expert function code of each high-frequency expert network in the memory address space.

[0099] Optionally, the above apparatus further includes an expert network branch optimization module, used for: acquiring model branch monitoring data generated by the performance monitoring unit in real-time monitoring of the current expert network during the operation of the current expert network; wherein, the performance monitoring unit further includes a branch recording module, the branch recording module includes a set of second registers, each of the second registers using a circular buffer to record and store the model branch monitoring data in real time; generating a branch execution configuration file based on the model branch monitoring data; determining the high-frequency execution branches among the branches to be executed in the currently running model based on the branch execution configuration file; and adjusting the instruction execution order of the high-frequency execution branches to prioritize the execution of branch instructions of the high-frequency execution branches in the current expert network.

[0100] Optionally, the expert network branch optimization module is further configured to: determine the branch compilation and execution mode based on the number of instruction entries in the high-frequency execution branch; and adjust the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation and execution mode.

[0101] Optionally, the expert network branch optimization module is further configured to: when the branch compilation and execution mode is determined to be a convergence barrier mode, insert a first convergence barrier instruction before each of the branches to be executed, insert a second convergence barrier instruction before the original convergence instruction of each of the branches to be executed, and insert an unconditional jump instruction at the end of the non-high-frequency execution branch; write the currently active thread mask into the target thread mask of the convergence barrier, and clear the completed thread mask of the convergence barrier; during instruction execution, prioritize the execution of the path instructions of the high-frequency execution branch, and push the program counter PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after the instructions of the high-frequency execution branch are executed, perform convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction; pop the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack according to the convergence barrier processing result, so as to execute the instructions of the non-high-frequency execution branch path; after the instructions of the non-high-frequency execution branch are executed, perform convergence barrier processing according to the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

[0102] Optionally, the expert network branch optimization module is further configured to: perform an OR operation on the active thread mask and the completed thread mask of the high-frequency execution branch according to the second convergence barrier instruction to obtain an updated completed thread mask; and compare the updated completed thread mask with the target thread mask to obtain the convergence barrier processing result.

[0103] Optionally, the expert network branch optimization module is further configured to: when it is determined that the branch compilation and execution mode is a non-convergence barrier mode, insert an unconditional jump instruction at the end of the non-high-frequency execution branch, prioritize the execution of the path instructions of the high-frequency execution branch, and push the PC and active thread mask of the first instruction of the non-high-frequency execution branch onto the stack; after it is determined that the path instructions of the high-frequency execution branch have been executed, pop the PC and active thread mask of the first instruction of the non-high-frequency execution branch from the stack to execute the path instructions of the non-high-frequency execution branch.

[0104] Optionally, the expert network branch optimization module is further configured to: insert a prefetch instruction before executing the high-frequency execution branch; during instruction execution, if it is determined that the current instruction is the prefetch instruction, parse the prefetch instruction to obtain the address of the target prefetch instruction; in response to the prefetch instruction, initiate an instruction prefetch request and load the target prefetch instruction into the instruction cache; wherein the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; if it is determined that the current instruction is a high-frequency execution branch instruction, read the cached target prefetch instruction from the instruction cache and directly execute the target prefetch instruction.

[0105] The aforementioned expert network branch compilation optimization apparatus can execute the expert network branch compilation optimization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the expert network branch compilation optimization method provided in any embodiment of the present invention.

[0106] Since the expert network branch compilation optimization apparatus described above is an apparatus capable of executing the expert network branch compilation optimization method in the embodiments of the present invention, those skilled in the art can understand the specific implementation and various variations of the expert network branch compilation optimization apparatus of this embodiment based on the expert network branch compilation optimization method described in the embodiments of the present invention. Therefore, how the expert network branch compilation optimization apparatus implements the expert network branch compilation optimization method in the embodiments of the present invention will not be described in detail here. Any apparatus used by those skilled in the art to implement the expert network branch compilation optimization method in the embodiments of the present invention falls within the scope of protection of this application.

[0107] Example 5 Figure 14A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0108] like Figure 14 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0109] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as branch compilation optimization methods for expert networks.

[0111] Optionally, the branch compilation optimization method for expert networks may include: obtaining expert call monitoring data generated in real time by a performance monitoring unit to monitor the expert call status of the current running model, and generating an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; determining high-frequency expert combinations based on the expert network execution configuration file; and adjusting the code layout order of each expert network in the current running model based on the high-frequency expert combinations.

[0112] In some embodiments, the branch compilation optimization method for expert networks can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the branch compilation optimization method for expert networks described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the branch compilation optimization method for expert networks by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0119] This invention also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the branch compilation optimization method for expert networks provided in any embodiment of this invention. This program product shares the same inventive concept as the branch compilation optimization methods for expert networks disclosed in the various embodiments of this invention, and therefore will not be described in detail here.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the embodiments of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in the embodiments of the present invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A branch compilation optimization method for expert networks, characterized in that, include: The performance monitoring unit acquires expert call monitoring data generated in real time by monitoring the expert call status of the current running model, and generates an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; The high-frequency expert combination is determined based on the configuration file executed by the expert network. Adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination; The step of determining the high-frequency expert combination based on the expert network execution configuration file includes: Generate an expert network call topology diagram based on the expert call monitoring data in the expert network execution configuration file; Statistically analyze the frequency of occurrence of each relation edge in the expert network call topology graph; The high-frequency nodes and high-frequency relationship edges are determined based on the frequency of occurrence of each relationship edge in the expert network call topology graph; The high-frequency expert combination is determined based on the high-frequency nodes and high-frequency relationship edges in the expert network topology graph.

2. The method according to claim 1, characterized in that, The step of adjusting the code layout order of each expert network in the current running model according to the high-frequency expert combination includes: The expert function code of the high-frequency expert network is rearranged in the memory address space according to the aforementioned high-frequency expert combination; or The high-frequency expert group performs code execution order adjustment instructions on the target high-frequency expert network according to the high-frequency expert group; wherein, the code execution order adjustment instructions include jump instructions or prefetch instructions.

3. The method according to claim 2, characterized in that, The step of rearranging the expert function code of the high-frequency expert network in the memory address space according to the high-frequency expert combination includes: The order in which each high-frequency expert network is invoked is determined based on the aforementioned high-frequency expert group; The code concatenation order of the expert function code of each high-frequency expert network in the memory address space is determined according to the calling order of each high-frequency expert network. According to the code concatenation order of the expert function codes of each high-frequency expert network in the memory address space, the expert function codes of each high-frequency expert network are concatenated and stored in the memory address space.

4. The method according to claim 1, characterized in that, The method further includes: During the operation of the current expert network, model branch monitoring data generated by the performance monitoring unit in real time is acquired; wherein, the performance monitoring unit further includes a branch recording module, the branch recording module includes a set of second registers, and each second register uses a circular buffer to record and store the model branch monitoring data in real time; Generate a branch execution configuration file based on the model branch monitoring data; The high-frequency execution branches in each branch to be executed in the current running model are determined according to the branch execution configuration file; Adjust the instruction execution order of the high-frequency execution branch so that the branch instructions of the high-frequency execution branch in the current expert network are executed first.

5. The method according to claim 4, characterized in that, The adjustment of the instruction execution order of the high-frequency execution branch includes: The branch compilation and execution mode is determined based on the number of instruction entries in the high-frequency execution branch; Adjust the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation and execution mode.

6. The method according to claim 5, characterized in that, The step of adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode includes: When the branch compilation execution mode is determined to be the convergence barrier mode, a first convergence barrier instruction is inserted before each of the branches to be executed, a second convergence barrier instruction is inserted before the original convergence instruction of each of the branches to be executed, and an unconditional jump instruction is inserted at the end of the non-high frequency execution branch. Write the currently active thread mask into the target thread mask of the convergence barrier, and clear the completed thread mask of the convergence barrier to zero; During instruction execution, the path instructions of the high-frequency execution branches are executed first, and the program counter PC and active thread mask of the first instruction of the low-frequency execution branches are pushed onto the stack. After the instructions of the high-frequency execution branch are executed, the active thread mask of the instructions of the high-frequency execution branch is subjected to convergence barrier processing according to the second convergence barrier instruction; Based on the convergence barrier processing result, pop the PC and active thread mask of the first instruction of the non-high frequency execution branch from the stack to execute the instructions of the non-high frequency execution branch path; After the instructions of the non-high-frequency execution branch are executed, a convergence barrier is performed based on the unconditional jump instruction, the second convergence barrier instruction, and the active thread mask of the instructions of the non-high-frequency execution branch.

7. The method according to claim 6, wherein performing convergence barrier processing on the active thread mask of the instructions of the high-frequency execution branch according to the second convergence barrier instruction comprises: According to the second convergence barrier instruction, the active thread mask of the instruction of the high-frequency execution branch is ORed with the completed thread mask to obtain the updated completed thread mask; The updated thread mask is compared with the target thread mask to obtain the convergence barrier processing result.

8. The method according to claim 5, characterized in that, The step of adjusting the instruction arrangement order of the high-frequency execution branch according to the type of the branch compilation execution mode includes: If the branch compilation execution mode is determined to be a non-convergence barrier mode, an unconditional jump instruction is inserted at the end of the non-high-frequency execution branch, the path instruction of the high-frequency execution branch is executed first, and the PC and active thread mask of the first instruction of the non-high-frequency execution branch are pushed onto the stack. After the path instructions of the high-frequency execution branch have been executed, the PC and active thread mask of the first instruction of the low-frequency execution branch are popped from the stack to execute the path instructions of the low-frequency execution branch.

9. The method according to any one of claims 5-8, characterized in that, Also includes: Insert a prefetch instruction before executing the high-frequency execution branch; During instruction execution, if it is determined that the current instruction is the prefetch instruction, the prefetch instruction is parsed to obtain the target prefetch instruction address; In response to the prefetch instruction, an instruction prefetch request is initiated to load the target prefetch instruction into the instruction cache; wherein, the target prefetch instruction is part or all of the instructions of the high-frequency execution branch; If it is determined that the current instruction is a high-frequency branch instruction, the target prefetch instruction is read from the instruction cache and the target prefetch instruction is executed directly.

10. A branch compilation optimization device for expert networks, characterized in that, include: An expert network execution configuration file generation module is used to obtain expert call monitoring data generated by the performance monitoring unit in real time monitoring the expert call status of the current running model, and to generate an expert network execution configuration file based on the expert call monitoring data; wherein, the performance monitoring unit includes an event sampling module, and the event sampling module includes a set of first registers, each of the first registers being used to record and store the expert call monitoring data in real time; A high-frequency expert portfolio determination module is used to determine a high-frequency expert portfolio based on the expert network execution configuration file. The code layout order adjustment module is used to adjust the code layout order of each expert network in the current running model according to the high-frequency expert combination. The high-frequency expert combination determination module is further configured to: generate an expert network call topology graph based on the expert call monitoring data in the expert network execution configuration file; count the occurrence frequency of each relationship edge in the expert network call topology graph; determine high-frequency nodes and high-frequency relationship edges based on the occurrence frequency of each relationship edge in the expert network call topology graph; and determine the high-frequency expert combination based on the high-frequency nodes and high-frequency relationship edges in the expert network call topology graph.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the branch compilation optimization method for the expert network as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the branch compilation optimization method for the expert network according to any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the branch compilation optimization method for the expert network as described in any one of claims 1-9.

Images