Microcode generation method and system based on reinforcement learning

Abstract

The invention provides a microcode generation method and system based on reinforcement learning, and relates to the technical field of data processing.The microcode generation method comprises the steps that a pre-trained reinforcement learning strategy model is adopted, a state feature vector represented in the middle of microcodes is extracted, action probability distribution is output, and a target action is selected based on the action probability distribution; according to the method and the system, the current micro-code intermediate representation is subjected to iterative updating, the model comprehensively balances the subsequent income in combination with the current state in the iterative updating process, and a better scheduling sequence can be gradually approached under the condition that the micro-operation dependency relationship and hardware resource competition coexist. Meanwhile, a better scheduling change operation is selected through iterative updating, so that unnecessary pauses and resource conflicts can be reduced, and the utilization efficiency of key execution resources is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a microcode generation method and system based on reinforcement learning. Background Technology

[0002] Microcode is a crucial intermediate layer in a computer architecture that connects the instruction set architecture (ISA) and hardware execution units. It is a core technology for complex instruction translation, hardware control, and performance optimization. The generation and optimization of microcode has always been a highly challenging technical problem, requiring consideration of multiple constraints such as hardware heterogeneity, instruction set diversity, performance, power consumption, and compatibility.

[0003] Traditional microcode generation and optimization processes rely on manually preset heuristic rules and experience-based tuning strategies, which are difficult to cope with the heterogeneity and complexity of modern processor architectures and the differentiated needs of multi-scenario applications, and are difficult to achieve near-optimal scheduling results in complex environments. Summary of the Invention

[0004] The purpose of this application is to provide a microcode generation method and system based on reinforcement learning, so as to overcome the shortcomings of traditional microcode generation and optimization processes, which rely on manually preset heuristic rules and empirical tuning strategies, making it difficult to cope with the heterogeneous and complex characteristics of modern processor architectures and the differentiated needs of multi-scenario applications, and making it difficult to stably obtain near-optimal scheduling results in complex environments.

[0005] Firstly, this application proposes a microcode generation method based on reinforcement learning, including:

[0006] Obtain the intermediate microcode representation of the target instruction to be implemented; A pre-trained reinforcement learning strategy model is used to extract the state feature vector of the current microcode intermediate representation, and the action probability distribution of each action is output based on the state feature vector; wherein, each type of action corresponds to a preset transformation operation performed on the microcode intermediate representation; The target action is selected based on the action probability distribution and applied to the current microcode intermediate representation to iteratively update the next generation of the microcode intermediate representation until the preset iteration condition is met to obtain the target microcode intermediate representation. Binary microcode is generated based on the intermediate representation of the target microcode.

[0007] In one embodiment, the method further includes: The target microcode intermediate representation is verified, and the verified target microcode intermediate representation is used to generate binary microcode; wherein the verification process includes at least one of semantic consistency verification, performance verification, and compatibility verification.

[0008] In one embodiment, the method of pre-training the reinforcement learning policy model includes: Load pre-built reinforcement learning policy models and basic microcode datasets; For each microcode sample in the aforementioned basic microcode dataset, the reinforcement learning policy model outputs an action probability distribution based on the current state; The action probability distribution is applied to the microcode intermediate representation corresponding to the current microcode sample to obtain the optimized microcode intermediate representation; the microcode execution simulator is invoked to simulate the optimized microcode intermediate representation to obtain new states and performance data; The incentive value is calculated based on the performance data and the preset reward function, and the strategy evaluation value is determined based on the incentive value. When the cumulative number of samples reaches a preset threshold, a policy update target is constructed based on the policy evaluation value, and the model parameters of the reinforcement learning policy model are updated. Reset the samples and iterate the training until the preset conditions are met, and output the trained reinforcement learning policy model.

[0009] In one embodiment, the method of pre-training the reinforcement learning policy model further includes: The training metrics during the iterative training process are analyzed in real time, and the training metrics include at least one of the following: average reward, execution cycle reduction rate, policy update magnitude, and value network loss. Based on the training metrics, the training process is monitored to detect any abnormal states, including training oscillations and gradient vanishing.

[0010] In one embodiment, the method for determining whether the preset conditions are met includes: If the average reward increase for consecutive preset rounds is less than a preset threshold and the fluctuation does not exceed a preset range, and / or the execution cycle of the optimized microcode is reduced by a preset proportion relative to the native version, then the preset conditions are considered to be met.

[0011] In one embodiment, the method of pre-building the reinforcement learning policy model includes: Construct the state space, action space, reward function, and execution environment; A policy network is constructed; the policy network includes three fully connected hidden layers and an output layer connected in sequence. Each fully connected hidden layer adopts a non-linear activation function, and the output layer adopts a normalization function to output the action probability distribution.

[0012] In one embodiment, the method of pre-building the reinforcement learning policy model further includes: Construct a value network; the value network has the same hidden layer structure as the policy network; the output layer of the value network is used for linear regression to output state values; The method of pre-training the reinforcement learning policy model also includes: The value network is used to output the state value corresponding to the new state, and the strategy evaluation value is determined based on the incentive value and the state value.

[0013] Secondly, this application proposes a microcode generation system based on reinforcement learning, the system comprising: The acquisition module is used to acquire the intermediate microcode representation of the target instruction to be implemented. The microcode generation module is used to extract the state feature vector of the current microcode intermediate representation using a pre-trained reinforcement learning strategy model, and output the action probability distribution of each action based on the state feature vector; wherein, each action corresponds to a preset transformation operation performed on the microcode intermediate representation; a target action is selected based on the action probability distribution and applied to the current microcode intermediate representation to iteratively update the next generation of the microcode intermediate representation until a preset iteration condition is met to obtain the target microcode intermediate representation; and binary microcode is generated based on the target microcode intermediate representation.

[0014] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method steps in the first aspect.

[0015] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the method steps of the first aspect.

[0016] The aforementioned microcode generation method and system based on reinforcement learning have at least the following advantages: This application employs a pre-trained reinforcement learning strategy model to extract the state feature vector of the microcode intermediate representation and output the action probability distribution. Based on this action probability distribution, a target action is selected and applied to the current microcode intermediate representation for iterative updates. During the iterative update process, the model comprehensively weighs the subsequent benefits in conjunction with the current state, enabling it to gradually approach a better scheduling sequence even when micro-operation dependencies and hardware resource competition coexist. Simultaneously, by iteratively updating and selecting better scheduling change operations, unnecessary pauses and resource conflicts can be reduced, improving the utilization efficiency of critical execution resources. Attached Figure Description

[0017] Figure 1This is a diagram illustrating the application environment of a microcode generation method based on reinforcement learning in one embodiment. Figure 2 This is a flowchart illustrating a microcode generation method based on reinforcement learning in one embodiment; Figure 3 This is a block diagram of a microcode generation system based on reinforcement learning in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0019] Some exemplary embodiments of this application have been described for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.

[0020] The microcode generation method based on reinforcement learning provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0021] The terminal 102 can send the microcode intermediate representation corresponding to the target instruction to be implemented to the server 104, so that the server 104 can process the microcode intermediate representation. For example, the server 104 uses a pre-trained reinforcement learning policy model to extract the state feature vector of the current microcode intermediate representation and outputs the action probability distribution of each action based on the state feature vector. Each action corresponds to a preset transformation operation executed on the microcode intermediate representation. Based on the action probability distribution, the target action is selected and applied to the current microcode intermediate representation to iteratively update and obtain the next generation of microcode intermediate representation until the preset iteration condition is met to obtain the target microcode intermediate representation. Binary microcode is generated based on the target microcode intermediate representation. The server 104 then feeds back the generated binary microcode to the terminal 102.

[0022] The aforementioned microcode generation method based on reinforcement learning employs a pre-trained reinforcement learning policy model to extract the state feature vector of the intermediate microcode representation and output an action probability distribution. Based on this action probability distribution, a target action is selected and applied to the current intermediate microcode representation for iterative updates. During the iterative update process, the model comprehensively weighs subsequent benefits in conjunction with the current state, enabling it to gradually approach a better scheduling sequence even when micro-operation dependencies and hardware resource competition coexist. Furthermore, by iteratively updating and selecting better scheduling change operations, unnecessary pauses and resource conflicts can be reduced, improving the utilization efficiency of critical execution resources.

[0023] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0024] In one exemplary embodiment, this application provides a microcode generation method based on reinforcement learning, which will be applied to the following examples. Figure 1 We will use server 104 as an example to illustrate this.

[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating a microcode generation method based on reinforcement learning in this embodiment, specifically including the following steps: Step 202: Obtain the intermediate microcode representation corresponding to the target instruction to be implemented.

[0026] Specifically, target instructions refer to instruction objects that are to be generated or optimized into microcode, and can be any instruction in the instruction set of the target processor architecture. For example, target instructions can be arithmetic instructions, logical instructions, memory access instructions, and branch jump instructions. Optionally, target instructions are output by a compiler backend, assembler, or instruction decoding module.

[0027] The microcode intermediate representation is used to characterize the execution control sequence of a target instruction on a microarchitecture, and is a structured representation prior to binary microcode. The microcode intermediate representation typically includes a set of micro-operations, dependency descriptions, scheduling and timing information, and resource binding information. The set of micro-operations consists of one or more micro-operations obtained by decomposing the target instruction; each micro-operation corresponds to a basic execution behavior, such as register read, ALU operation, address generation, memory access request, and result write-back. Dependency descriptions describe data dependencies, control dependencies, or resource dependencies between micro-operations, and can be represented by dependency edges, dependency depth, or ready conditions. Scheduling and timing information describes the issue order, concurrent combination relationships, and possible no-operation insertion positions of micro-operations in each execution cycle slot. Resource binding information describes the mapping relationship between micro-operations and hardware execution resources, such as ALU type selection, execution port selection, register port and bus channel selection, and memory access queues.

[0028] Step 204: Using a pre-trained reinforcement learning strategy model, extract the state feature vector of the current microcode intermediate representation, and output the action probability distribution of each action based on the state feature vector; wherein, each action corresponds to a preset transformation operation executed in the microcode intermediate representation.

[0029] Specifically, the reinforcement learning policy model used in this application is constructed based on the Proximal Policy Optimization (PPO) reinforcement learning algorithm. PPO belongs to the policy gradient class of reinforcement learning algorithms, which directly outputs the probability distribution of micro-operations by optimizing the policy network. By introducing a pruning objective function, the step size of each policy update is limited to avoid training collapse. Its objective function is as follows:

[0030] Where: r(θ) is the probability ratio of the new and old strategies; A is the advantage function, used to evaluate the value of actions; ε is the pruning coefficient.

[0031] Furthermore, this application extracts multi-dimensional state feature vectors from the current microcode intermediate representation based on a pre-constructed state space (S). For example, the specific mapping content of the state space includes hardware resource status, micro-operation dependencies, and current timing progress, specifically corresponding to a 56-dimensional feature vector, including register occupancy, arithmetic logic unit (ALU) load, micro-operation dependency depth, remaining execution cycles, etc.

[0032] The multidimensional state feature vectors are concatenated in a preset order and then normalized or truncated to ensure numerical scale stability, ultimately resulting in a fixed-length state feature vector.

[0033] Furthermore, the action space (A) is predefined as a set containing multiple types of discrete actions. The pre-trained reinforcement learning policy model outputs the action probability distribution of each action type in the action space based on the current state feature vector. For example, the specific mapping content of the action space includes micro-operation scheduling actions and resource reallocation actions, specifically corresponding to a discrete action set including 8 types of actions, such as single micro-operations, double micro-operations, and resource preemption adjustment. Each type of discrete action corresponds to a preset transformation operation. For example, a single micro-operation corresponds to a single-issue scheduling transformation operator, that is, selecting a micro-operation from the micro-operation set and placing it in the target periodic slot, while updating the dependency state and resource occupancy information.

[0034] Furthermore, this application also pre-constructs the reward function (R) and the execution environment (E).

[0035] The specific mapping content of the reward function includes execution efficiency, resource utilization, and timing compliance. The corresponding practical definition expression is: R = α×(1 / execution cycle) + β×resource utilization - γ×timing violation penalty; where α=0.6, β=0.3, γ=0.1 in this embodiment, and can be fine-tuned as needed in actual applications.

[0036] Furthermore, the specific mapping content of the execution environment includes the microcode execution simulator and performance counters, specifically corresponding to the microcode simulator and the microcode intermediate representation (IR) environment.

[0037] Step 206: Select the target action based on the action probability distribution and apply it to the current microcode intermediate representation to iteratively update and obtain the next generation of microcode intermediate representation until the preset iteration conditions are met to obtain the target microcode intermediate representation.

[0038] Specifically, in each iteration, a pre-trained reinforcement learning policy model is used to output the action probability distribution for the current microcode intermediate representation, and then the target action for the current iteration is determined based on this probability distribution. For example, during the training phase, a probability sampling method can be used to select the target action to maintain the exploration capability; during the deployment phase, a method can be used to select the action with the highest probability or to select the best action from the set of high-probability actions to improve the stability of the generated results.

[0039] The determined target action is mapped to a predefined microcode intermediate representation transformation operation, and the corresponding structural adjustment is performed on the current microcode intermediate representation to obtain the microcode intermediate representation for the next iteration. Based on the microcode intermediate representation for the next iteration, the state feature vector is re-extracted, and the action decision and transformation are entered into the next round, thus forming an iterative optimization process.

[0040] For example, the preset iteration conditions include at least one of the following: whether all micro-operations have been scheduled, whether preset timing constraints have been met, whether resource constraints have been met, whether the preset maximum number of iterations has been reached, and performance improvement threshold.

[0041] Step 208: Generate binary microcode based on the intermediate representation of the target microcode.

[0042] Specifically, after obtaining the intermediate representation of the target microcode through iterative optimization, the intermediate representation is converted into binary microcode that can be loaded and executed by the target processor. The conversion is based on the target processor's microinstruction format specification and uses preset encoding rules to perform field mapping, bit-width concatenation, and integrity verification on the intermediate microcode representation.

[0043] The aforementioned microcode generation method based on reinforcement learning employs a pre-trained reinforcement learning policy model to extract the state feature vector of the intermediate microcode representation and output an action probability distribution. Based on this action probability distribution, a target action is selected and applied to the current intermediate microcode representation for iterative updates. During the iterative update process, the model comprehensively weighs subsequent benefits in conjunction with the current state, enabling it to gradually approach a better scheduling sequence even when micro-operation dependencies and hardware resource competition coexist. Furthermore, by iteratively updating and selecting better scheduling change operations, unnecessary pauses and resource conflicts can be reduced, improving the utilization efficiency of critical execution resources.

[0044] Optionally, the above-mentioned reinforcement learning-based microcode generation method further includes: The intermediate representation of the target microcode is verified, and the verified intermediate representation of the target microcode is used to generate binary microcode; wherein the verification process includes at least one of semantic consistency verification, performance verification and compatibility verification.

[0045] Specifically, after obtaining the intermediate representation of the target microcode, further verification processing is performed on it to improve the engineering reliability and deployment security of the generated results.

[0046] For example, this application converts the intermediate representation of the target microcode into a verification version that can be executed in a simulator, and runs the optimized microcode sequence corresponding to the target instruction in a microcode execution simulator such as the QEMU simulator, while simultaneously invoking a baseline microcode sequence to execute under the same input data and the same initial state. The execution results of the two are compared to confirm semantic consistency.

[0047] Furthermore, after the semantic consistency verification is passed, the optimized microcode sequence corresponding to the intermediate representation of the target microcode is executed on the target hardware. Data such as execution cycle, resource utilization, and power consumption are collected and compared with the corresponding data of the benchmark microcode sequence to confirm whether the performance improvement meets the target.

[0048] Furthermore, after semantic consistency verification and / or performance verification are passed, the optimized microcode corresponding to the target microcode intermediate representation is integrated into the firmware packaging process, and compatibility testing is performed on the target hardware platform to confirm whether the above compatibility triggers preset faults or alarm conditions, so as to confirm whether the compatibility meets the standards.

[0049] Optionally, the above-mentioned reinforcement learning-based microcode generation method further includes: The generated binary microcode is packaged according to the microcode update specification of the target hardware to generate a firmware file that can be recognized and loaded by the target hardware. The firmware file can be in a preset format corresponding to the target hardware, such as the .cap format for Intel platforms and the .bin format for AMD platforms.

[0050] Furthermore, when packaging the aforementioned firmware file, a version identifier is written to the firmware file and verification information is generated for integrity verification before and after loading.

[0051] Furthermore, the firmware file is written to the microcode update channel of the target hardware via the BIOS / UEFI update interface, and a reboot is performed after the update is complete. After rebooting, a performance monitoring tool is invoked to confirm the microcode loading status, verifying that the binary microcode in the firmware file has been loaded and is effective. The performance monitoring tool can be Intel VTune or a similar performance analysis tool.

[0052] Optionally, methods for pre-training reinforcement learning policy models include: Load pre-built reinforcement learning policy models and basic microcode datasets; For each microcode sample in the basic microcode dataset, a reinforcement learning policy model is used to output the action probability distribution based on the current state; The action probability distribution is applied to the intermediate microcode representation corresponding to the current microcode sample to obtain the optimized intermediate microcode representation; the microcode execution simulator is called to simulate the optimized intermediate microcode representation to obtain the new state and performance data; The incentive value is calculated based on performance data and a preset reward function, and the strategy evaluation value is determined based on the incentive value. When the cumulative number of samples reaches a preset threshold, a policy update target is constructed based on the policy evaluation value, and the model parameters of the reinforcement learning policy model are updated. Reset the samples and iterate the training until the preset conditions are met, and output the trained reinforcement learning policy model.

[0053] Specifically, during the initialization phase, the initial reinforcement learning policy model and basic microcode dataset are loaded, and the microcode execution simulator is set to training mode. The basic microcode dataset refers to a sample set of native microcode sequences and their corresponding micro-operations and scheduling information, collected and organized for the target processor architecture and covering typical instruction types. This dataset serves as the initial sample source for training and evaluating the reinforcement learning model. For example, native microcode sequences for the target architecture (such as x86) are collected, covering micro-operation sequences corresponding to core instructions such as addition, multiplication, and memory read / write, as initial training samples.

[0054] During the interactive sampling phase, a reinforcement learning strategy model is used to extract features from each microcode sample and output the action probability distribution based on the current state. Then, the current microcode sample is adjusted according to the action probability distribution to obtain the optimized intermediate representation of the microcode. Finally, a microcode execution simulator is used to simulate it and return the new state and performance data.

[0055] During the reward calculation phase, the incentive value for each step is calculated based on performance data and a preset reward function, thereby obtaining the policy evaluation value used for policy updates. The policy evaluation value characterizes the superiority or inferiority of the selected action relative to the baseline policy and serves as a weighting factor for policy updates to construct the policy update objective.

[0056] During the policy update phase, after accumulating a preset threshold number of samples, a policy update objective is constructed based on the policy evaluation value, and the model parameters of the reinforcement learning policy model are updated using the objective function of PPO. The current sample cache is then cleared before proceeding to the next iteration.

[0057] Optionally, the methods for determining whether the preset conditions are met include: If the average reward increase over consecutive preset rounds is less than a preset threshold and the fluctuation does not exceed a preset range, and / or the execution cycle of the optimized microcode is reduced by a preset percentage relative to the original version, then the preset conditions are considered met. For example, in this embodiment, training is stopped when the average reward no longer increases after 100 consecutive iterations (fluctuation ≤ 1%), or the execution cycle of the optimized microcode is reduced by ≥ 20% compared to the original version.

[0058] Optionally, before pre-training the reinforcement learning policy model, the following steps are also included: We construct a basic microcode dataset as initial training samples and build a reinforcement learning policy model.

[0059] Obtain hardware performance benchmark data. Execute native microcode using a microcode execution simulator, and record the execution cycle, resource utilization, and timing parameters of each sequence as benchmark values ​​for reward function calculation.

[0060] Establish validity verification rules. By importing the semantic specifications of the target ISA, a microcode semantic verification library is built to avoid introducing logical errors after optimization.

[0061] Furthermore, this application also defines a software toolchain. For example, Python (3.8–3.10) is selected as the runtime environment for training scripts and policy logic; PyTorch (1.12 and above) is selected to build and train the policy network and value network of the PPO; LLVM (14.0 and above) is selected for the generation, parsing, and transformation of the microcode intermediate representation (IR), and to interface with the compiler flow; QEMU (7.0 and above) is selected as a microcode execution simulator to collect execution cycle, resource utilization, and / or timing-related performance data; and Microcode Builder (Intel official version) is selected to encode the target microcode intermediate representation into binary microcode and further complete firmware encapsulation.

[0062] By adopting the above scheme, this application has engineered the PPO reinforcement learning process with microcode generation and optimization. Specifically, it provides reproducible initial samples and quantitative evaluation baselines for training through basic microcode datasets and hardware performance benchmark data, directly aligning reward signals with microarchitectural goals such as execution cycle, resource utilization, and timing metrics. By importing the ISA semantic specification to construct a legality verification rule base, semantic consistency and timing compliance are used as hard constraints for training and generation, preventing the policy from learning undeployable illegal transformations. Furthermore, by leveraging LLVM's IR processing capabilities, QEMU's execution simulation and performance acquisition capabilities, and Microcode Builder's binary encoding encapsulation capabilities, a closed-loop pipeline is formed sequentially, consisting of IR, state features, PPO decision-making, IR transformation, simulation feedback, verification, and firmware output. This allows PPO to not only remain at the algorithm level but also directly apply to actual engineering objects of micro-operation scheduling and resource allocation, achieving a balance between performance improvement and engineering reliability while increasing deployment efficiency.

[0063] Optionally, methods for pre-building reinforcement learning policy models include: Construct the state space, action space, reward function, and execution environment.

[0064] Construct a policy network; the policy network consists of three fully connected hidden layers and an output layer connected in sequence. Each fully connected hidden layer uses a non-linear activation function, and the output layer uses a normalization function to output the probability distribution of actions.

[0065] Specifically, the policy network (Actor) takes a 56-dimensional state feature vector as input and outputs an action probability distribution for 8 action classes. The dimensions of the three fully connected hidden layers are 256, 128, and 64, respectively; the ReLU function is used as the non-linear activation function; and the output layer uses the Softmax normalization function.

[0066] Optionally, methods for pre-building reinforcement learning policy models also include: Construct a value network; the value network and the policy network have the same hidden layer structure; the output layer of the value network is used for linear regression to output state values; Methods for pre-training reinforcement learning policy models also include: The value network is used to output the state value corresponding to the new state, and the strategy evaluation value is determined based on the incentive value and the state value.

[0067] Specifically, the input to the value network (Critic) is the same as that of the policy network, both being 56-dimensional state feature vectors. The output of the value network is the state value, and its output layer has no activation function.

[0068] By using a value network to output the state value of a new state and combining it with the incentive value to determine the policy evaluation value, the immediate incentive can be debiased and smoothed using the state value as a baseline. This allows the policy evaluation to more accurately reflect the gain or loss of the selected action relative to the average level, thereby reducing the variance of policy updates and improving the stability of PPO policy updates.

[0069] Furthermore, pre-constructing reinforcement learning policy models also includes configuring training hyperparameters. Training hyperparameters include: learning rate, batch size, number of iterations, number of update steps per round, pruning factor, discount factor, and value loss weights.

[0070] For example, the training hyperparameters in this application are configured as follows: learning rate = 3e-4, batch size = 64, number of iterations = 1000, number of update steps per round = 2048, pruning factor ε = 0.2, discount factor γ = 0.99, and value network loss weight = 0.5.

[0071] By employing the above scheme, the multi-layer fully connected structure of the policy network performs progressive nonlinear abstraction of state features and outputs action probability distributions through normalization. This more accurately characterizes the complex coupling relationship between micro-operation dependencies and resource consumption, thereby improving the expressive and generalization capabilities of scheduling decisions. Furthermore, the value network adopts a feature extraction structure that matches the policy network and linearly outputs state values, providing a stable evaluation baseline for policy updates, reducing gradient variance, and suppressing training oscillations. Simultaneously, through the coordinated configuration of multiple hyperparameters, sample utilization efficiency and convergence speed can be improved while ensuring controlled update step size, making the training process more stable and the generated results more reliable.

[0072] Furthermore, to achieve a data loop between the reinforcement learning strategy model, the microcode execution environment, and the compiler intermediate representation, this application also adapts the environment and interface.

[0073] First, the microcode execution simulator is adapted to output runtime information consistent with the reinforcement learning state space during training mode. For example, a performance counting output interface is added to the microcode execution simulator, enabling it to output state data such as execution cycle, register usage metrics, and arithmetic logic unit load in real time when executing optimized microcode sequences. This state data is then output to the training program in a structured data format, allowing the training program to obtain new state and performance data after each round of interaction for subsequent reward calculation and sample construction.

[0074] Secondly, the reinforcement learning policy model is integrated with the microcode intermediate representation processing flow to achieve automated microcode (IR), decision-making, and IR update workflows. For example, the IR is integrated into the policy decision-making process through the compiler's intermediate representation processing mechanism. This allows the current IR to be input into the state feature extractor to generate a state feature vector during training or deployment. The policy network then outputs scheduling actions and triggers corresponding IR transformation operations, thereby obtaining the optimized IR and proceeding to the next iteration or output stage.

[0075] Furthermore, to ensure the semantic and temporal usability of the microcode generated or optimized during training, this embodiment provides a legality verification interface. For example, the semantic verification rules, temporal and resource constraint verification rules of the target instruction set are encapsulated into a callable interface. This allows for the initial verification of each optimized microcode sequence to confirm the absence of semantic errors and non-compliance with temporal constraints. Only when the verification passes is the optimization result allowed to enter the reward calculation and sample accumulation process, thereby improving the quality of training samples and preventing the policy from being misled by illegal sequences.

[0076] By adopting the above scheme, the state feature extraction, PPO strategy decision-making, microcode execution simulation, performance data collection and legality verification are formed into a closed-loop automated process. This enables repeatable training and rapid iterative optimization of microcode while ensuring semantic and temporal compliance, thereby improving optimization efficiency and engineering implementation reliability.

[0077] Alternatively, methods for pre-training reinforcement learning policy models also include: Real-time analysis of training metrics during iterative training, including at least one of average reward, execution cycle reduction rate, policy update magnitude, and value network loss; Based on training metrics, the system detects abnormal states during the training process to monitor the training process. Abnormal states include training oscillations and gradient vanishing.

[0078] Specifically, during the iterative training of the pre-trained reinforcement learning policy model, the training program writes training metrics to a log file after each round of sampling or parameter update. Then, a monitoring tool is launched to read the log file and visualize the changing trends of these training metrics in real time. The monitoring tool used is TensorBoard, which is commonly used to display metrics generated during training in real time as curves, histograms, etc., facilitating the observation of any training anomalies.

[0079] By adopting the above scheme, anomalies such as training oscillations and gradient vanishing can be detected in a timely manner and adjustments can be triggered, thereby improving the stability and convergence efficiency of reinforcement learning policy model training and reducing the cost of ineffective training.

[0080] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0081] Based on the same inventive concept, this application also provides a microcode generation system based on reinforcement learning. This system is applicable to the above-described microcode generation method based on reinforcement learning. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more system embodiments provided below can be found in the limitations of the method above, and will not be repeated here.

[0082] Please see Figure 3 In one embodiment, the reinforcement learning-based microcode generation system includes an acquisition module and a microcode generation module.

[0083] The acquisition module is used to acquire the intermediate microcode representation of the target instruction to be implemented. The microcode generation module uses a pre-trained reinforcement learning strategy model to extract the state feature vector of the current microcode intermediate representation and output the action probability distribution of each action based on the state feature vector. Each action corresponds to a preset transformation operation performed on the microcode intermediate representation. The module selects a target action based on the action probability distribution and applies it to the current microcode intermediate representation to iteratively update the next generation of microcode intermediate representations until a preset iteration condition is met to obtain the target microcode intermediate representation. Finally, binary microcode is generated based on the target microcode intermediate representation.

[0084] Optionally, the above-mentioned reinforcement learning-based microcode generation system also includes a verification module.

[0085] The verification module is used to verify the intermediate representation of the target microcode and generate binary microcode from the verified intermediate representation of the target microcode; wherein the verification process includes at least one of semantic consistency verification, performance verification and compatibility verification.

[0086] Optionally, the above-mentioned reinforcement learning-based microcode generation system also includes an encapsulation module.

[0087] The encapsulation module encapsulates the generated binary microcode according to the target hardware's microcode update specifications, generating a firmware file that can be recognized and loaded by the target hardware. During the encapsulation process, it writes a version identifier and generates verification information for the firmware file. Then, it writes the firmware file to the target hardware's microcode update channel via the BIOS / UEFI update interface and performs a reboot operation after the update is complete. After rebooting, a performance monitoring tool is invoked to confirm the microcode loading status, verifying that the binary microcode in the firmware file has been successfully loaded and is effective. This performance monitoring tool can be Intel VTune or a similar performance analysis tool.

[0088] Optionally, the aforementioned microcode generation system based on reinforcement learning also includes a training module.

[0089] The training module loads a pre-built reinforcement learning policy model and a basic microcode dataset. For each microcode sample in the basic microcode dataset, the reinforcement learning policy model outputs an action probability distribution based on the current state. This action probability distribution is applied to the intermediate microcode representation corresponding to the current microcode sample to obtain an optimized intermediate microcode representation. A microcode execution simulator is invoked to simulate the optimized intermediate microcode representation to obtain new states and performance data. An incentive value is calculated based on the performance data and a preset reward function, and a policy evaluation value is determined based on the incentive value. When the accumulated number of samples reaches a preset threshold, a policy update objective is constructed based on the policy evaluation value, and the model parameters of the reinforcement learning policy model are updated. Samples are reset and iterative training is performed until preset conditions are met, and the trained reinforcement learning policy model is output. The criteria for meeting the preset conditions include: if the average reward increase over consecutive preset rounds is less than a preset threshold and the fluctuation does not exceed a preset range, and / or the execution cycle of the optimized microcode is reduced by a preset proportion relative to the original version, then the preset conditions are considered met.

[0090] Optionally, the above-mentioned reinforcement learning-based microcode generation system also includes a preprocessing module.

[0091] The preprocessing module is used to build a basic microcode dataset as initial training samples and to build a reinforcement learning policy model; to obtain hardware performance benchmark data; to build legality verification rules; and to determine the software toolchain.

[0092] Optionally, the preprocessing module pre-constructs the reinforcement learning policy model in the following ways: constructing the state space, action space, reward function, and execution environment; constructing the policy network; the policy network includes three fully connected hidden layers and an output layer connected in sequence, each fully connected hidden layer using a non-linear activation function, and the output layer using a normalization function to output the action probability distribution.

[0093] Optionally, the preprocessing module can pre-build a reinforcement learning policy model by: constructing a value network; the value network having the same hidden layer structure as the policy network; and the output layer of the value network being used for linear regression to output state values. In this case, the training module is also used to use the value network to output the state value corresponding to the new state, and to determine the policy evaluation value based on the incentive value and the state value.

[0094] Optionally, the preprocessing module is also used to adapt the environment and interface.

[0095] Optionally, the method of pre-training the reinforcement learning policy model in the training module further includes: real-time parsing of training metrics during the iterative training process, including at least one of average reward, execution cycle reduction rate, policy update magnitude, and value network loss; and based on the training metrics, detecting whether abnormal states occur during the training process to monitor the training process, including training oscillations and gradient vanishing.

[0096] The aforementioned microcode generation system based on reinforcement learning employs a pre-trained reinforcement learning policy model to extract state feature vectors from the intermediate representations of the microcode and output action probability distributions. Based on these probability distributions, a target action is selected and applied to the current intermediate microcode representation for iterative updates. During these iterations, the model comprehensively weighs subsequent gains against the current state, enabling it to gradually approach a better scheduling sequence even amidst micro-operation dependencies and hardware resource competition. Furthermore, by iteratively updating and selecting superior scheduling change operations, unnecessary pauses and resource conflicts are reduced, improving the utilization efficiency of critical execution resources.

[0097] The modules in the reinforcement learning-based microcode generation system described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each module.

[0098] In one feasible embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the aforementioned microcode generation method based on reinforcement learning. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0099] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] In one feasible embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps in the reinforcement learning-based microcode generation method described above.

[0101] In one feasible embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method steps in the reinforcement learning-based microcode generation method described above.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A microcode generation method based on reinforcement learning, characterized in that, The method includes: Obtain the intermediate microcode representation of the target instruction to be implemented; A pre-trained reinforcement learning strategy model is used to extract the state feature vector of the current microcode intermediate representation, and the action probability distribution of each action is output based on the state feature vector; wherein, each type of action corresponds to a preset transformation operation performed on the microcode intermediate representation; The target action is selected based on the action probability distribution and applied to the current microcode intermediate representation to iteratively update the next generation of the microcode intermediate representation until the preset iteration condition is met to obtain the target microcode intermediate representation. Binary microcode is generated based on the intermediate representation of the target microcode.

2. The method according to claim 1, characterized in that, The method further includes: The target microcode intermediate representation is verified, and the verified target microcode intermediate representation is used to generate binary microcode; wherein the verification process includes at least one of semantic consistency verification, performance verification, and compatibility verification.

3. The method according to claim 1, characterized in that, Methods for pre-training the reinforcement learning policy model include: Load pre-built reinforcement learning policy models and basic microcode datasets; For each microcode sample in the basic microcode dataset, the reinforcement learning policy model outputs an action probability distribution based on the current state; The action probability distribution is applied to the microcode intermediate representation corresponding to the current microcode sample to obtain the optimized microcode intermediate representation; The microcode execution simulator is invoked to simulate the optimized intermediate representation of the microcode in order to obtain new states and performance data. The incentive value is calculated based on the performance data and the preset reward function, and the strategy evaluation value is determined based on the incentive value. When the cumulative number of samples reaches a preset threshold, a policy update target is constructed based on the policy evaluation value, and the model parameters of the reinforcement learning policy model are updated. Reset the samples and iterate the training until the preset conditions are met, and output the trained reinforcement learning policy model.

4. The method according to claim 3, characterized in that, The method of pre-training the reinforcement learning policy model also includes: The training metrics during the iterative training process are analyzed in real time, and the training metrics include at least one of the following: average reward, execution cycle reduction rate, policy update magnitude, and value network loss. Based on the training metrics, the training process is monitored to detect any abnormal states, including training oscillations and gradient vanishing.

5. The method according to claim 3, characterized in that, The methods for determining whether the preset conditions are met include: If the average reward increase for consecutive preset rounds is less than a preset threshold and the fluctuation does not exceed a preset range, and / or the execution cycle of the optimized microcode is reduced by a preset proportion relative to the native version, then the preset conditions are considered to be met.

6. The method according to claim 3, characterized in that, Methods for pre-building reinforcement learning policy models include: Construct the state space, action space, reward function, and execution environment; A policy network is constructed; the policy network includes three fully connected hidden layers and an output layer connected in sequence. Each fully connected hidden layer adopts a non-linear activation function, and the output layer adopts a normalization function to output the action probability distribution.

7. The method according to claim 6, characterized in that, Other methods for pre-building reinforcement learning policy models include: Construct a value network; the value network has the same hidden layer structure as the policy network; the output layer of the value network is used for linear regression to output state values; The method of pre-training the reinforcement learning policy model also includes: The value network is used to output the state value corresponding to the new state, and the strategy evaluation value is determined based on the incentive value and the state value.

8. A microcode generation system based on reinforcement learning, characterized in that, The system includes: The acquisition module is used to acquire the intermediate microcode representation of the target instruction to be implemented. The microcode generation module is used to extract the state feature vector of the current microcode intermediate representation using a pre-trained reinforcement learning strategy model, and output the action probability distribution of each action based on the state feature vector; wherein, each action corresponds to a preset transformation operation performed on the microcode intermediate representation; a target action is selected based on the action probability distribution and applied to the current microcode intermediate representation to iteratively update the next generation of the microcode intermediate representation until a preset iteration condition is met to obtain the target microcode intermediate representation; and binary microcode is generated based on the target microcode intermediate representation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images