RTL code generation reinforcement learning method based on multilevel hierarchical reward model driving

Abstract

The invention discloses an RTL code generation reinforcement learning method based on multi-level hierarchical reward model driving, and relates to the technical field of reinforcement learning, and the method comprises the steps: constructing a preference data set; performing comparative learning on the large language model by using the preference data set to obtain a reward model RM-1; performing reinforcement learning on the large language model by using the reward model RM-1 to generate an intermediate strategy model; constructing a preference data set and carrying out comparative learning on the reward model RM-1 to obtain a reward model RM-2, and then carrying out reinforcement learning on the intermediate strategy model to generate a final strategy model; according to the method, the reward model is used for replacing a fixed rule reward function, the reward tree is introduced to relieve reward sparseness, the comprehensive feasibility and the PPA index are introduced into a unified reward signal, and multi-dimensional optimization and flexible expansion can be achieved while it is guaranteed that the RTL function is correct.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in software code generation, their potential in hardware design has also attracted increasing attention. Current research has shown that LLMs can directly generate RTL code from natural language descriptions, thereby improving the automation level of chip design and reducing labor costs. However, traditional LLMs still have the following problems in the RTL code generation process: 1) Most methods can only guarantee syntactic correctness and functional correctness, and it is difficult to take into account key hardware design indicators such as synthesis and PPA (power consumption, performance, and area). 2) High-quality open-source datasets for RTL are currently scarce. Existing methods such as supervised fine-tuning (SFT) usually improve performance by training on domain datasets to make the model mimic the target behavior. However, SFT is too dependent on the dataset and often requires more than 10,000 samples to make a significant change in the model behavior. Using manual annotation to build RTL datasets is too costly. 3) Some studies (such as ChipSeek-R1) have proposed schemes based on reinforcement learning and reward functions, incorporating feedback from compilers, simulators, and EDA tools into the training process, achieving some success. However, their reward functions remain fixed and rule-based, lacking scalability and flexibility.

[0003] 4) The reward signal is sparse and noisy. In RTL code generation, the reward signal usually relies on compilation and simulation feedback. However, the initial model performance is limited, and the feedback results are represented as binary "pass / fail," lacking fine-grained reward signals, which leads to instability in the training process. At the same time, a large number of samples in the generated results contain grammatical or functional errors, producing noisy and distorted reward information, making it difficult for the model to extract effective learning signals from them, and easily causing overfitting or forgetting problems.

[0004] Therefore, there is an urgent need for a new training method that can ensure the correctness of RTL functions while also achieving multidimensional optimization and flexible expansion. Summary of the Invention

[0005] The purpose of this invention is to provide a reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model. This method addresses the problems of existing technologies that mostly use fixed-rule reward functions, resulting in sparse and noisy reward signals that make it difficult for models to extract effective learning signals; existing technologies mainly focus on semantic and functional correctness, lacking unified optimization for comprehensive feasibility and PPA; existing technologies heavily rely on large-scale SFT data, making it difficult to continuously benefit from toolchain feedback; and existing technologies lack scalability, with fixed rules difficult to migrate between different process libraries / project objectives.

[0006] To achieve the above objectives, this invention provides a reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model, comprising the following steps: Step 100, Construct the preference dataset ; Step 200: Select a large language model as the base model and use the preference dataset. Contrastive learning is performed, and the reward model RM-1 is obtained through contrastive learning based on preference pairs; Step 300: Use the reward model RM-1 and the reinforcement learning algorithm to perform reinforcement learning on the large language model to generate an intermediate policy model; Step 400, Construct the preference dataset ; Step 500: Select the reward model RM-1 as the base model and use the preference dataset. We conduct contrastive learning, and through contrastive learning based on preference pairs, we obtain the reward model RM-2. Step 600: Use the reward model RM-2 and the reinforcement learning algorithm to perform reinforcement learning on the intermediate policy model to generate the final policy model.

[0007] Furthermore, construct a preference dataset. The steps include: Step 101: Obtain the original dataset, which includes the functional description of the code. Step 102: Randomly sample a problem from the original dataset; Step 103, construct the reward tree RT1; Step 104, within the same question q, when , Included when the following conditions are met To form a dataset The formula for judgment is as follows: ; in, Let represent the true reward score for the j-th candidate answer text for question q. This is the hyperparameter for the difference interval. The lowest available threshold hyperparameter, For logical AND operator, This is the text of the i-th candidate answer to question q; This is the text of the j-th candidate answer to question q; Step 105: By repeating the above steps, construct the preference dataset. .

[0008] Furthermore, the method for constructing the reward tree RT1 includes: For each question q, output k output codes using the large language model. ; For each output generated Perform syntax checking and functional correctness checks to obtain each output. , ,Will( , , This serves as the first level of the reward tree; For the first layer The generated nodes are marked, and the process is traversed downwards. For each marked node, k child nodes are generated, and the true reward score of each child node is obtained. The average of the true reward scores of the child nodes is then used to replace the parent node's score. Each marked node can recursively go down at most two levels; in For hyperparameters, This represents the true reward score for the i-th candidate answer text for question q; During each round of downward traversal of the marked nodes, the large language model is used, combined with the information of each marked node. , A new reasoning process is performed to obtain k new outputs, and a new true reward score is obtained through functional correctness testing. Finally, the result is obtained by backtracking all parent nodes. Updated; in This represents the error messages generated by the i-th candidate code during syntax / compilation checks and the execution of the functional testbench.

[0009] Furthermore, the loss function of the reward model RM-1, based on contrastive learning of preference pairs, is calculated as follows: ; Let be the loss function used to update the reward model RM-1 during the t-th training iteration; t represents the trainable parameter vector of the reward model RM-1; t represents the training iteration round; m represents the total number of preference sample pairs participating in the calculation in the current training batch. For the input question text; For the reward model RM-1 pair problem With the i-th candidate answer The calculated reward score; For the reward model RM-1 pair problem With the j-th candidate answer The calculated reward score; σ is the Sigmoid function; The loss function of the reward model RM-2 based on contrastive learning of preference pairs is calculated using the same formula as the loss function of the reward model RM-1 based on contrastive learning of preference pairs.

[0010] Furthermore, preferred datasets Construction steps and preference dataset The difference in the construction steps is the preference dataset. 20% of the data comes from the preference dataset. The remaining 80% are randomly sampled; the rest are generated according to rules in the second layer. When constructing the reward tree, for each question q, the intermediate policy model outputs k output codes. The second layer generates functional preference pairs, comprehensive preference pairs, and PPA preference pairs: Define a set For functional correctness testing The candidate set, whose elements are in the form of ;gather This is the candidate set for synthesis failures, with elements in the form of... ;gather The candidate set that passes the comprehensive evaluation has elements in the form of... ; Indicates the minimum usable threshold for functional correctness; This represents the PPA score for the i-th candidate answer text for question q.

[0011] Furthermore, functional preference pairs: within the same question q, for candidates , When the following conditions are met, Incorporating preference datasets The formula for judgment is as follows: ; in This represents the text of the i-th candidate answer to question q; This represents the text of the j-th candidate answer to question q; This represents the pass rate of the i-th candidate answer text for question q; This represents the pass rate of the j-th candidate answer text for question q; The pass rate is calculated as the ratio of the number of successful test cases in the functional testbench to the total number of test cases, and is used to measure the functional correctness of the candidate code.

[0012] Furthermore, consider the overall preference pair: Will Add to preference dataset , and The calculation expression is as follows: ; ; in Indicates in set The selection of PPA scores The smallest candidate code index operation yields the corresponding candidate code. ; Indicates in set pass rate of the selected function The highest candidate code index operation is used to obtain the corresponding candidate code. ; Indicates in set In China, based on PPA scoring The worst sample of PPA selected; Indicates in set Among them, the sample with the highest functional pass rate.

[0013] Furthermore, PPA preference pair: Included in dataset Learn PPA ranking within the comprehensive sample. and The calculation expression is as follows: ; ; in Indicates in set The selection of PPA scores The largest candidate code index operation is used to obtain the corresponding candidate code. ; Indicates in set The selection of PPA scores The smallest candidate code index operation yields the corresponding candidate code. ; Indicates in set In China, based on PPA scoring The optimal sample of PPAs selected; Indicates in set In China, based on PPA scoring The worst sample of PPA was selected.

[0014] Furthermore, determine when Only then will Included in dataset , This represents the minimum interval threshold in the construction of the PPA preference pair.

[0015] Furthermore, the formula for calculating the PPA score is as follows: ; in , , These are the hyperparameters for power consumption, performance, and area, respectively. ; ; The weighting coefficients representing power consumption metrics; Indicates the weighting coefficient of the performance index; The weighting coefficient for area indicators; Indicates candidate code The normalized value of the power consumption index obtained after synthesis; Indicates candidate code Normalized values ​​of the performance metrics obtained after synthesis Indicates candidate code The normalized value of the area index obtained after synthesis.

[0016] Therefore, the reinforcement learning method for RTL code generation driven by the multi-level hierarchical reward model described above has the following beneficial effects: 1. A reward model is used to replace the fixed rule reward function, and a "reward tree" is introduced to alleviate the sparsity of rewards. The error feedback is used to trigger self-reflection and resampling to generate child nodes, and the old and new nodes are re-scored. This makes the original binary pass / fail feedback finely densified into a multi-level score distribution, significantly expanding the available preference pairs, and thus training a reward model that is only oriented towards grammatical and functional correctness. 2. A hierarchical reward mechanism is proposed. The first level uses compilation / syntax and functional correctness as qualification gates to train a reward model RM-1. Based on the reward model RM-1, the second level only performs synthesis and PPA evaluation on candidates with "compilation + functional pass". The reward model RM-2 is trained by constructing preference pairs through normalization and gating, thereby introducing synthesis feasibility and PPA objectives into a unified reward signal, realizing multi-objective optimization and gradual constraints and balance on syntax correctness, functional correctness, synthesizability and PPA optimization. 3. Adopt a reward model + reinforcement learning joint training paradigm: Use toolchains such as compiler / simulation / synthesis and PPA to automatically construct preference pairs, and use contrastive learning to train the reward model to replace manual preference scoring; in the policy optimization stage, use the differentiable unified reward generated by the reward model as the optimization target to achieve large-scale training and continuous self-improvement with low human intervention. 4. In the second level of the tiered reward system (after passing the "compilation + functionality" gating), a library-aware PPA scoring function is introduced to construct preference data pairs. When the process library or optimization objective changes, the reward model only needs to be retrained or incrementally fine-tuned based on the newly constructed preference pairs. Optimization can continue under the new reward model without changing the training framework, demonstrating strong scalability.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart of the reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model, as described in this invention. Detailed Implementation

[0019] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] It should be noted that: LLM: LLM is an abbreviation for "Large Language Model," a type of artificial intelligence model based on deep learning with massive parameters. Through learning and training on large-scale text data, it can understand and generate natural language text, and is often used for natural language processing tasks such as text generation, machine translation, and question answering.

[0021] RL: RL is an abbreviation for "Reinforcement Learning," a learning method in artificial intelligence. Its core idea is that an agent interacts with its environment and continuously adjusts its strategy based on rewards or penalties received, thereby maximizing long-term cumulative gains. RL is widely used in fields such as game competitions, autonomous driving, and robot control.

[0022] RTL: RTL stands for "Register Transfer Level," which is an abstraction level in digital circuit design. RTL primarily describes circuit behavior through data transfer and logical operations between registers. It is typically written in hardware description languages ​​such as Verilog or VHDL and is an important foundation for synthesis, simulation, and chip implementation.

[0023] EDA: EDA is an abbreviation for "Electronic Design Automation," which refers to the techniques and methods used to design, verify, and optimize integrated circuits and systems using computer software tools. EDA tools are widely used in circuit simulation, synthesis, placement and routing, and performance analysis, and are an indispensable supporting technology for modern chip design.

[0024] PPA: PPA is an abbreviation for "Power, Performance, Area," a core metric for measuring chip design quality. Power represents the circuit's power consumption, Performance represents the circuit's operating speed or timing performance, and Area represents the silicon area occupied by the chip. PPA is commonly used to evaluate the merits of design solutions and is an important target for integrated circuit design optimization.

[0025] RM stands for "Reward Model," an artificial intelligence model used to evaluate and score model outputs. By learning human preferences or automatically generated comparative data, RM can assign relatively superior scores to different candidate outputs, thus providing optimization signals to policy models during reinforcement learning or preference optimization processes.

[0026] SFT stands for "Supervised Fine-Tuning," which is a process of supervised training of a large language model using manually labeled or high-quality task data, building upon its pre-training. By pairing input with desired output samples, SFT can significantly improve the model's performance on specific tasks and is a commonly used method for downstream adaptation of large language models.

[0027] Please see Figure 1A reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model includes the following steps: S001: The original dataset, containing a functional description of the code; S002: Take a question q from the original dataset (the entire question bank can be traversed in parallel). It should be noted that problem q is a problem randomly sampled during the current step's loop, and a new problem is randomly sampled when the step loop restarts.

[0028] S003-S004: Construct the reward tree RT1; For each question q, k output codes are generated using a large language model (in this embodiment, a custom model Deepseekcoder-SFT, trained under supervised fine-tuning based on the publicly available DeepSeek-Coder base model). ; For each output generated in the first step Perform syntax checking and functional correctness checks to obtain each output. , ,Will As the first layer of the reward tree, but due to the low code generation capability of the base model, multiple outputs will be found. This can easily lead to reward sparsity, affecting the normal training of the model. To avoid this situation, a reward tree is constructed to solve the reward sparsity problem. The answers manually labeled as "better" in the preference pair This represents the true reward score for the i-th candidate answer text for question q, calculated based on the pass rate of the candidate code in the functional testbench; This represents the error messages generated by the i-th candidate code during syntax / compilation checks and the execution of the functional testbench.

[0029] For the first layer The generated nodes are marked, and the process continues downwards. For each marked node, k child nodes are generated, and the actual reward score of each child node is obtained. The average reward of the child nodes is then used to replace the parent node's reward. Each marked node can recursively go down at most two levels; During each round of downward traversal of the marked nodes, a large language model (GPT-4o is selected in this embodiment) is used, combined with the information of each marked node. , A new reasoning process is performed to obtain k new outputs, and a new true reward score is obtained through functional correctness testing. Finally, the result is obtained by backtracking all parent nodes. Updated.

[0030] S005-S006: Determine the pass rate of functions and construct preference data pairs; S005's functional pass rate assessment determines whether the functional pass rate of positive samples is significantly higher than that of negative samples and exceeds the minimum usable threshold; specifically: within the same question q, when the candidate The preference dataset is included when the following conditions are met. The formula for judgment is as follows: ; in: This is the hyperparameter for the difference interval. The lowest available threshold hyperparameter, For logical AND operator, For the j-th candidate answer text of question q, the answer that is manually labeled as "poor" in the preference pair. This represents the true reward score for the j-th candidate answer text for question q, calculated based on the pass rate of the candidate code in the functional testbench.

[0031] S007: By repeating the above steps, construct the preference dataset. ; S008-S009: Training the RM-1 reward model; A large language model (in this embodiment, a custom model Deepseekcoder-SFT, trained under supervised fine-tuning based on the publicly available DeepSeek-Coder base model) is selected as the base model, and a preference dataset is used. By conducting contrastive learning, a reward model RM-1 is obtained through contrastive learning based on preference pairs.

[0032] The loss function of the reward model RM-1, which is based on contrastive learning of preference pairs, is calculated using the following formula: ; in This represents the loss function used to update the reward model RM-1 at the t-th training iteration; This represents the trainable parameter vector of the reward model RM-1; t represents the training iteration round, used to identify which parameter update is being performed; m represents the total number of preference sample pairs participating in the calculation in the current training batch. This represents the input question text; This represents the trainable parameter vector of the reward model RM-1 (e.g., neural network weights and biases), which is minimized during training by a loss function. Update; This represents the RM-1 reward model for the problem, determined by parameter 𝜑. With the i-th candidate answer The calculated reward score; This indicates that the reward model RM-1 is used for the problem. With the j-th candidate answer The calculated reward score; σ represents the Sigmoid function.

[0033] S010-S012: Using the reward model RM-1, a reinforcement learning algorithm is used to perform reinforcement learning on the large language model (in this embodiment, a custom model Deepseekcoder-SFT is obtained by supervised fine-tuning training based on the publicly available DeepSeek-Coder base model) to generate an intermediate policy model (in this embodiment, DeepseekCoder-SFT-RL1). In this embodiment, the reinforcement learning algorithm used is the GRPO / Reinforce++ algorithm.

[0034] S013-S022: Preference Dataset Build; Preference dataset Construction and Preference Dataset The difference in the construction steps is the preference dataset. 20% of the data comes from the preference dataset. Random sampling ensures that the "functional ranking" soft constraint is retained in the second layer. The remaining 80% is generated according to rules in the second layer. The generation rules of the second layer include functional preference pairs, comprehensive preference pairs, and PPA preference pairs; In S014, when constructing the reward tree, for each question q, k output codes are generated using the intermediate policy model (DeepseekCoder-SFT-RL1). ; Define a set Indicates functional correctness test The candidate set, whose elements are in the form of ;gather This represents the candidate set of synthesis failures, with elements in the form of... ;gather This represents the candidate set that passed the synthesis, with elements in the form of... .

[0035] This represents the minimum acceptable functional correctness threshold, used to filter candidate code with insufficient functional correctness, and its value ranges from [0,1]. When the functional pass rate of a candidate code... At that time, the candidate was assigned to the set. These are considered low-quality samples with clearly insufficient functional correctness, and can be used to construct negative samples in functional preference pairs.

[0036] Function Preference: In S015, the function pass rate judgment specifically determines whether the function pass rate of positive samples is significantly higher than that of negative samples and exceeds the minimum usable threshold; maintaining the constraint of "better functions should be ranked higher" prevents prioritizing only PPA and biasing candidate functions within the same question q. , When the following conditions are met, Incorporating preference datasets The formula for judgment is as follows: ; in This represents the i-th candidate answer text for question q, which is manually labeled as "better" in the preference pair; This represents the j-th candidate answer text for question q, which is manually labeled as "poor" in the preference pair; This represents the pass rate of the i-th candidate answer text for question q; This represents the pass rate of the j-th candidate answer text for question q.

[0037] The pass rate is calculated as the ratio of the number of successful test cases in the functional testbench to the total number of test cases, and is used to measure the functional correctness of the candidate code.

[0038] Synthesis-Feasibility Preference: Add to preference dataset , and The calculation expression is as follows: ; ; in: Indicates in set The selection of PPA scores The smallest candidate code index operation yields the corresponding candidate code. ; Indicates in set pass rate of the selected function The highest candidate code index operation is used to obtain the corresponding candidate code. ; Indicates in set In China, based on PPA scoring The worst-performing sample in the PPA is selected as the "positive sample" in this preference pair; Indicates in set The sample with the highest functional pass rate is considered a "negative sample" in this preference pair, reflecting the feasibility priority constraint that "overall success should be better than overall failure".

[0039] PPA preference pairs: Learn PPA ranking within the comprehensive sample. and The calculation expression is as follows: ; ; in: Indicates in set The selection of PPA scores The largest candidate code index operation is used to obtain the corresponding candidate code. ; Indicates in set The selection of PPA scores The smallest candidate code index operation yields the corresponding candidate code. ; Indicates in set In China, based on PPA scoring The optimal sample of the selected PPA is taken as the "positive sample" in this preference pair; Indicates in set In China, based on PPA scoring The worst-performing sample in the PPA is selected as the "negative sample" in the preference pair; The PPA determination in S019 specifically involves determining whether the PPA score of the positive sample is significantly better than that of the negative sample and whether the difference exceeds the minimum interval threshold. The specific determination is as follows: Judgment when Only then will Included in dataset This is to filter out sample pairs with excessively small PPA differences and unclear preferences. denoted as the minimum interval threshold in the construction of PPA preference pairs, and is a non-negative real number.

[0040] The formula for calculating PPA scores is as follows: ; in , , These are the hyperparameters for power consumption, performance, and area, respectively. ; ; This represents the PPA score for the i-th candidate answer text for question q, used to rank candidate codes and construct preferences under multi-objective constraints (power consumption, performance, area); a higher score indicates better overall performance under the current weighted configuration. The weighting coefficient for power consumption is a non-negative hyperparameter; it is used to adjust the importance of power consumption in the overall score, and its value ranges from [0,1]. The weighting coefficient of the performance indicator is a non-negative hyperparameter; it is used to adjust the importance of performance (such as clock frequency, latency or throughput) in the overall score, and its value ranges from [0,1]. The weighting coefficient for the area index is a non-negative hyperparameter; it is used to adjust the importance of chip area in the overall score, and its value ranges from [0,1]. Indicates candidate code The normalized value of the power consumption index obtained after synthesis is usually obtained by mapping the actual power consumption result to [0,1] according to a preset interval, and is used to characterize the relative power consumption level of the candidate implementation. Indicates candidate code The normalized values ​​of the performance metrics obtained after synthesis are usually normalized for timing delay and the highest operating frequency performance results, and are used to characterize the relative performance level of the candidate implementation. Indicates candidate code The normalized value of the area index obtained after synthesis is obtained by normalizing the results such as the equivalent number of gates, the number of cells, or the area of ​​the layout, and is used to characterize the relative level of the area of ​​the candidate implementation.

[0041] S023-S024: Training the RM-2 reward model; The reward model RM-1 was selected as the base model, and the preference dataset was used. By conducting contrastive learning, a reward model RM-2 is obtained through contrastive learning based on preference pairs.

[0042] It should be noted that the loss function of the reward model RM-2 based on contrastive learning of preference pairs is calculated using the same formula as the loss function of the reward model RM-1 based on contrastive learning of preference pairs.

[0043] S025-S027: Using the reward model RM-2, reinforcement learning algorithms (GRPO / Reinforce++) are used to perform reinforcement learning on the intermediate policy model (DeepseekCoder-SFT-RL1) to generate the final policy model (DeepseekCoder-SFT-RL2).

[0044] In an embodiment of the present invention, the training algorithm for the reward model RM-1 is as follows:

[0045] In an embodiment of the present invention, the training algorithm for the reward model RM-2 is as follows:

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for RTL code generation reinforcement learning based on multi-level hierarchical reward model driving, characterized in that, Includes the following steps: Step 100, constructing a preference dataset ; At step 200, a large language model is selected as a base model, and a preference data set is used Contrastive learning is performed, and a reward model RM-1 is obtained through contrastive learning based on the preference pair. Step 300: Use the reward model RM-1 and the reinforcement learning algorithm to perform reinforcement learning on the large language model to generate an intermediate policy model; Step 400, constructing a preference dataset ; At step 500, the reward model RM-1 is selected as a base model, and a preference dataset is used Contrastive learning is performed, and the reward model RM-2 is obtained through contrastive learning based on the preference pair. Step 600: Use the reward model RM-2 and the reinforcement learning algorithm to perform reinforcement learning on the intermediate policy model to generate the final policy model.

2. The reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model according to claim 1, characterized in that, Constructing a preference dataset comprises: Step 101: Obtain the original dataset, which includes the functional description of the code. Step 102: Randomly sample a problem from the original dataset; Step 103, construct the reward tree RT1; Step 104, within the same question q, when , , , the data set is formed , the judgment formula is as follows: ； wherein, represents the true reward score for the jth candidate answer text for the question q, is a gap interval hyperparameter, is a minimum availability threshold hyperparameter, is a logical AND operator, is the ith candidate answer text for the question q; is the jth candidate answer text for the question q; Step 105, constructing the preference dataset by looping the above steps .

3. The RTL code generation reinforcement learning method driven based on a multi-level hierarchical reward model according to claim 2, characterized in that, The methods for constructing the reward tree RT1 include: For each question q, use the large language model to output k output codes ; performing a syntax check and a functional correctness check on each output , , , , , , For the first layer of the generating nodes, mark, traverse down, generate k child nodes for each marked node, and get the real reward score of each child node, and replace the average value of the real reward score of the child node with the parent node , each marked node at most two layers of recursive down; wherein is a hyperparameter, represents the true reward score for the i-th candidate answer text for question q; In each round of traversing down the mark nodes, a large language model is used to combine the , , new inference process is carried out, k new outputs are obtained, and new true reward scores are obtained through functional correctness detection, and finally the update is obtained by backtracking all parent nodes . wherein represents the error information generated by the i-th candidate code during syntax / compile checking and functional testbench execution.

4. The RTL code generation reinforcement learning method driven based on a multi-level hierarchical reward model according to claim 3, characterized in that, The loss function of the reward model RM-1, which is based on contrastive learning of preference pairs, is calculated using the following formula: ； L(t) = -E[log(P(R | S))], (1) where P(R | S) is the probability of the reward R given the state S, and E is the expectation over the distribution of the state S. is the trainable parameter vector of the reward model RM-1. t is the training iteration round; m is the total number of preference sample pairs participating in the calculation in the current training batch; is the input question text; is the reward model RM-1 for the question is the ith candidate answer is the reward score calculated; is the reward model RM-1 for the question is the jth candidate answer is the reward score calculated; and σ is a Sigmoid function. The loss function of the reward model RM-2 based on contrastive learning of preference pairs is calculated using the same formula as the loss function of the reward model RM-1 based on contrastive learning of preference pairs.

5. The reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model according to claim 4, characterized in that, Preference dataset The construction step of the preference dataset The construction step of the preference dataset 20% of the data comes from the preference dataset random sampling; The remaining 80% are generated in the second layer by rules, building the reward tree using the intermediate policy model output k output codes for each question q The second layer generates a set of function preference pairs, a set of composite preference pairs, and a set of PPA preference pairs: Definition set For a candidate set of functional correctness tests, elements are of the form ; set For a candidate set of integration failures, elements are of the form ; set For a candidate set of integration passes, elements are of the form ; set ; represents the lowest threshold of usability for functional correctness; represents the PPA score for the i-th candidate answer text for question q.

6. The reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model according to claim 5, characterized in that, Functional preference pair: within the same question q, for candidate , The following conditions are included in the preference dataset The judgment formula is as follows:​ ； wherein represents the i-th candidate answer text for the question q; represents the j-th candidate answer text for the question q; represents the pass rate of the function of the i-th candidate answer text for the question q; represents the pass rate of the function of the j-th candidate answer text for the question q; The pass rate is calculated as the ratio of the number of successful test cases in the functional testbench to the total number of test cases, and is used to measure the functional correctness of the candidate code.

7. The RTL code generation reinforcement learning method driven based on a multi-level hierarchical reward model according to claim 6, characterized in that, Overall preference pair: The Add preference dataset , and computational expression is as follows: ； ； wherein represents the candidate code index operation that selects the candidate code with the minimum PPA score in the set ; represents the candidate code index operation that selects the candidate code with the minimum PPA score in the set ; represents the candidate code index operation that selects the candidate code with the highest functional pass rate in the set ; represents the candidate code index operation that selects the candidate code with the highest functional pass rate in the set ; represents the PPA worst sample selected according to the PPA score in the set ; represents the PPA worst sample selected according to the PPA score in the set ; represents the sample with the highest functional pass rate in the set 8. The reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model according to claim 7, characterized in that, PPA preferences are incorporated into the dataset PPA preferences are incorporated into the dataset PPA preferences are incorporated into the dataset and The computational expression for PPA is as follows: ； ； wherein represents the operation of selecting the candidate code index in the set such that the PPA score is the maximum, and obtaining the corresponding candidate code ; represents the operation of selecting the candidate code index in the set such that the PPA score is the minimum, and obtaining the corresponding candidate code ; represents the PPA optimal sample selected according to the PPA score in the set ; represents the PPA worst sample selected according to the PPA score in the set .

9. The reinforcement learning method for RTL code generation driven by a multi-level hierarchical reward model according to claim 8, characterized in that, determining when to include data sets , representing the minimum separation threshold for PPA preferences in construction.

10. The RTL code generation reinforcement learning method driven based on a multi-level hierarchical reward model according to claim 9, characterized in that, The formula for calculating PPA scores is as follows: ； wherein , , are a power consumption index super parameter, a performance index super parameter and an area index super parameter, respectively, and ; ; represents a weight coefficient of the power consumption index; represents a weight coefficient of the performance index; represents a weight coefficient of the area index; represents a normalized value of the power consumption index obtained after the synthesis of the candidate code represents a normalized value of the power consumption index obtained after the synthesis of the candidate code represents a normalized value of the performance index obtained after the synthesis of the candidate code represents a normalized value of the performance index obtained after the synthesis of the candidate code represents a normalized value of the area index obtained after the synthesis of the candidate code represents a normalized value of the area index obtained after the synthesis of the candidate code

Images