Reinforcement learning-based ai processor architecture optimization method and device, and storage medium

Abstract

The present invention aims to provide a method, apparatus, and storage medium for optimizing AI processor architecture based on reinforcement learning. The method includes: constructing an AI processor evaluation model using a convolutional neural network and a hardware platform; obtaining a function from the AI ​​processor evaluation model that yields the action with the maximum reward; setting up an experience replay cache pool to store sampled data; when sufficient samples are collected, constructing a policy function using the function that yields the action with the maximum reward and the neural network, and interfering with the policy function by adding Gaussian noise; obtaining training samples from the experience replay cache pool; training the policy function; and obtaining the maximum reward through the policy function to obtain the parameters of the optimized AI processor architecture. The method described in this invention can achieve better sample utilization, stability, and exploration capability, and can obtain optimal AI processor architecture design parameters.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to a method, apparatus, and storage medium for optimizing AI processor architecture based on reinforcement learning. Background Technology

[0002] Convolutional Neural Networks (CNNs) have wide applications in computer vision, pattern recognition, and object detection. However, as the size and complexity of network models increase, the computational and energy demands of CNNs also grow. Field-Programmable Gate Arrays (FPGAs), due to their customizable compute units (PEs) and data paths, offer better energy efficiency in the inference phase of CNN implementations and are increasingly favored by academia and industry. While custom computing platforms like FPGAs hold great potential, designing an efficient AI processor on top of the massive and irregular network structure of CNNs is exceptionally complex. Furthermore, designing AI processors on resource-constrained platforms requires not only pursuing high performance but also ensuring that power consumption and resource utilization do not exceed the constraints of the hardware platform. These constraints lead to trade-offs between parameters and result in irregular processor architectures.

[0003] Current optimization methods for AI processor architectures on resource-constrained FPGA platforms include exhaustive search and random search. Exhaustive search methods include grid search (GS), depth-first search (DFS), and breadth-first search (BFS). Exhaustive search is a commonly used search method for AI processor architectures; although it is inefficient, it ensures global optimization of the search results.

[0004] Therefore, exhaustive search is a popular choice when processor architecture parameters and designs are not too complex. However, it has a significant drawback: exhaustive search is very inefficient when dealing with processor architectures that have many complex parameters and large design space.

[0005] Random search, by increasing randomness, makes it easier to discover optimization points in efficient AI processor architectures, thus being more efficient than exhaustive search methods in terms of time cost. However, it is difficult to guarantee the global optimality and stability of the search results. In practical problems, optimization objectives often exhibit certain regularities as design parameters change, and their distribution is not completely random.

[0006] Faced with complex AI processor architectures, AI processor architectures suffer from a lack of effective exploration methods. The associated parameters and hardware resource constraints constitute a large-scale and irregular design space, making it difficult for traditional exhaustive search methods, random search methods, or general optimization algorithms to balance exploration efficiency and exploration space. Summary of the Invention

[0007] The purpose of this invention is to provide a method, apparatus and storage medium for optimizing AI processor architecture based on reinforcement learning, so as to solve at least one technical problem in the prior art.

[0008] The technical solution of this invention is:

[0009] A reinforcement learning-based method for optimizing AI processor architecture includes:

[0010] An AI processor evaluation model is constructed using convolutional neural networks and a hardware platform, and a function is obtained from the AI ​​processor evaluation model that yields the action with the greatest reward.

[0011] Set up an experience replay cache pool to store the sampled samples in the cache pool;

[0012] When enough samples are collected, a policy function is constructed by combining the function that takes the action that will yield the maximum reward with the neural network, and Gaussian noise is added to the policy function to disturb it, and training samples are obtained from the experience replay cache pool.

[0013] The policy function is trained, and the maximum reward is obtained through the policy function to obtain the parameters of the optimized AI processor architecture.

[0014] The "AI processor evaluation model constructed through convolutional neural networks and hardware platforms" includes:

[0015] Set the design points for AI processor architecture parameters;

[0016] Set the performance metrics of the AI ​​processor architecture, with the computational latency of the AI ​​processor architecture as the target, and take any action that can maximize the reduction of latency as the maximum reward for that action.

[0017] A function that yields the sum of all rewards from the current state to the final state, and the action that maximizes the reward in the current state.

[0018] The phrase "using maximum reward to approximate gradient to train AI processor evaluation models and obtain policy function types" includes:

[0019] The AI ​​processor architecture parameter design points include: parameters in the AI ​​processor architecture and indicator: t;

[0020] s t ={a1, a2, a3, ..., a t ,t}.

[0021] The phrase “setting the performance metrics of the AI ​​processor architecture” includes one or more of the following: latency, power consumption, and hardware resource utilization.

[0022] The performance indicators are expressed by the following formula:

[0023] ;

[0024] Among them, O i (s t ) refers to s t Performance metrics obtained under the given conditions; P j (s t ) is the constraint indicator; C j It is the threshold of the constraint indicator; j It is the penalty factor for the constraint indicator;

[0025] The penalty factor l of the constraint index j The expression is as follows:

[0026] .

[0027] The phrase "obtaining the sum of all rewards from the current state to the final state" includes:

[0028] The sum of all the rewards is obtained using the following formula:

[0029]

[0030] Wherein, R(s) t ) is the reward at time t, and γ is the discount factor.

[0031] The "function that takes the action that yields the greatest reward in the current state" includes:

[0032] The function that yields the maximum reward is obtained through training, denoted as π(a|s);

[0033] The policy function, which is the action that yields the maximum reward, is represented by a neural network and is denoted as π(a|s, θ), where θ is a trainable parameter in the neural network.

[0034] The phrase "using increased Gaussian noise to interfere with the strategy function" includes:

[0035] When sampling samples in the experience replay buffer, Gaussian noise interference n[μ(a), σ] is added to the policy function π(a|s, θ), resulting in:

[0036] π noise (a|s, θ)=π(a|s, θ)+n[μ(a), σ];

[0037] The Gaussian noise n[μ(a), σ] follows a normal distribution N[μ(a), σ], where μ(a) is the mean and σ is the standard deviation.

[0038] The "training of the policy function" includes:

[0039] The policy function π(a|s, θ) is trained using gradient descent with the state value function V(s, θ). The gradient formula for the state value function V(s, θ) is as follows:

[0040]

[0041] The gradient is approximated using the trajectory obtained from Monte Carlo sampling, and expressed as:

[0042]

[0043] And select the trajectory τ with the highest return from multiple trajectories τ. max The trajectory for gradient descent is represented as:

[0044]

[0045] The parameters θ of the neural network are updated using gradient ascent, i.e.:

[0046] Where τ: s0→a0→s1→…→s T-1 →a T-1 →s T This refers to a complete trajectory from the initial point to the end point; G(τ) represents the product of probabilities of sampling a trajectory τ under the policy function π(a|s, θ); t＜T γ t R(s t ), representing the reward of trajectory τ, describes the total reward value that can be obtained through trajectory τ;

[0047] R(s t () is the reward at time t;

[0048] γ is the discount factor.

[0049] An electronic device comprising:

[0050] Storage media, used to store computer programs

[0051] The processing unit exchanges data with the storage medium and is used to execute the computer program through the processing unit when optimizing the AI ​​processor architecture, performing the steps of the reinforcement learning-based AI processor architecture optimization method described above.

[0052] A computer-readable storage medium: the computer-readable storage medium stores a computer program;

[0053] When the computer program is run, it executes the steps of the reinforcement learning-based AI processor architecture optimization method described above.

[0054] The beneficial effects of the present invention include at least the following:

[0055] The method described in this invention is based on a reinforcement learning algorithm. It employs empirical replay to decouple the sampling and learning phases, and uses Gaussian noise interference and maximum reward to approximate the gradient, further refining the sampling and learning phases respectively. Compared to exhaustive search and random search methods, the method described in this invention achieves better sample utilization, stability, and exploration capability, and can obtain optimal AI processor architecture design parameters. Attached Figure Description

[0056] Figure 1 This is a flowchart of the method described in this invention;

[0057] Figure 2 A comparison chart showing the optimization efficiency of AI processor architectures based on general optimization methods. Detailed Implementation

[0058] The present application will now be further described with reference to the accompanying drawings.

[0059] Specific Implementation Example I:

[0060] To improve the design efficiency of resource-constrained convolutional neural network hardware accelerators, the method described in this invention optimizes AI processor architecture based on reinforcement learning, such as... Figure 1 The specific steps are as follows:

[0061] 1. Establishing a reinforcement learning model:

[0062] In the reinforcement learning model described above, "Environment" refers to the AI ​​processor evaluation model composed of a convolutional neural network and a hardware platform; "agent" refers to the designer of this model; and "action" refers to the parameters in the AI ​​processor architecture, as shown in Table 1.

[0063] Table 1. Specific definitions of action and action space

[0064]

[0065] In Table 1, the action at contains a1, a2, and a3. a1 represents PE-size, and the range of values ​​for a1 is A1 = 1×1, 2×2, 3×3, 4×4, and 5×5.

[0066] The state refers to the design points of the AI ​​processor architecture parameters, including a set of actions and indicators: t, which can be represented by the following formula:

[0067] s t ={a1, a2, a3, ..., a t ,t} (1)

[0068] In reinforcement learning models, "reward" represents the reward. In this embodiment, it refers to performance metrics of the AI ​​processor architecture, such as latency, power consumption, and hardware resource utilization. The expression for the reward function is as follows:

[0069]

[0070] Among them O i (s t ) refers to s t The performance metrics obtained under the given conditions, P j (s t ) is the constraint indicator, C j It is the threshold of the constraint indicator, l j It is the penalty factor for the constraint indicator, and its specific expression is as follows:

[0071]

[0072] For example, if the goal is to reduce the computational latency of an AI processor architecture, then latency is a performance metric. If a certain action results in a state that minimizes latency, then that action will receive a large reward.

[0073] The return refers to the sum of all rewards obtained from the current state to the final state, which can also be defined as the reward. Its calculation formula is as follows:

[0074]

[0075] Where R(s) t ) is the reward at time t, and γ is the discount coefficient, which takes the value of γ∈[0,1], indicating that the closer a state is to the current state, the higher its value, and the farther a state is from the current state, the lower its value.

[0076] Policy refers to the function that determines the action that yields the maximum reward in the current state, denoted as π(a|s). Obviously, the policy function π(a|s) is not a function that can be obtained directly, but it can be trained through a neural network. Therefore, the policy function can be represented by a neural network, namely π(a|s, θ), where θ is a trainable parameter in the neural network.

[0077] In this step, the focus of reinforcement learning is to obtain a good policy function through training to guide the action taken in response to the current state, which will yield the maximum reward, and ultimately select the optimal design point for the AI ​​processor architecture optimization parameters.

[0078] 2. Training the policy function:

[0079] The policy function π(a|s, θ) obtained in the previous step is trained by evaluating the value of actions taken in the current state and then selecting the action with the highest value. Therefore, a state value function V(s, θ) is needed to train the policy function π(a|s, θ) using gradient descent. The gradient formula of the state value function V(s, θ) is as follows:

[0080]

[0081] Where τ: s0→a0→s1→…→s T-1 →a T-1 →s T This refers to a complete trajectory from the initial point to the end point;

[0082] It represents the product of probabilities of sampling a trajectory τ under the policy function π(a|s, θ);

[0083] G(τ)=∑ t＜T γ t R(st) represents the reward of trajectory τ, describing the total reward value that can be obtained through trajectory τ;

[0084] γ is the discount factor, and its value range is γ∈[0,1].

[0085] In practical calculations, it is difficult to calculate the rewards and probabilities of all trajectories without knowing the policy function. Therefore, this embodiment uses the trajectories obtained by Monte Carlo sampling to approximate the gradient. Equation (5) can be expressed as follows:

[0086]

[0087] The policy function represented by the neural network described above can be approximated by the maximum reward during training; that is, the trajectory τ with the highest reward is selected from multiple trajectories τ. max As the trajectory of gradient descent, equation (6) can be expressed as follows:

[0088]

[0089] Ultimately, the preferred method is to use gradient ascent to update the parameters θ of the neural network, that is:

[0090]

[0091] 3. Specific steps for optimizing AI processor architecture based on reinforcement learning:

[0092] 3.1 Add an experience replay cache pool:

[0093] Traditional reinforcement learning algorithms immediately train the network using samples obtained from each Monte Carlo sampling, discarding the samples after training. This leads to resource waste and loss of effective samples. Therefore, this embodiment adds an experience replay buffer. Samples obtained from each Monte Carlo sampling are stored in the buffer, the size of which can be set as needed. When enough samples are collected, samples are retrieved from the buffer to train the network. Thus, effective samples can be reused multiple times, and the sampling and learning phases are decoupled, improving resource utilization.

[0094] 3.2 Add Gaussian noise during the sampling phase:

[0095] When sampling samples in the experience replay buffer, the policy function π(a|s, θ) is generally used. However, this can cause the policy function π(a|s, θ) to fall into a certain probability distribution, reducing its exploration capability and making it impossible to find the optimal parameter design point. To increase the exploration capability of the policy function, in this embodiment, Gaussian noise interference n[μ(a), σ] is added to the policy function π(a|s, θ), resulting in:

[0096] π noise (a|s, θ)=π(a|s, θ)+n[μ(a), σ] (9)

[0097] The Gaussian noise n[μ(a), σ] follows a normal distribution N[μ(a), σ], where μ(a) is the mean and σ is the standard deviation. Usually, μ(a) can be set to 0, so that the value of the policy function fluctuates around its original value, which can improve the exploration ability and find the AI ​​processor architecture optimization parameters faster and easier in complex design spaces.

[0098] 3.3 The learning phase uses the maximum value reward to approximate the gradient.

[0099] When training a neural network with a policy function, the gradient can be approximated by maximizing the reward, that is, selecting the trajectory τ with the highest reward among multiple trajectories τ. max The trajectory for gradient descent is shown in Equation (7).

[0100] The complete process steps described in this embodiment are as follows:

[0101]

[0102] Verification process:

[0103] like Figure 2 As shown, the method described in this invention has better performance compared to other optimization algorithms.

[0104] First, VGG-16 and MobileNet-V3 were selected as convolutional neural network models, and computational latency was optimized across different platforms. Here, the method described in this invention is compared with the AI ​​processor architecture optimization performance of exhaustive search, random search, and other reinforcement learning methods.

[0105] An AI processor architecture optimization method based on Random Search (RGS), Exhaustive Search (GA), Bayesian Optimization (BO), the proposed method (RI), Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG) was implemented. The sampling time was limited to 3000. Power was limited to below 50W, and the number of DSPs was limited to below 2800.

[0106] Under the most stringent constraints, this method achieves latency improvements of 1.2x-1.7x (on VGG-16) and 2.3x-4.9x (on MobileNet-V3) at the same sampling time compared to other methods.

[0107] In summary, this embodiment proposes an AI processor architecture optimization method based on reinforcement learning, which addresses the complex and vast design space of complex AI processor architectures. This method is faster than exhaustive search and better than random search in exploring the globally optimal structure.

[0108] The present invention also provides an embodiment:

[0109] An electronic device includes: a storage medium and a processing unit; wherein the storage medium is used to store a computer program, and the processing unit exchanges data with the storage medium for executing the computer program through the processing unit when optimizing an AI processor architecture, performing the steps of the reinforcement learning-based AI processor architecture optimization method as described in Specific Embodiment I.

[0110] A computer-readable storage medium storing a computer program; when the computer program is run, it executes the steps of the reinforcement learning-based AI processor architecture optimization method as described in Specific Embodiment I.

[0111] In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0112] The above descriptions only cover a few specific embodiments of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention. The above-mentioned serial numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

Claims

1. A method for optimizing AI processor architecture based on reinforcement learning, characterized in that, include: An AI processor evaluation model is constructed using convolutional neural networks and a hardware platform, and a function is obtained from the AI ​​processor evaluation model that yields the action with the greatest reward. Set up an experience replay cache pool to store the sampled samples in the cache pool; When enough samples are collected, a policy function is constructed using the function of the action that can obtain the maximum reward and the neural network, and the policy function is disturbed by adding Gaussian noise, and training samples are obtained from the experience replay cache pool. The policy function is trained, and the maximum reward is obtained through the policy function to obtain the parameters of the optimized AI processor architecture; The training of the policy function includes: The policy function π(a|s,θ) is trained using gradient descent with the state value function V(s,θ). The gradient formula for the state value function v(s,θ) is as follows: The gradient is approximated using the trajectory obtained from Monte Carlo sampling, and expressed as: And select the trajectory τ with the highest return from multiple trajectories τ. max The trajectory for gradient descent is represented as: The parameters θ of the neural network are updated using gradient ascent, i.e.: Where τ: s0→a0→s1→…→s T-1 →a T-1 →s T This refers to a complete trajectory from the initial point to the end point; G(τ) represents the product of probabilities of sampling a trajectory τ under the policy function π(a|s,θ); t<T γ t R(s t R(s) represents the reward of trajectory τ, describing the total reward value obtainable through trajectory τ; t ) is the reward at time t; γ is the discount factor.

2. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 1, characterized in that, The AI ​​processor evaluation model, constructed using convolutional neural networks and hardware platforms, includes: Set the design points for AI processor architecture parameters; Set the performance metrics of the AI ​​processor architecture, with the computational latency of the AI ​​processor architecture as the target, and take any action that can maximize the reduction of latency as the maximum reward for that action. A function that yields the sum of all rewards from the current state to the final state, and the action that maximizes the reward in the current state.

3. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 1, characterized in that, The process of obtaining the maximum reward through the policy function to obtain the parameters of the optimized AI processor architecture includes: The AI ​​processor architecture parameter design points include: parameters in the AI ​​processor architecture and indicator: t; s t ={a1,a2,a3,…,a t ,t}。 4. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 2, characterized in that, The performance metrics for the AI ​​processor architecture include one or more of latency, power consumption, and hardware resource utilization.

5. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 2, characterized in that, The process of obtaining the sum of all rewards from the current state to the final state includes: The sum of all the rewards is obtained using the following formula: Wherein, R(s) t ) is the reward at time t, and γ is the discount factor.

6. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 2, characterized in that, The function that determines the action that yields the maximum reward in the current state includes: The function that yields the maximum reward is obtained through training, denoted as π(a|s); The policy function, which is the action that yields the maximum reward, is represented by a neural network and is denoted as π(a|s, θ), where θ is a trainable parameter in the neural network.

7. The AI ​​processor architecture optimization method based on reinforcement learning according to claim 1, characterized in that, The method of interfering with the policy function by adding Gaussian noise includes: When sampling samples in the experience replay buffer, Gaussian noise interference n[μ(a), σ] is added to the policy function π(a|s, θ), resulting in: p noise (α|s,θ)=π(a|s,θ)+n[μ(a),σ]; The Gaussian noise n[μ(a), σ] follows a normal distribution N[μ(a), σ], where μ(a) is the mean and σ is the standard deviation.

8. An electronic device, characterized in that, include: Storage media, used to store computer programs The processing unit exchanges data with the storage medium and is used to execute the computer program through the processing unit when optimizing the AI ​​processor architecture, performing the steps of the reinforcement learning-based AI processor architecture optimization method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is run, it performs the steps of the AI ​​processor architecture optimization method based on reinforcement learning as described in any one of claims 1-7.

Images