Deployment method of deep reinforcement learning model based on NPU and related equipment
By using NPU to train SAC algorithms on the DSP chip and combining FLASH and SDRAM storage mechanisms, the resource limitation problem of reinforcement learning algorithms on embedded devices is solved, and low-power and high-efficiency model deployment is achieved to meet real-time application needs.
Patent Information
- Application Number
- CN202510589386.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2025-08-08
AI Technical Summary
The existing reinforcement learning algorithms have resource limitations in deploying on embedded devices, resulting in low energy efficiency and difficulty in meeting real-time and computing needs.
The deep reinforcement learning model deployment method based on NPU is adopted, and the SAC algorithm is trained using NPU on the DSP chip, model parameters are stored through FLASH, and intermediate data is cached in SDRAM, combining data encryption and dual-channel redundancy mechanisms to achieve efficient and reliable model deployment.
It realizes the deployment of deep reinforcement learning models with low power consumption and high efficiency, improves training efficiency and real-time processing capabilities, enhances the stability and security of the model, and adapts to model parameter storage and transmission of different scales.
Smart Images
Figure CN120447914A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of embedded chip technology, and in particular to a deployment method and related equipment for a deep reinforcement learning model based on an NPU. Background Art
[0002] Reinforcement learning (RL), a key branch of machine learning, has made significant progress in recent years and demonstrated strong application potential in many fields, such as gaming, robotic control, and autonomous driving. However, the deployment of these algorithms on hardware devices still faces a series of challenges. Existing RL algorithm training typically relies on general-purpose computing resources such as GPUs or CPUs, and is typically deployed on the following types of hardware devices: 1) Cloud-based deployment: Many RL algorithms are initially trained and deployed on cloud servers. These servers are typically equipped with high-performance CPUs, GPUs, or Tensor Processing Units (TPUs), capable of processing large datasets and complex model structures. However, this deployment approach is limited by network stability and latency, and poses risks regarding data privacy and security. 2) Edge computing-based deployment: Edge computing reduces latency and improves response speed by offloading data processing and decision-making tasks from the cloud to devices closer to the data source. However, edge devices have high deployment requirements, making them unsuitable for many scenarios, such as drone control. 3) Hardware-based deployment: Dedicated hardware accelerators, such as Google's Tensor Processing Unit (TPU), are optimized for specific algorithms and provide efficient computing power. However, this specialized hardware is expensive and has limited versatility, making it difficult to extend to other types of applications. 4) FPGA- and ASIC-based deployment: This approach uses field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). While this hardware can achieve hardware-level acceleration of algorithms, it also suffers from long development cycles, high costs, and limited flexibility.
[0003] In summary, existing deployment methods are limited in these areas. Energy efficiency is a key consideration for mobile and embedded devices. Existing reinforcement learning algorithms often require a large amount of high-performance general-purpose computing resources, which can lead to rapid battery drain and reduce device usage time. With the increasing application of deep reinforcement learning (RL) in complex decision-making problems, how to efficiently deploy RL algorithms to meet the real-time and energy efficiency requirements of embedded devices has become a pressing technical challenge. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned existing technologies and provide a deployment method and related equipment for a deep reinforcement learning model based on NPU, so as to solve the technical problem of resource limitations in current deep learning algorithms.
[0005] The purpose of the present invention is achieved by the following technical solutions: In a first aspect, the present invention provides a method for deploying a deep reinforcement learning model based on an NPU. The method is applied on a DSP chip, and the method specifically includes: Use NPU to train the SAC algorithm and obtain the parameters of the deep reinforcement learning model; Convert the trained deep reinforcement learning model parameters into a setting language and store them in the FLASH on the DSP chip according to the setting language; When the DSP chip is running, it loads the deep reinforcement learning model parameters from the FLASH to the memory, and caches the intermediate data through the SDRAM to complete the deployment of the deep reinforcement learning model; The SAC algorithm includes a policy network, a target Q network and a value network.
[0006] As a further improvement of the present invention, the SAC algorithm is trained using the NPU, specifically including: A multi-layer perceptron is used to construct a policy network, a value network, and a target Q network; the target Q network includes a reward factor, a discount factor, a minimum Q value, and an entropy term. Initialize the policy network, value network, and target Q network, set the parameters of the target Q network to be the same as those of the value network, and initialize the optimizer; Optimize forward propagation and reverse gradient calculations through the NPU's matrix acceleration unit; Iteratively calculate the target Q value and update the policy network and value network through the optimizer; Update the parameters of the value network to the target Q network; Until the value network output is close to the target Q value, the action generated by the policy network can maximize the expected reward.
[0007] As a further improvement of the present invention, the deep reinforcement learning model introduces an entropy regularization term during the training process to balance the policy network according to the entropy regularization term.
[0008] As a further improvement of the present invention, the trained deep reinforcement learning model parameters are converted into a setting language and stored in the FLASH on the DSP chip according to the setting language, specifically including: Get deep reinforcement learning model parameters in tensor format; Convert the tensor parameters into a NumPy array and generate a binary stream based on the NumPy array; split the binary stream into data blocks of fixed-length hexadecimal strings, and add an index and set a checksum field to the header of each data block; The divided data blocks are encapsulated according to the storage pages of the FLASH on the DSP chip; Upload the data blocks separately through the data transmission channel.
[0009] As a further improvement of the present invention, data transmission adopts a dual-channel redundancy mechanism, specifically including: The main channel transmits data blocks via TCP protocol; The auxiliary channel sends the parameter summary in the data block through the UDP protocol for real-time verification of data integrity.
[0010] As a further improvement of the present invention, uploading the data blocks through the data transmission channel also includes encrypted transmission, specifically including: using Huffman coding to losslessly compress the data blocks, and then encrypting them with AES-256 algorithm before transmission.
[0011] As a further improvement of the present invention, the DSP chip loads the deep reinforcement learning model parameters from FLASH to the memory during operation, and caches the intermediate data through SDRAM, specifically including: Pre-allocate the computation buffers for the policy network and the value function network in SDRAM; Directly load parameters from FLASH to SDRAM through the DMA controller, bypassing CPU intervention; A dynamic memory pool management algorithm is used to dynamically allocate and release SDRAM space according to computing task requirements.
[0012] In a second aspect, the present invention provides an NPU-based deep reinforcement learning model deployment system for implementing the above-mentioned NPU-based deep reinforcement learning model deployment method, including: The training module uses the NPU to train the SAC algorithm to obtain the parameters of the deep reinforcement learning model; the SAC algorithm includes a policy network, a target Q network, and a value network; The deployment module converts the trained deep reinforcement learning model parameters into a setting language and stores them in the FLASH on the DSP chip according to the setting language; The storage module loads the deep reinforcement learning model parameters from FLASH to memory when the DSP chip is running, caches intermediate data through SDRAM, and completes the deployment of the deep reinforcement learning model.
[0013] As a further improvement of the present invention, the deployment module includes a parameter conversion submodule, and the parameter conversion submodule specifically includes: The trained deep reinforcement model parameters are converted into multiple hexadecimal data blocks, which are losslessly compressed using Huffman coding and then encrypted using the AES-256 algorithm before transmission.
[0014] In a third aspect, the present invention provides an electronic device, wherein the electronic device is equipped with a DSP chip, and the electronic device includes: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the above-mentioned NPU-based deep reinforcement learning model deployment method.
[0015] The beneficial effects of the present invention are as follows: the NPU-based deep reinforcement learning model deployment method provided by the present invention utilizes the NPU to train the SAC algorithm, and converts the trained model parameters into the setting language of the DSP chip and stores them in FLASH. The DSP chip loads the parameters from FLASH during operation and caches the intermediate data through SDRAM to complete the deployment of the deep reinforcement learning model. This method combines the efficient computing power of the NPU and the real-time processing capability of the DSP chip to achieve low-power and high-efficiency deployment of deep reinforcement learning models. Training the SAC algorithm through the NPU can significantly accelerate the training process and improve training efficiency. The parallel computing capability of the NPU makes it possible to train large-scale models. The DSP chip performs well in the inference stage and can process complex deep reinforcement learning models in real time to meet the needs of real-time applications. The FLASH memory has low power consumption characteristics and is suitable for use in resource-constrained embedded systems. The fast loading mechanism from FLASH to memory and the caching mechanism of SDRAM of the present invention enable the model to start quickly and respond in real time.
[0016] Furthermore, utilizing the NPU's matrix acceleration unit significantly increases the speed of forward propagation and reverse gradient calculations, significantly reducing training time. Soft updates to the target Q network parameters increase training stability and avoid drastic fluctuations during training. The introduction of an entropy term increases the exploratory nature of the policy, enabling the policy network to generate diverse actions and avoid local optimal solutions. The NPU's dedicated hardware acceleration and parallel computing capabilities make resource utilization more efficient and reduce computing costs.
[0017] Furthermore, introducing an entropy regularization term during deep reinforcement learning model training can significantly enhance the policy network's exploratory capabilities, improve training stability, improve sample efficiency, enhance policy robustness, and optimize policy distribution. Entropy regularization enables the policy network to more effectively utilize sample information during training. This is because diverse action generation provides more information about the environment, accelerating the learning process.
[0018] Furthermore, converting the model parameters into a binary stream can significantly reduce data storage space and improve storage efficiency. Binary representation is naturally efficient in computer systems and is suitable for large-scale data storage. By encapsulating data blocks into FLASH storage pages, the storage space of the FLASH memory can be fully utilized to avoid wasting storage space. Adding index and check fields in each data block can effectively ensure the order and integrity of the data blocks. The index field facilitates data recovery, and the check field can detect and correct errors during data transmission or storage. By utilizing the high-speed data transmission channel provided by the DSP chip, parallel transmission of data blocks can be achieved, further improving transmission efficiency. This method can adapt to deep reinforcement learning model parameters of different sizes. By adjusting the fixed length of the data block and the capacity of the FLASH storage page, model parameters of different sizes can be flexibly stored.
[0019] Furthermore, during data transmission and storage, data encryption technology is combined to encrypt model parameters to prevent data leakage and tampering. Through the access control mechanism, access rights to model parameters stored in FLASH can be restricted, improving data security.
[0020] Furthermore, the primary channel uses the TCP protocol to ensure the order and integrity of data blocks. TCP's automatic retransmission mechanism effectively mitigates packet loss and errors in the network, ensuring reliable data transmission. The secondary channel uses the UDP protocol to send parameter summaries for real-time data integrity verification. By comparing parameter summaries, errors during data transmission can be promptly detected, enhancing data reliability. The dual-channel redundancy mechanism provides redundant data transmission paths. Even if the primary channel fails, the secondary channel can still provide parameter summaries for verification, enhancing system robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0022] Figure 1 Schematic diagram of the overall framework of the method proposed in the present invention.
[0023] Figure 2 This is a flowchart of the method proposed in the present invention for training the NPU model and deploying it on the DSP.
[0024] Figure 3This is a flow chart of the method proposed by the present invention for performing intermediate step reasoning on DSP using SDRAM. DETAILED DESCRIPTION
[0025] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0026] Explanation of terms: NPU: Neural Processing Unit, neural network processor.
[0027] CPU: Central Processing Unit, central processing unit.
[0028] GPU: Graphics Processing Unit, graphics processing processor.
[0029] TPU: Tensor Processing Unit, tensor processing processor.
[0030] FPGA: Field-Programmable Gate Array, field programmable gate array.
[0031] DSP: Digital Signal Processor.
[0032] FLASH: A non-volatile memory technology.
[0033] SDRAM: Synchronous Dynamic Random-Access Memory, synchronous dynamic random access memory.
[0034] SAC: Soft Actor-Critic, soft actor-critic algorithm.
[0035] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings and specific embodiments. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.
[0036] Example 1 like Figures 1 to 3 As shown, this embodiment provides a method for deploying a deep reinforcement learning model based on an NPU, which is applied to a DSP chip. The following is a specific implementation method.
[0037] like Figure 1As shown, the deployment method steps include: The SAC algorithm is trained using the NPU on the DSP chip to obtain the parameters of the deep reinforcement learning model. The training parameters of the deep reinforcement learning model include network weights and bias values. The NPU used in this embodiment is a Huawei NPU. As a neural network processor, the Huawei NPU (Neural Processing Unit) has stronger neural network computing acceleration capabilities than traditional GPUs and CPUs and is suitable for neural network models that require a large number of matrix operations. By training the SAC algorithm on the Huawei NPU, this embodiment improves the operating efficiency of the embedded reinforcement learning algorithm.
[0038] The SAC algorithm is an entropy-based deep reinforcement learning algorithm consisting of three main networks: a policy network, a target Q network (i.e., a target-value network), and a value network (i.e., a value function estimator). In this embodiment, two target Q networks and two value networks are used. The policy network generates actions. Its input is the current state, and its output is the probability distribution of the action. The value network is used to evaluate the value of an action, and its input is .
[0039] The policy network and value network use a multi-layer perceptron (MLP) architecture, comprised of several fully connected layers and using the ReLU activation function. Furthermore, this example uses the Adam optimizer to optimize the policy network and the two value networks separately. Different learning rates are set for the policy network and the two value networks, and cosine annealing is used to accelerate model training.
[0040] The SAC algorithm is trained using Huawei's NPU, specifically including: A multi-layer perceptron is used to construct a policy network, a value network and a target Q network; the target Q network includes a reward factor, a discount factor, a minimum Q value and an entropy term.
[0041] Initialize the policy network, value network, and target Q network, set the parameters of the target Q network to be the same as those of the value network, and initialize the optimizer; Optimize forward propagation and reverse gradient calculations through the NPU's matrix acceleration unit; Iteratively calculate the target Q value and update the policy network and value network through the optimizer; Update the parameters of the value network to the target Q network; Until the value network output is close to the target Q value, the action generated by the policy network can maximize the expected reward.
[0042] The deep reinforcement learning model introduces an entropy regularization term during the training process to balance the policy network according to the entropy regularization term.
[0043] The trained deep reinforcement learning model parameters are converted into a setting language and stored in the FLASH on the DSP chip according to the setting language.
[0044] Leveraging the NPU's matrix acceleration unit, the speed of forward propagation and reverse gradient calculations is significantly increased, significantly reducing training time. Soft updates to the target Q network parameters increase training stability and avoid drastic fluctuations during training. The introduction of an entropy term increases the exploratory nature of the policy, enabling the policy network to generate diverse actions and avoid local optimal solutions.
[0045] Specifically, obtain the deep reinforcement learning model parameters in tensor format; Convert the tensor parameters to NumPy arrays and generate a binary stream based on the NumPy arrays. Split the binary stream into fixed-length hexadecimal string data blocks and add an index and set a checksum field to the header of each data block. The divided data blocks are encapsulated according to the storage pages of the FLASH on the DSP chip; The data blocks are uploaded separately through the data transmission channel; the data transmission channel in this embodiment adopts a dual-channel redundancy mechanism, specifically including: the main channel transmits the data block through the TCP protocol; the auxiliary channel sends the parameter summary in the data block through the UDP protocol for real-time verification of data integrity.
[0046] In addition, to prevent data leakage and tampering during data transmission, this embodiment also adds an encrypted transmission mechanism. After the data block is losslessly compressed using Huffman coding, it is encrypted using the AES-256 algorithm before transmission.
[0047] When the DSP chip is running, it loads the deep reinforcement learning model parameters from FLASH to the memory, and caches the intermediate data through SDRAM to complete the deployment of the deep reinforcement learning model.
[0048] Specifically, the computation buffers for the policy network and the value function network are pre-allocated in SDRAM; Directly load parameters from FLASH to SDRAM through the DMA controller, bypassing CPU intervention; A dynamic memory pool management algorithm is used to dynamically allocate and release SDRAM space based on computing task requirements. By converting the trained deep reinforcement learning model parameters into a configuration language and storing them in the Flash memory on the DSP chip according to the aforementioned steps, efficient, reliable, and easy-to-manage and maintain data storage is achieved. This method not only improves data storage and transmission efficiency, but also improves the overall efficiency of the data storage and transmission.
[0049] The training data used in this embodiment is obtained by the interaction between an intelligent agent (in this embodiment, the intelligent agent is a drone) and an environment. The environment used is an aircraft simulation training platform. The aircraft simulation training platform is a simulation environment designed for aircraft, allowing intelligent agents to interact and train in a virtual environment. In this environment, the intelligent agent collects data through interaction with the environment, including state, action, reward, next state, and completion indicator.
[0050] like Figure 1 As shown in Figure 2, the interactor acts as the interface between the agent and the environment, and is responsible for coordinating the agent's actions and the environment's responses. Figure 1 In the system shown, the interactor communicates with the external system through the TCP protocol, and its core function focuses on sending and receiving data.
[0051] As the agent performs actions, the environment transitions from its current state to the next. This involves the processing and transmission of various data types, for which the struct library plays a key role. The struct library in Python is specifically designed to process data in C structure format. It serializes data such as model parameters, agent actions, and state retrieved from the environment, converting this data into a byte stream format. This ensures that the data format complies with the requirements of the network transmission protocol, enabling efficient and accurate data transmission between different computing platforms (primarily digital signal processors (DSPs) and other systems).
[0052] In addition, model decision-making is also an important part of the entire process. Model decision-making refers to the process in which the agent chooses an action based on the current state and the learned strategy. Figure 1 In the process of model decision making, there are two steps: first, the model parameters are read from FLASH, and then the inference operation is performed on the DSP.
[0053] The entire logic revolves around the interactor, data processing (with the help of the struct library), and model decision-making. These parts work together to realize the interaction between the intelligent agent and the environment and the decision-making actions of the intelligent agent.
[0054] Example 2 As a preferred embodiment in Example 1, it specifically includes the following steps.
[0055] 1. Initialize the network and optimizer: Initialize the Actor network (policy network) and two Critic networks (value function estimators). Initialize the target Critic network and set its parameters to the same as the Critic network. Initialize the optimizer, typically using the Adam optimizer.
[0056] 2. Experience Replay: Several agents interact with the environment and use a ReplayBuffer to store states, actions, rewards, next states, and completion flags. By porting the agent's algorithmic reasoning model to the Huawei NPU framework, the agent's reasoning speed can be accelerated, enabling the parallel operation of a large number of agents and significantly improving the efficiency of collecting samples from the agent-environment interaction.
[0057] Huawei's Ascend NPU provides powerful computing capabilities, significantly accelerating the algorithmic reasoning of intelligent agents. This means that when executing forward and backward propagation of deep learning models, the NPU can handle a large amount of parallel computing, thereby reducing the time required for intelligent agent training and reasoning.
[0058] Parallel operation: Leveraging the parallel processing capabilities of Huawei's NPUs, multiple agents can be run simultaneously, helping to collect more sample data in the same amount of time. Through data parallelism, multiple NPUs can work together to process data from different agents, enabling more efficient sample collection.
[0059] Model migration: Migrating the agent's algorithm inference model to the Huawei NPU framework can be achieved using the torch-npu library, which provides an interface for running PyTorch models on the Ascend NPU. After the model is migrated, the dedicated AI computing power of the NPU can be utilized to further improve model operation efficiency.
[0060] Using the ATC tool provided by Huawei, mainstream open source framework models can be converted to Ascend models, which is particularly useful for migrating PyTorch models to the Ascend platform. This makes it easy to migrate the SAC algorithm's policy and value network to the Ascend platform, ensuring the model runs at optimal performance on the NPU.
[0061] The specific steps for storing algorithm parameters in the on-chip FLASH of the DSP chip and the deployment process of the model framework are summarized as follows.
[0062] 1. Model training: Train the agent in a simulation training platform (such as FlyAndDodge) to obtain optimized model parameters.
[0063] 2. Model saving: Use PyTorch's torch.save function to save the trained model parameters as a .pt file.
[0064] 3. Parameter conversion: Use the load_ckpt method to convert the parameters in the model file into hexadecimal data. This process first converts the model parameters from the PyTorch tensor format to a NumPy array, and then converts them into the corresponding hexadecimal string according to the specified pack_type (float or double).
[0065] 4. Data Packaging: Pack the converted hexadecimal data and prepare it for transmission via TCP. This requires converting the hexadecimal string into binary data and dividing it into blocks according to the max_bytes_per parameter.
[0066] 5. Data transmission: Use TCP communication to send the packaged data to the DSP chip. This requires first establishing a TCP connection and then sending the data through this connection.
[0067] 6. Data Reception and Storage: After receiving data, the DSP chip stores it in the on-chip FLASH memory. The DSP's memory management module (MEM module) provides memory control, including dynamic memory allocation and memory release. These API functions can be used to manage model parameters in the FLASH memory.
[0068] 7. Model Loading and Execution: Implementing model parameter loading and execution, as well as temporary storage of intermediate data, on the DSP chip requires a series of precise steps. First, the DSP's firmware supports loading model parameters from the on-chip Flash into memory. This process is typically performed by the DSP's bootloader or initialization code, which is responsible for reading the parameters from Flash and loading them into memory. Subsequently, the memory management unit (MMU) or similar memory management mechanism allocates memory space for model parameters and intermediate data, ensuring efficient management of data storage and access. When executing inference tasks, the DSP's processor core reads the model parameters from memory and performs calculations. During this process, the generated intermediate data needs to be temporarily stored. The DMA controller is responsible for automatically transferring data between the processor core and SDRAM to improve data transfer efficiency and reduce CPU burden.
[0069] 3. Training loop: A. Sample a batch of data from the Replay Buffer.
[0070] B. Calculate the target Q value, including the reward, discount factor, minimum Q value and entropy term.
[0071] C. Update the Critic network so that its output is close to the target Q value.
[0072] D. Update the Actor Network so that the actions it generates can maximize the expected reward while taking into account the entropy term to encourage exploration.
[0073] 4. Soft update: Regularly update the parameters of the critic network to the target critic network to stabilize the training process.
[0074] 5. Entropy Regularization: The SAC algorithm introduces an entropy regularization term to encourage the policy network to explore more action spaces. The balance between exploration and exploitation is achieved by optimizing α (entropy coefficient).
[0075] Example 3 This embodiment provides an NPU-based deep reinforcement learning model deployment system for implementing the NPU-based deep reinforcement learning model deployment method in Example 1. The system includes: The training module uses the NPU to train the SAC algorithm to obtain the parameters of the deep reinforcement learning model. The SAC algorithm includes a policy network, a target Q network, and a value network. The NPU used in this embodiment is a Huawei NPU. As a neural network processor, the Huawei NPU (Neural Processing Unit) has stronger neural network computation acceleration capabilities than traditional GPUs and CPUs and is suitable for neural network models that require extensive matrix operations. By training the SAC algorithm on the Huawei NPU, this embodiment improves the operational efficiency of the embedded reinforcement learning algorithm.
[0076] The deployment module converts the trained deep reinforcement learning model parameters into a setting language and stores them in the FLASH on the DSP chip according to the setting language; The storage module loads the deep reinforcement learning model parameters from FLASH to memory when the DSP chip is running, caches intermediate data through SDRAM, and completes the deployment of the deep reinforcement learning model.
[0077] In addition, the deployment module includes a parameter conversion submodule, which specifically includes: The trained deep reinforcement model parameters are converted into multiple hexadecimal data blocks, which are losslessly compressed using Huffman coding and then encrypted using the AES-256 algorithm before transmission.
Claims
1. A deep reinforcement learning model deployment method based on NPU, characterized in that: The deployment method is applied on a DSP chip, and the deployment method specifically includes: Use NPU to train the SAC algorithm and obtain the parameters of the deep reinforcement learning model; Convert the trained deep reinforcement learning model parameters into a setting language and store them in the FLASH on the DSP chip according to the setting language; When the DSP chip is running, it loads the deep reinforcement learning model parameters from the FLASH to the memory, and caches the intermediate data through the SDRAM to complete the deployment of the deep reinforcement learning model; The SAC algorithm includes a policy network, a target Q network and a value network.
2. The NPU-based deep reinforcement learning model deployment method according to claim 1, characterized in that: Using the NPU to train the SAC algorithm includes: A multi-layer perceptron is used to construct a policy network, a value network, and a target Q network; the target Q network includes a reward factor, a discount factor, a minimum Q value, and an entropy term. Initialize the policy network, value network, and target Q network, set the parameters of the target Q network to be the same as those of the value network, and initialize the optimizer; Optimize forward propagation and reverse gradient calculations through the NPU's matrix acceleration unit; Iteratively calculate the target Q value and update the policy network and value network through the optimizer; Update the parameters of the value network to the target Q network; Until the value network output is close to the target Q value, the action generated by the policy network can maximize the expected reward.
3. The NPU-based deep reinforcement learning model deployment method according to claim 2, characterized in that: The deep reinforcement learning model introduces an entropy regularization term during the training process to balance the policy network according to the entropy regularization term.
4. The NPU-based deep reinforcement learning model deployment method according to claim 1, characterized in that: Convert the trained deep reinforcement learning model parameters into a setting language and store them in the FLASH on the DSP chip according to the setting language, specifically including: Get deep reinforcement learning model parameters in tensor format; Convert the tensor parameters into a NumPy array and generate a binary stream based on the NumPy array; split the binary stream into data blocks of fixed-length hexadecimal strings, and add an index and set a checksum field to the header of each data block; The divided data blocks are encapsulated according to the storage pages of the FLASH on the DSP chip; Upload the data blocks separately through the data transmission channel.
5. The NPU-based deep reinforcement learning model deployment method according to claim 4, characterized in that: Data transmission adopts a dual-channel redundancy mechanism, which includes: The main channel transmits data blocks via TCP protocol; The auxiliary channel sends the parameter summary in the data block through the UDP protocol for real-time verification of data integrity.
6. The NPU-based deep reinforcement learning model deployment method according to claim 4, characterized in that: After uploading the data blocks through the data transmission channel, the method further includes encrypted transmission, specifically including: using Huffman coding to losslessly compress the data blocks, and then encrypting them with AES-256 algorithm before transmission.
7. The NPU-based deep reinforcement learning model deployment method according to claim 1, characterized in that: When the DSP chip is running, it loads the deep reinforcement learning model parameters from the FLASH memory and caches the intermediate data through the SDRAM. Specifically, it includes: Pre-allocate the computation buffers for the policy network and the value function network in SDRAM; Directly load parameters from FLASH to SDRAM through the DMA controller, bypassing CPU intervention; A dynamic memory pool management algorithm is used to dynamically allocate and release SDRAM space according to computing task requirements.
8. A deep reinforcement learning model deployment system based on an NPU, used to implement the deep reinforcement learning model deployment method based on an NPU according to any one of claims 1 to 7, characterized in that: include: The training module uses the NPU to train the SAC algorithm to obtain the parameters of the deep reinforcement learning model; the SAC algorithm includes a policy network, a target Q network, and a value network; The deployment module converts the trained deep reinforcement learning model parameters into a setting language and stores them in the FLASH on the DSP chip according to the setting language; The storage module loads the deep reinforcement learning model parameters from FLASH to memory when the DSP chip is running, caches intermediate data through SDRAM, and completes the deployment of the deep reinforcement learning model.
9. The NPU-based deep reinforcement learning model deployment system according to claim 8, characterized in that The deployment module includes a parameter conversion submodule, which specifically includes: The trained deep reinforcement model parameters are converted into multiple hexadecimal data blocks, which are losslessly compressed using Huffman coding and then encrypted using the AES-256 algorithm before transmission.
10. An electronic device, characterized in that: The electronic device is equipped with a DSP chip, and the electronic device includes: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the NPU-based deep reinforcement learning model deployment method according to any one of claims 1 to 7.