Reinforced learning parallel processing accelerator, acceleration method and electronic equipment

Abstract

The invention discloses a reinforcement learning parallel processing accelerator, an acceleration method and electronic equipment, and relates to the technical field of accelerators, and the reinforcement learning parallel processing accelerator comprises the following steps: a controller respectively writes current view features, at least two groups of batch historical view features and instruction sequences into corresponding memories; the instruction loading and distributing part reads and decodes the instruction sequence, distributes parameters and starts a calculation instruction to the calculation part; the data loading control component selects the required feature data from the memory according to the parameters of each calculation layer and loads the feature data to the corresponding feature cache; and after receiving the calculation starting instruction, the calculation component reads various view feature data at the same time and performs parallel processing in combination with the parameters, so that the data processing efficiency can be greatly improved. Through mutual cooperation of the components, the characteristics of the reinforcement learning model can be deeply analyzed, parameters and processing flows are flexibly adjusted according to different reinforcement learning tasks and data characteristics, the processing efficiency of the reinforcement learning model is improved, meanwhile, the resource utilization rate of an accelerator is reduced, and the response speed is high.

Description

Technical Field

[0001] The present invention relates to the technical field of accelerators, and in particular to a reinforcement learning parallel processing accelerator, an acceleration method and an electronic device. Background Art

[0002] The reinforcement learning model is a composite network in terms of architecture, usually composed of a convolutional network, a generative pretrained transformer (GPT), and a feed forward neural network (FFN). During the operation of the model, a large number of computing operations are used to process various types of input data, but these data vary in quantity and scale. Traditionally, such reinforcement learning models are calculated using graphics processing units (GPUs) or some traditional accelerators. However, in the relevant technical solutions, the accelerators do not combine the structural characteristics of such models, and the acceleration effect is not good. Summary of the invention

[0003] The purpose of the present invention is to provide a reinforcement learning parallel processing accelerator, acceleration method and electronic device, which can deeply analyze the characteristics of the reinforcement learning model, improve the processing efficiency of the model, and at the same time reduce the resource utilization rate of the accelerator and improve the acceleration efficiency.

[0004] In order to solve the above technical problems, the present invention provides a reinforcement learning parallel processing accelerator, comprising: A controller, used to write the current view features, at least two groups of batch history view features and the instruction sequence into corresponding memories respectively; An instruction loading and distributing component is connected to the memory, and is used to read the instruction sequence in the memory, decode the read instruction sequence, and distribute the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component; A data loading control component, connected to the memory, the instruction loading distribution component and each of the computing components, respectively, for selecting required feature data from the memory according to the parameters of each layer and loading it into a corresponding feature cache; The computing component is connected to the instruction loading and distributing component, and is used to simultaneously read the current view feature data and at least two groups of batch historical view feature data in the feature cache after receiving the start computing instruction, and perform parallel processing in combination with corresponding memory access parameters and computing parameters to obtain intermediate layer result data and final result data and return them to the controller.

[0005] In order to solve the above technical problems, the present invention also provides an acceleration method of a reinforcement learning parallel processing accelerator, wherein the reinforcement learning parallel processing accelerator is the reinforcement learning parallel processing accelerator provided by the present invention, and the acceleration method comprises: The controller writes the current view features, at least two sets of batch history view features and the instruction sequence into corresponding memories respectively; The instruction loading and distributing component reads the instruction sequence in the memory, decodes the read instruction sequence, and distributes the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component; The data loading control component selects required feature data from the memory according to the parameters of each layer and loads it into the corresponding feature cache; After receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two groups of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, performs parallel processing, obtains the intermediate layer result data and the final result data and returns them to the controller.

[0006] It can be seen from the above technical solution that an electronic device provided by the present invention includes the above reinforcement learning parallel processing accelerator provided by the present invention.

[0007] The beneficial effects of the present invention are that in the above-mentioned reinforcement learning parallel processing accelerator provided by the present invention, the controller writes the current view features, at least two groups of batch historical view features and the instruction sequence into the corresponding memory respectively to prepare for subsequent processing; the instruction loading and distribution component can quickly read and decode the instruction sequence and distribute parameters and start calculation instructions to the calculation component; the data loading control component selects the required feature data from the memory according to the parameters of each calculation layer and loads it to the corresponding feature cache; after receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two groups of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, and performs parallel processing, so that multiple view feature data are read simultaneously and combined with parameters for parallel processing, fully utilizing hardware resources, greatly improving data processing efficiency, reducing processing time, accelerating the operation of reinforcement learning algorithms, and being suitable for rapid decision-making in complex scenarios. The above components have clear division of labor and cooperate with each other. They can deeply analyze the characteristics of the reinforcement learning model, flexibly adjust parameters and processing procedures according to different reinforcement learning tasks and data characteristics, adapt to a variety of application scenarios, and make data reading, instruction processing and calculation execution proceed in an orderly manner, thereby improving the processing efficiency of the reinforcement learning model and reducing the resource utilization rate of the accelerator, achieving a better acceleration ratio and fast response speed.

[0008] In addition, the present invention also provides a corresponding acceleration method and electronic device for a reinforcement learning parallel processing accelerator, which has the same or corresponding technical features as the reinforcement learning parallel processing accelerator mentioned above, and has the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A schematic diagram of the structure of a reinforcement learning parallel processing accelerator provided by an embodiment of the present invention; Figure 2 A schematic diagram of the topological structure of the reinforcement learning parallel processing accelerator provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a flattened splicing assembly provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a stacked parallel assembly provided in an embodiment of the present invention; Figure 5 A schematic diagram of data processing of stacked parallel components provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a query key value splitting component provided in an embodiment of the present invention; Figure 7 A schematic diagram of data processing of a query key value splitting component provided in an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a split and spliced ​​assembly provided by an embodiment of the present invention; Fig. 9 A schematic diagram of data processing of a split and spliced ​​component provided by an embodiment of the present invention; Fig.10 One of the schematic diagrams of the calculation process of the reinforcement learning parallel processing accelerator provided by the embodiment of the present invention; Fig.11 A second schematic diagram of the calculation process of the reinforcement learning parallel processing accelerator provided by an embodiment of the present invention; Fig.12 A flow chart of an acceleration method for a reinforcement learning parallel processing accelerator provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A schematic diagram of the structure of a reinforcement learning parallel processing accelerator provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the reinforcement learning parallel processing accelerator includes: A controller, used to write the current view features, at least two groups of batch history view features and the instruction sequence into corresponding memories respectively; An instruction loading and distributing component is connected to the memory, and is used to read the instruction sequence in the memory, decode the read instruction sequence, and distribute the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component; A data loading control component is connected to the memory, the instruction loading distribution component and each computing component respectively, and is used to select required feature data from the memory according to the parameters of each layer and load it into the corresponding feature cache; The computing component is connected to the instruction loading and distributing component, and is used to simultaneously read the current view feature data and at least two sets of batch historical view feature data in the feature cache after receiving the start computing instruction, and perform parallel processing in combination with the corresponding memory access parameters and calculation parameters to obtain the intermediate layer result data and the final result data and return them to the controller.

[0013] In the above reinforcement learning parallel processing accelerator provided by the embodiment of the present invention, the controller writes the current view feature, at least two groups of batch historical view features and the instruction sequence into the corresponding memory respectively to prepare for subsequent processing; the instruction loading and distribution component can quickly read and decode the instruction sequence and distribute parameters and start calculation instructions to the calculation component; the data loading control component selects the required feature data from the memory according to the parameters of each calculation layer and loads it to the corresponding feature cache; after receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two groups of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, and performs parallel processing, so that multiple view feature data are read at the same time and combined with parameters for parallel processing, making full use of hardware resources, greatly improving data processing efficiency, reducing processing time, accelerating the operation of reinforcement learning algorithms, and being suitable for rapid decision-making in complex scenarios. The above components have clear division of labor and cooperate with each other, can deeply analyze the characteristics of the reinforcement learning model, flexibly adjust parameters and processing procedures according to different reinforcement learning tasks and data characteristics, adapt to a variety of application scenarios, so that data reading, instruction processing and calculation execution are carried out in an orderly manner, improve the processing efficiency of the reinforcement learning model, and reduce the resource utilization rate of the accelerator at the same time, achieve a better acceleration ratio, and fast response speed.

[0014] It should be noted that the controller can use a peripheral component interconnect express (PCIE) controller, which is an interface for high-speed data transmission between the entire system and the outside. The memory can use a high-bandwidth memory (HBM), which is a high-performance memory chip. The present invention can send a data transmission instruction to the PCIE controller through the host computer. For example, after receiving the data transmission instruction from the host computer, the PCIE controller can preload the key parameters of the reinforcement learning network into the corresponding HBM memory. In this way, before the actual reinforcement learning calculation process begins, the relevant data and instructions are loaded into a specific HBM memory in advance, so that they can be quickly obtained during subsequent use, reducing waiting time and improving system operation efficiency.

[0015] The controller can write the current image features and historical image features into the corresponding feature input buffers through direct memory access (DMA). Direct memory access allows external devices to transfer data directly to computer memory without the need for continuous processor involvement.

[0016] The above image can use a bird's-eye view (BEV), which refers to a view of the scene from above the vehicle. BEV is used in fields such as autonomous driving and computer vision. It can convert image data obtained from multiple perspectives (such as cameras at different angles around the vehicle) through algorithms to present environmental information in a perspective similar to looking down from the air, forming a top-view data expression. This data representation method can more intuitively display information such as the position, shape, and relative relationship of objects around the vehicle, which is helpful for subsequent tasks such as target detection and path planning. Autonomous driving perception models can use BEV features as input.

[0017] Figure 2 A schematic diagram of the topological structure of the reinforcement learning parallel processing accelerator provided by an embodiment of the present invention. Figure 2 As shown, the controller (such as a PCIE controller) can write the current view features (such as C-BEV), at least two groups of batch history view features (such as the first group of batch history view features H0-BEV, the second group of batch history view features H1-BEV) and instruction sequences into the corresponding memory (such as HBM) through DMA transmission. The controller can specifically access memories 0-7 and memories 10-11 through H2D (Host to Device), that is, transmit the host's data to these memories, and can also receive the data returned by memories 8-9 through D2H (Device to Host). The bit width of the data output by the memory can be 512 bits.

[0018] It should be noted that the calculation of the present invention is run in an offline mode. During the process, the parameter distribution component can parse the instructions of each computing layer in a set order, obtain the simulation parameters, calculation parameters and startup instructions of each computing layer, and transmit them to the corresponding computing components. After receiving the startup instruction, each computing component simultaneously reads the current BEV feature data and historical BEV feature data in the input cache, and reads the corresponding simulation parameters and calculation parameters to perform corresponding parallel computing processing.

[0019] Furthermore, in the specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by the embodiment of the present invention, the controller can also be used to write the state vector, action sequence, reward sequence, weight data, bias data and embedded data into the corresponding memory respectively. Correspondingly, the data loading control component can also be used to select the required state vector, action sequence and reward sequence from the memory according to the parameters of each layer and load them into the corresponding register group, and write the weight data, bias data and embedded data in the memory into the corresponding cache in sequence.

[0020] In the implementation, the state vector represents the state information, and the reward sequence stores the reward value obtained by the corresponding action sequence. Weight data is the strength of the connection between neurons, which determines the degree of influence of the input data on the output of the neuron. Bias data is a constant term added in the neuron calculation, which can adjust the activation function of the neuron at different positions and ranges, increasing the flexibility and expression ability of the network. Embedding data can map high-dimensional discrete data (such as words and categories in text) to low-dimensional continuous vector space. The present invention can write the state vector, action sequence, reward sequence, weight data, bias data and embedded data into the corresponding memory (such as HBM) through the controller, so that the data storage is regular and orderly, which is convenient for subsequent use, and is also conducive to the safe storage and management of data, reducing data confusion and errors. The data loading control component selects the required state vector from the memory according to the parameters of each layer, and the action sequence and reward sequence are loaded into the corresponding register group, which can quickly provide necessary data for the operation and shorten the data search time. The weight data, bias data and embedded data in the memory are written into the corresponding cache in sequence, so that the processor can quickly access these frequently used data, reduce waiting time and improve the overall computing speed.

[0021] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the computing component can be used to perform parallel processing according to the state vector, action sequence and reward sequence in the register group, the weight data, bias data and embedded data in the cache, and the read corresponding feature data, memory access parameters and calculation parameters to obtain intermediate layer result data and final result data; the intermediate layer result data is transferred between each group of caches and register groups.

[0022] In implementation, each computing component can perform parallel processing based on the state vector, action sequence and reward sequence in the register group, the weight data, bias data and embedded data in the cache, as well as the corresponding feature data, memory access parameters and calculation parameters read, to obtain the intermediate layer result data and the final result data. This can make full use of hardware resources, execute multiple computing tasks at the same time, greatly shorten the overall computing time, and is suitable for processing large-scale and complex computing tasks. The intermediate layer result data can be transferred between the cache and the register group, which is convenient for flexible calling in different computing steps, and can adapt to the complex computing processes of different algorithms and models. There is no need to re-read the original data every time, which improves the flexibility and convenience of data processing.

[0023] Furthermore, in a specific implementation, the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention may also include: a first in first out queue (FIFO), which is used to convert the bus bit width of the intermediate layer result data and the final result data, and write the converted data into a memory.

[0024] In implementation, Figure 2 As shown, a first-in-first-out queue is a rule or strategy for data storage and processing. Just like queuing, the data or elements that enter first will be processed or taken out first, and the data that enter later will be processed or taken out later, to ensure that the data is processed in the order in which it arrives, and to ensure the certainty and fairness of the processing order. The present invention uses a first-in-first-out queue to convert the intermediate layer result data and the final result data into bus bit width, and writes the converted data into a memory. In this way, the first-in-first-out queue can be used as a buffer area to temporarily store the intermediate layer and final result data, so that data can be smoothly transmitted between modules of different speeds to avoid data loss or data conflict. Figure 2 The data bit width written into the first-in-first-out queue can be 2048 bits.

[0025] Furthermore, in a specific implementation, the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention may also include: a cache cross selection component, which is connected to each computing component respectively, and may include a feature cache, a cache for storing weight data, bias data and embedded data respectively, and a register group for storing state vectors, action sequences and reward sequences respectively; the cache cross selection component may be used to select and schedule data sources among the feature cache, the cache for storing weight data, bias data and embedded data respectively, and the register group.

[0026] In implementation, Figure 2 As shown, the cache cross selection component may include a feature cache, a cache for storing weight data, bias data and embedded data respectively, and a register group for storing state vectors, action sequences and reward sequences respectively, such as caches 1-3 corresponding to C, caches 1-3 corresponding to H0, caches 1-3 corresponding to H1, register groups corresponding to C, H0 and H1, and separate register groups 2-H0, 2-H1, 3-H0, 3-H1, 4-H0, 4-H1, register groups 5-8, etc. The cache cross selection component can store different types of data in the feature cache, special cache and register group respectively, and perform unified selection and scheduling, so that the storage and search of data are more orderly. When the computing component needs specific data, the cache cross selection component can quickly locate and provide it, reducing data retrieval time and improving data acquisition efficiency.

[0027] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by an embodiment of the present invention, the computing component may include: a general matrix multiply (GEMM) computing component respectively connected to the instruction load distribution component and the cache cross selection component; the matrix multiplication computing component includes a first matrix multiplier component for performing matrix multiplication calculations on current view features, a second matrix multiplier component for performing matrix multiplication calculations on a first group of batch history view features, and a third matrix multiplier component for performing matrix multiplication calculations on a second group of batch history view features.

[0028] In implementation, Figure 2 As shown, the matrix multiplication calculation component in the calculation unit can include multiple matrix multiplier components. The first matrix multiplier component, the second matrix multiplier component and the third matrix multiplier component can respectively perform matrix multiplication calculations for the current view features, the first group of batch historical view features and the second group of batch historical view features at the same time, thereby realizing the parallelization of calculations, greatly improving the calculation speed, and being able to process a large amount of data in a shorter time, especially when processing complex image or video data and other tasks containing multi-view features, which can significantly reduce the calculation time. In addition, different matrix multiplier components are responsible for different tasks, avoiding the bottleneck problem that may occur when a single component handles all calculation tasks. Each component focuses on a specific type of feature calculation, which can more efficiently utilize hardware resources and improve overall computing efficiency.

[0029] Further, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the first matrix multiplier component, the second matrix multiplier component and the third matrix multiplier component may all include: a scaling (Scale) unit, a mask (Mask) unit, an attention mask (Atten_mask) unit, a bias (Bias) unit, a residual unit, an embedding (Embedding) unit and an activation (Active) unit. One or a combination of the following: the scaling unit is used to perform a scaling operation on the data under the control of a scaling enable signal; the mask unit is used to perform a mask calculation on the data under the control of a mask enable signal; the attention mask unit is used to perform an attention mask calculation on the data under the control of an attention mask enable signal; the bias unit is used to perform a bias calculation on the data under the control of a bias enable signal; the residual unit is used to perform a residual calculation on the data under the control of a residual enable signal; the embedding unit is used to perform an embedding calculation on the data under the control of an embedding enable signal; and the activation unit is used to perform an activation calculation on the data under the control of an activation enable signal.

[0030] In implementation, the scaling unit can flexibly scale the data according to the scaling enable signal, which can be used to adjust the dynamic range of the data, make the data more suitable for subsequent calculations and processing, and help improve the accuracy and stability of the calculation. The mask unit performs mask calculations on the data through the mask enable signal, which can selectively shield or retain certain features in the data, which helps to highlight key information. The attention mask unit performs attention mask calculations under the control of the attention mask enable signal, which can enable the model to pay more attention to the important parts of the data and automatically allocate computing resources, so as to better capture the long sequence dependencies and key features in the data, and enhance the model's understanding and processing capabilities of complex data. The bias unit uses the bias enable signal to perform bias calculations on the data, which can add a fixed offset to the data, help adjust the distribution of the data, and enable the model to better fit the data. The residual unit performs residual calculations through the residual enable signal, which can fuse the original data with the processed data, effectively retain the information in the original data, avoid information loss or gradient disappearance in multi-layer calculations, and help train deeper models and improve the model's expressiveness. The embedding unit performs embedding calculations under the control of the embedding enable signal, which can convert the data into a low-dimensional vector space that is more suitable for model processing, and represent discrete or high-dimensional data as continuous low-dimensional vectors, reducing the dimensionality disaster of the data, while extracting the potential features of the data to facilitate model learning and utilization. The activation unit performs activation calculations on the data under the control of the activation enable signal, introducing nonlinear factors into the model, enabling the model to learn the complex nonlinear relationships in the data, enhancing the model's expressive power, enabling the model to fit various complex functions, and improving the model's accuracy and generalization ability. The existence of these components enables the model to adjust parameters more effectively during the training process, and by performing various preprocessing and calculation operations on the data, the model is easier to converge, reducing training time and consumption of computing resources.

[0031] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by an embodiment of the present invention, the computing component may also include: a flattening and concatenation (Flatten_Concat) component respectively connected to the instruction load distribution component and the cache cross selection component; the flattening and concatenation component includes a first flattening and concatenation subcomponent for performing matrix multiplication calculations on current view features, a second flattening and concatenation subcomponent for performing matrix multiplication calculations on a first group of batch historical view features, and a third flattening and concatenation subcomponent for performing matrix multiplication calculations on a second group of batch historical view features.

[0032] In implementation, Figure 2As shown, the flattening and splicing component can contain multiple flattening and splicing subcomponents. Since different view features and batch history view features usually have different dimensions and shapes, the flattening and splicing subcomponent can flatten these features and convert multidimensional data into one-dimensional data for subsequent processing and calculation, so that data from different sources can be integrated and analyzed in a unified format, improving the efficiency and accuracy of data processing. Through the splicing operation, the current view features, the first set of batch history view features, and the second set of batch history view features are combined, which can integrate information from different time dimensions and perspectives, enrich the feature representation of the data, help the model capture more comprehensive and complex patterns and relationships, and improve the model's ability to understand and analyze data.

[0033] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by an embodiment of the present invention, the first flattening and splicing subcomponent, the second flattening and splicing subcomponent and the third flattening and splicing subcomponent all include a first input port, a second input port, a channel control input unit, a first data selection controller, a first multiplexer and a first output port; the first input port is connected to the channel control input unit, and is used to read the feature data of each channel grouping one by one through the channel control input unit; the first data selection controller is connected to the first multiplexer, and is used to rearrange the feature data of each channel grouping and transmit the rearranged feature data to the first multiplexer; the second input port is connected to the first multiplexer, and is used to read the state vector and transmit it to the first multiplexer; the first output port is used to output the feature data and state vector selected by the first multiplexer.

[0034] Figure 3 This is a schematic diagram of the structure of the flattened splicing assembly provided by an embodiment of the present invention. Figure 3As shown, the first input port can read the characteristic data of each channel group one by one through the channel control input part, which enables the subcomponent to flexibly process information from different channels, is suitable for processing data with multi-channel characteristics, can read and process the characteristics of each channel in a targeted manner according to specific needs, and improves the processing ability of complex data. The first data selection controller can rearrange the characteristic data of each channel group. This rearrangement operation can adjust the order and structure of the data according to different tasks and algorithm requirements, optimize the processing efficiency of the data in subsequent calculations, facilitate the model to better extract features, and enhance the configurability and adaptability of data processing. The second input port reads the state vector and transmits it to the first multiplexer, and merges it with the rearranged characteristic data. This fusion method combines different types of information (characteristic data and state vectors), which can provide richer and more comprehensive information for subsequent calculations and analysis, and help the model better understand the context and dynamic changes of the data. The first multiplexer can select a suitable combination of characteristic data and state vectors as needed, and output them through the first output port, which enables the subcomponent to flexibly select and output the most useful data according to different conditions and task requirements, improve the efficiency and pertinence of data processing, and avoid unnecessary data transmission and calculation. The above structures of the first flattening and splicing sub-component, the second flattening and splicing sub-component and the third flattening and splicing sub-component can improve the scalability and compatibility of the entire system.

[0035] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by an embodiment of the present invention, the computing component may also include: a stack parallel (Stack) component respectively connected to the instruction load distribution component and the cache cross selection component; the stack parallel component includes a third input port, a fourth input port, a fifth input port, a second data selection controller, and a second output port; the third input port is used to read the state vector corresponding to at least two groups of historical view features; the fourth input port is used to read the action sequence corresponding to at least two groups of historical view features; the fifth input port is used to read the reward sequence corresponding to at least two groups of historical view features; the second data selection controller is used to rearrange the data read from the third input port, the fourth input port, and the fifth input port, and output the rearranged data from the second output port.

[0036] Figure 4 A schematic diagram of the structure of stacked parallel components provided by an embodiment of the present invention. Figure 4As shown, the third input port (i.e., input port 1-H0, 1-H1), the fourth input port (i.e., input port 2-H0, 2-H1), and the fifth input port (i.e., input port 3-H0, 3-H1) are respectively responsible for reading the state vector, action sequence, and reward sequence corresponding to at least two sets of historical view features. This parallel reading mechanism can simultaneously acquire multiple types of data, greatly improving the data reading speed and processing efficiency. Compared with the sequential data reading method, it can collect various information required by the model in a shorter time, laying the foundation for subsequent rapid processing and analysis. Figure 5 A schematic diagram of data processing of stacked parallel components provided by an embodiment of the present invention, such as Figure 5 As shown, the second data selection controller can reorder the data from the three input ports, which can reduce the time overhead of data access, improve the flow efficiency of data in the entire system, and thus improve the overall data processing performance.

[0037] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the computing component may also include: a query key value split (QKV_Split) component respectively connected to the instruction load distribution component and the cache cross selection component; the query key value split component includes a sixth input port, a branch controller, a third output port, a fourth output port and a fifth output port; the sixth input port is used to read the hidden layer features to be query key value split; the branch controller is used to sequentially divide the feature data read by the sixth input port into query cache feature data, key cache feature data and value cache feature data; the third output port is used to output the query cache feature data; the fourth output port is used to output the key cache feature data; the fifth output port is used to output the value cache feature data.

[0038] Figure 6 The structure diagram of the query key value splitting component provided by the embodiment of the present invention is as follows. Figure 6 As shown, in the attention calculation, the query key-value splitting component inputs the hidden layer features to be queried key-value split. The branch controller can accurately divide the read hidden layer features into three groups of data, namely query cache feature data, key cache feature data and value cache feature data. This clear classification method enables different types of data to be accurately identified and separated, provides a basis for subsequent specific processing of different types of data, avoids the possibility of data confusion and erroneous processing, and improves the accuracy and reliability of data processing. Figure 7 A data processing diagram of a query key value splitting component provided in an embodiment of the present invention, such as Figure 7As shown, the third output port outputs query cache feature data; the fourth output port outputs key cache feature data; and the fifth output port outputs value cache feature data. Query cache feature data can be used for fast retrieval and matching operations, key cache feature data can be used for indexing and positioning, and value cache feature data can be used to store and obtain specific information content. This targeted processing can give full play to the role of each type of data and improve the processing efficiency and effect of data in various application scenarios.

[0039] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided by an embodiment of the present invention, the computing component may also include: a split-concatenation (Split_Concat) component respectively connected to the instruction loading and distribution component and the cache cross selection component; the split-concatenation component includes a seventh input port, an eighth input port, a third data selection controller, a second multiplexer, and a sixth output port; the seventh input port is connected to the second multiplexer, and is used to read the state sequence corresponding to the current view feature, and transmit the state sequence corresponding to the current view feature to the second multiplexer; the eighth input port is connected to the third data selection controller, and is used to group the input hidden layer features into a reward sequence, a state sequence and an action sequence through the third data selection controller, and transmit them to the second multiplexer; the sixth output port is used to output the data selected by the second multiplexer.

[0040] Figure 8 The schematic diagram of the structure of the splitting and splicing assembly provided by the embodiment of the present invention. Figure 8 As shown, the seventh input port is responsible for reading the state sequence corresponding to the current view feature, and the eighth input port can group the input hidden layer features into reward sequences, state sequences, and action sequences. This design enables the component to process data from different sources and types at the same time, integrate multi-source data, and provide richer and more comprehensive information for subsequent analysis and calculation. The third data selection controller can group the hidden layer features into reward sequences, state sequences, and action sequences as needed, and the second multiplexer can select from these data according to specific needs. This flexible data grouping and selection mechanism allows the component to process and output data in a targeted manner according to different tasks and scenarios, thereby improving the efficiency and accuracy of data processing. Fig. 9 A schematic diagram of data processing of the splitting and splicing components provided in an embodiment of the present invention, such as Fig. 9As shown in the figure, the splitting and splicing component performs operations such as splitting, grouping, and selecting data in a centralized manner, and finally outputs the selected data through an output port. This enables these important sequences to be accurately grouped and flexibly combined, providing strong support for the training of reinforcement learning models. By reasonably selecting and outputting these sequence data, the model can better learn the relationship between environmental states, actions, and rewards, thereby optimizing strategies, improving the accuracy and efficiency of decision-making, and reducing the number and complexity of data transmissions between different components, reducing the delay in data transmission, and improving the overall efficiency of data processing.

[0041] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the computing component may also include: a normalized index (Softmax) component respectively connected to the instruction load distribution component and the cache cross selection component; and a normalized index component for performing normalized index operations on the read data.

[0042] In implementation, Figure 2 As shown in the figure, the normalized index component can uniformly map the data to a specific range or scale, making different data comparable, and the normalized data can make the model converge more easily, which helps to improve the generalization ability and accuracy of the model.

[0043] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the computing component may also include: a layer normalization (LN) component respectively connected to the instruction load distribution component and the cache cross selection component; a layer normalization component, used to receive weight data and bias data, and perform normalization processing in combination with the read data.

[0044] In implementation, Figure 2 As shown in the figure, the layer normalization component can make the model more adaptable to different input data by normalizing the data in each layer, reduce the model's dependence on specific data distribution, and thus improve the generalization ability of the model. The weight data (LN_Weight) and bias data (LN_Bias) in the layer normalization component are two important parameters in the layer normalization operation, which are used to scale and translate the input data, respectively, improving the adaptability and processing ability of the computing component to different data, and improving the reliability and stability of the model in various practical application scenarios.

[0045] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, two groups of memories are used as ping-pong caches to interactively store calculation results and provide feature inputs for the next layer.

[0046] In implementation, the ping-pong cache mechanism allows data storage and reading to be performed alternately between two groups of memories. When one group of memories is providing data to the computing component as the next layer of feature input, the other group can simultaneously store the calculation results, thereby achieving seamless data connection, avoiding waiting time during data transmission and processing, improving overall computing efficiency, and ensuring the continuity of computing.

[0047] Furthermore, in a specific implementation, the above-mentioned reinforcement learning parallel processing accelerator provided in the embodiment of the present invention may also include: a first selector connected to the computing component, the cache cross selection module and the first-in-first-out queue.

[0048] In implementation, the first selector can flexibly select to transfer data from the computing component to the cache cross selection module or the first-in-first-out queue, ensuring that data flows along appropriate paths between different components. In other words, the existence of the first selector helps to achieve parallel transmission and processing of data between the computing component, the cache cross selection module and the first-in-first-out queue, thereby speeding up the execution speed of the overall computing task.

[0049] Furthermore, in a specific implementation, the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention may also include: a second selector, connected to an instruction load distribution component, a matrix multiplication calculation component, a cache cross selection component, and a parameter cache for storing weight data, bias data, and embedded data.

[0050] In implementation, the second selector can accurately select and transmit instructions to the corresponding components, ensuring that the matrix multiplication calculation component can obtain the required instructions in a timely manner and perform operations in sequence, thereby improving the accuracy and efficiency of instruction execution. Figure 2 As shown, the weight cache, bias cache, and embedding cache receive data from the controller (such as data with a bit width of 512 bits), and after cache processing, output weight data, bias data, and embedding data (such as data with a bit width of 2048 bits) are written into the parameter cache to provide data support for the matrix multiplication calculation component. The cache cross selection component can provide attention key value (KV) matrix data, etc. for the matrix multiplication calculation component, and the attention KV matrix data and weight data can be input into the matrix multiplication calculation component after being processed by the second selector.

[0051] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the controller can also be used to read the data obtained after the first-in-first-out queue conversion from the memory for data comparison.

[0052] During implementation, the controller reads the converted data from the first-in-first-out queue for comparison. The controller can verify whether the data remains consistent during the conversion process, ensure that there are no errors or losses in the storage, transmission and processing of the data, ensure the accuracy and integrity of the data in the system, and provide a reliable data basis for subsequent operations and decisions.

[0053] Furthermore, in a specific implementation, the above-mentioned reinforcement learning parallel processing accelerator provided in the embodiment of the present invention may also include: an instruction control component connected to the controller and the instruction loading and distributing component, and used to issue a start command to the instruction loading and distributing component.

[0054] In implementation, Figure 2 As shown, the controller can write the relevant instruction data to the instruction control component through register control, and the bit width of the register control channel can be 32 bits. The relevant instruction data here may include debugging instructions, reset instructions, startup instructions, etc., which are used to control the operating state of the entire system. The present invention can send debugging instructions, reset instructions, start calculation instructions, etc. to the instruction loading and distribution component to start executing the calculation task of the entire network; the instruction loading and distribution component can return a completion signal, the number of running layers and the calculation time to the instruction control component.

[0055] Furthermore, in a specific implementation, in the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, the instruction control component is also used to notify the host computer in an interrupt manner after all layers of calculation are completed; or, it is also used to feedback the operation completion status to the host computer when receiving a query instruction from the host computer after all layers of calculation are completed.

[0056] In implementation, the instruction control component has the function of notifying the host computer by interruption or feeding back the operation completion status according to the host computer query instruction after all layers of calculation are completed. Notifying the host computer by interruption allows the host computer to get the message as soon as the calculation is completed, without the need for the host computer to continuously poll the bottom-level calculation status, which greatly saves the host computer's time and resources. The instruction control component receives the host computer query instruction and feeds back the operation completion status, providing the host computer with a flexible way to obtain information. The present invention uses interruption notification or query feedback, which can let the host computer clearly know the status of the calculation task, avoid misoperation or repeated operation caused by unclear information, and enhance system stability and reliability.

[0057] In practical applications, the present invention analyzes the reinforcement learning network structure, as shown in Table 1-5, which is the calculation order of the network, where C is the number of blocks in each row of the block matrix, R is the number of rows in the block matrix, H is the number of attention heads, X is the feature matrix, W is the weight matrix, E is the residual matrix, b is the bias, D is the embedding, S is the scaling, M is the mask, and A is the attention mask. Conv means convolution, i in RAMi represents input, RAM represents memory, ReLU represents rectified linear, Softplus represents an activation function, RegGrp is a register group, and Att represents the attention score.

[0058] Table 1 Feature cache location example 1

[0059]

[0060] Table 2 Feature cache location example 2

[0061]

[0062] Table 3 Feature cache location example 3

[0063]

[0064] Table 4 Feature cache location example 4

[0065]

[0066] Table 5 Feature cache location example 5

[0067]

[0068] Table 1-5 illustrates the order of switching the feature cache storage location in each step of the network. According to the data cache requirements, the K matrix and the V matrix are stored in the cache at the same time. -H0.

[0069] Bias, weights, and embeddings use independent caches. The feature cache includes the input caches of C-BEV, H0-BEV, and H1-BEV. -C-- -C. -H0-- -H0 and -H1-- -H1 feature calculation flow cache and RegGrp1, RegGrp2-H0—RegGrp4-H0, RegGrp2-H1—RegGrp4-H1, RegGrp5—RegGrp8 dedicated register groups, where register groups 1 and 2 are input state vectors, register groups 3 and 4 are input action sequences and reward sequences, register group 5 is the state feature calculated by C-BEV, registers 6 and 7 are the distribution mean and distribution variance of the calculated policy network, and register group 8 is the value scalar of the calculated value network. In addition, in Table 1, when calculating the split operation numbered 19, since three groups of matrices need to be generated, in order to reduce the total number of caches used and since the K matrix and the V matrix will not participate in the calculation at the same time, the K matrix and the V matrix can be placed on the same group of caches and stored in two areas. This avoids the space waste caused by allocating independent caches for the K matrix and the V matrix, and significantly reduces the overall cache usage. In the case of limited memory resources, this optimization can free up more cache space for other data or operations, improve the memory usage efficiency of the entire system, reduce the overall memory access scheduling overhead of the system, and make the system structure more compact. Compared with allocating a large cache space to each matrix, this method can make more full use of every byte of the cache, making the cache space more effectively utilized, reducing the problem of idle space caused by fixed allocation, and allowing subsequent data access to hit the cache faster, reducing the number of cache misses and improving computing performance.

[0070] It should be noted that the present invention adopts an offline scheduling method. After the host computer preloads the current view, at least two groups of batch historical views, and the weight data into the memory, it sends a start calculation command to the instruction control component. After the instruction control component sends a start instruction to the instruction loading and distribution component, the instruction loading and distribution component controls each calculation component to complete the corresponding calculation process in sequence. Each component completes data loading, calculation and result writing back according to the calculation parameters.

[0071] Fig.10 This is one of the computational flow diagrams of the reinforcement learning parallel processing accelerator provided in an embodiment of the present invention. Fig.11 The second schematic diagram of the calculation process of the reinforcement learning parallel processing accelerator provided by the embodiment of the present invention. The calculation process may include: Fig.10 As shown, convolution feature extraction is performed on the current image and at least two groups of batch historical images at the same time. It is assumed that convolution feature extraction is performed on the C-BEV image, the H0-BEV image and the H1-BEV image to obtain C-BEV features, H0-BEV features and H1-BEV features; Fig.10N in it represents the number of cycles. According to the C-BEV feature, H0-BEV feature and H1-BEV feature, the C-state vector, H0-state vector and H1-state vector are obtained. Then, the C-state vector, H0-state vector and H1-state vector are subjected to matrix multiplication calculation to obtain the C-state feature, H0-state feature and H1-state feature. The C-state vector and the C-BEV feature are flattened and concatenated to obtain the C-concatenation feature, which is subjected to matrix multiplication calculation to obtain the C-state sequence. At the same time, the H0 state vector and the H0-BEV feature are flattened and concatenated to obtain the H0-concatenation feature, which is subjected to matrix multiplication calculation to obtain the H0-state sequence; the H1-state vector and the H1-BEV feature are flattened and concatenated to obtain the H1-concatenation feature, which is subjected to matrix multiplication calculation to obtain the H1-state sequence. Next, as Fig.11 As shown, embedding processing is performed on the H0-state sequence and the H1-state sequence respectively to obtain H0-state feature embedding and H1-state feature embedding, and then the H0-action sequence and H1-action sequence are obtained. The H0-action sequence and the H1-action sequence are embedded to obtain H0-action feature embedding and H1-action feature embedding. According to the action feature embedding, a reward sequence is obtained. The reward sequence is embedded to obtain H0-embedded reward feature embedding and H1-embedded reward feature embedding. Then, the H0-state feature embedding, H0-action feature embedding, H0-reward feature embedding, H1-state feature embedding, H1-action feature embedding, and H1-reward feature embedding are stacked. After layer normalization and one-dimensional convolution operation are performed on the data after stacking calculation, the query key value is split to obtain the corresponding query, key, and value, and the attention score is obtained under the attention mask, and then the calculation result is obtained after operations such as normalized exponential function. If the calculation of all data blocks is completed, the layer normalization processing will continue, and the C-state sequence will be split and spliced ​​to obtain the state characteristics. Then a series of calculation steps will be carried out, including multiple fully connected layers (FC), rectified linear units (ReLU), normalized exponential functions, activation functions (Softplus) and other calculation links to obtain the distribution mean and distribution variance of the strategy network, and finally obtain the value scalar of the value network to complete the calculation process.

[0072] After the calculation process is completed, the host computer can be notified through an interrupt or the host computer can query the completion status through the instruction control module. After reading the results, the next set of feature data is loaded and a start calculation command is issued, and this cycle is repeated until all feature data calculations are completed. The calculation process is controlled by the instruction parameter distribution module according to the parameters obtained by decoding the pre-loaded instructions, without the participation of the host computer, which improves processing efficiency and avoids interaction delays.

[0073] In the above embodiments, the reinforcement learning parallel processing accelerator is described in detail. Based on the same inventive concept, the embodiments of the present invention also provide an acceleration method of the reinforcement learning parallel processing accelerator and a corresponding embodiment of the electronic device.

[0074] Fig.12 Flow chart of the acceleration method of the reinforcement learning parallel processing accelerator provided by the embodiment of the invention. The acceleration method of the reinforcement learning parallel processing accelerator provided by this embodiment is as follows: Fig.12 As shown, the following steps are included: S1201. The controller writes the current view features, at least two groups of batch history view features and the instruction sequence into corresponding memories respectively.

[0075] S1202, the instruction loading and distributing component reads the instruction sequence in the memory, decodes the read instruction sequence, and distributes the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component.

[0076] S1203: The data loading control component selects required feature data from the memory according to the parameters of each layer and loads it into the corresponding feature cache.

[0077] S1204. After receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two sets of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, performs parallel processing, obtains the intermediate layer result data and the final result data and returns them to the controller.

[0078] In the acceleration method of the above-mentioned reinforcement learning parallel processing accelerator provided by the embodiment of the present invention, the controller writes the current view feature, at least two groups of batch historical view features and the instruction sequence into the corresponding memory respectively to prepare for subsequent processing; the instruction loading and distribution component can quickly read and decode the instruction sequence and distribute parameters and start calculation instructions to the calculation component; the data loading control component selects the required feature data from the memory according to the parameters of each calculation layer and loads it to the corresponding feature cache; after receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two groups of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, and performs parallel processing, so that multiple view feature data are read at the same time and combined with parameters for parallel processing, making full use of hardware resources, greatly improving data processing efficiency, reducing processing time, accelerating the operation of reinforcement learning algorithms, and being suitable for rapid decision-making in complex scenarios. Through the mutual cooperation of the above-mentioned components, the characteristics of the reinforcement learning model can be deeply analyzed, and the parameters and processing procedures can be flexibly adjusted according to different reinforcement learning tasks and data characteristics, adapting to a variety of application scenarios, so that data reading, instruction processing and calculation execution are carried out in an orderly manner, improving the processing efficiency of the reinforcement learning model, while reducing the resource utilization rate of the accelerator, achieving a better acceleration ratio, and fast response speed.

[0079] Since the embodiments of the acceleration method part correspond to the embodiments of the reinforcement learning parallel processing accelerator part, the embodiments of the acceleration method part refer to the description of the embodiments of the reinforcement learning parallel processing accelerator part, which will not be repeated here. And it has the same beneficial effects as the reinforcement learning parallel processing accelerator mentioned above.

[0080] Further, in the specific implementation, in the acceleration method of the above-mentioned reinforcement learning parallel processing accelerator provided by the embodiment of the present invention, while executing step S1201, it can also include: the controller writes the state vector, action sequence, reward sequence, weight data, bias data and embedded data into the corresponding memory respectively. When executing step S1203, the data loading control component selects the required state vector, action sequence and reward sequence from the memory according to the parameters of each layer and loads them into the corresponding register group, and writes the weight data, bias data and embedded data in the memory into the corresponding cache in sequence. When executing step S1204, the calculation component performs parallel processing according to the state vector, action sequence and reward sequence in the register group, the weight data, bias data and embedded data in the cache, and the corresponding feature data, memory access parameters and calculation parameters read, to obtain the intermediate layer result data and the final result data; the intermediate layer result data is transferred between each group of caches and register groups.

[0081] Furthermore, in a specific implementation, in the acceleration method of the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention, after executing step S1204, it may also include: using a first-in-first-out queue to convert the bus bit width of the intermediate layer result data and the final result data, and writing the converted data into a memory.

[0082] Furthermore, in the specific implementation, in the acceleration method of the above-mentioned reinforcement learning parallel processing accelerator provided by the embodiment of the present invention, it can also include: a cache cross selection component selects and schedules data sources among feature caches, caches, and register groups. The cache cross selection component is connected to each computing component respectively, and can include a feature cache, a cache for storing weight data, bias data, and embedded data respectively, and a register group for storing state vectors, action sequences, and reward sequences respectively.

[0083] Furthermore, in a specific implementation, the acceleration method of the reinforcement learning parallel processing accelerator provided in the embodiment of the present invention may also include: two groups of memories serving as ping-pong caches to interactively store calculation results and provide feature inputs for the next layer.

[0084] Furthermore, in a specific implementation, the acceleration method of the above-mentioned reinforcement learning parallel processing accelerator provided in an embodiment of the present invention may also include: the controller reads the data obtained after the first-in-first-out queue conversion from the memory for data comparison.

[0085] For more specific working processes of the above steps, please refer to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0086] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, including the above-mentioned reinforcement learning parallel processing accelerator. Since the principle of solving the problem by the electronic device is similar to that of the above-mentioned reinforcement learning parallel processing accelerator, the implementation of the electronic device can refer to the implementation of the reinforcement learning parallel processing accelerator, and the repeated parts will not be repeated.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] Finally, it should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "include", "comprise" and "have" and any other variations thereof in the present invention are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0089] For the above embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the order of the actions described, because according to the present invention, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily required by the present invention.

[0090] The above is a detailed introduction to the reinforcement learning parallel processing accelerator, acceleration method and electronic device provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A reinforcement learning parallel processing accelerator, characterized in that: include: A controller, used to write the current view features, at least two groups of batch history view features and the instruction sequence into corresponding memories respectively; An instruction loading and distributing component is connected to the memory, and is used to read the instruction sequence in the memory, decode the read instruction sequence, and distribute the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component; A data loading control component, connected to the memory, the instruction loading distribution component and each of the computing components, respectively, for selecting required feature data from the memory according to the parameters of each layer and loading it into a corresponding feature cache; The computing component is connected to the instruction loading and distributing component, and is used to simultaneously read the current view feature data and at least two groups of batch historical view feature data in the feature cache after receiving the start computing instruction, and perform parallel processing in combination with corresponding memory access parameters and computing parameters to obtain intermediate layer result data and final result data and return them to the controller.

2. The reinforcement learning parallel processing accelerator according to claim 1, characterized in that: The controller is further used to write the state vector, action sequence, reward sequence, weight data, bias data and embedded data into the corresponding memory respectively; The data loading control component is also used to select the required state vector from the memory according to the parameters of each layer, load the action sequence and reward sequence into the corresponding register group, and write the weight data, bias data and embedded data in the memory into the corresponding cache in sequence.

3. The reinforcement learning parallel processing accelerator according to claim 2, characterized in that: The computing component is used to perform parallel processing according to the state vector, action sequence and reward sequence in the register group, the weight data, bias data and embedded data in the cache, and the read corresponding feature data, memory access parameters and calculation parameters to obtain intermediate layer result data and final result data; The intermediate layer result data is transferred between each set of cache and register groups.

4. The reinforcement learning parallel processing accelerator according to claim 1, characterized in that: Also includes: The first-in-first-out queue is used to convert the bus bit width of the intermediate layer result data and the final result data, and write the data obtained after the conversion into the memory.

5. The reinforcement learning parallel processing accelerator according to claim 1, characterized in that: Also includes: A cache cross-selection component, connected to each of the computing components, includes the feature cache, a cache for storing weight data, bias data and embedding data, and a register group for storing state vectors, action sequences and reward sequences; The cache cross selection component is used to select and schedule data sources among the feature cache, the cache for storing weight data, bias data and embedded data respectively, and the register group.

6. The reinforcement learning parallel processing accelerator according to claim 5, characterized in that: The calculation unit includes: a matrix multiplication calculation component connected to the instruction load distribution unit and the cache cross selection unit respectively; The matrix multiplication calculation component includes a first matrix multiplier component for performing matrix multiplication calculation on current view features, a second matrix multiplier component for performing matrix multiplication calculation on a first set of batch history view features, and a third matrix multiplier component for performing matrix multiplication calculation on a second set of batch history view features.

7. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The first matrix multiplier component, the second matrix multiplier component and the third matrix multiplier component each include: one or a combination of a scaling part, a mask part, an attention mask part, a bias part, a residual part, an embedding part and an activation part; The scaling unit is used to perform a scaling operation on the data under the control of a scaling enable signal; The mask part is used to perform mask calculation on the data under the control of the mask enable signal; The attention mask unit is used to perform attention mask calculation on the data under the control of the attention mask enable signal; The biasing unit is used to perform bias calculation on the data under the control of the bias enabling signal; The residual part is used to perform residual calculation on the data under the control of the residual enable signal; The embedding unit is used to perform embedding calculation on the data under the control of the embedding enable signal; The activation unit is used to perform activation calculation on the data under the control of the activation enable signal.

8. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The computing component further includes: a flattening and splicing component connected to the instruction load distribution component and the cache cross selection component respectively; The flattening and splicing component includes a first flattening and splicing subcomponent for performing matrix multiplication calculation on current view features, a second flattening and splicing subcomponent for performing matrix multiplication calculation on a first group of batch historical view features, and a third flattening and splicing subcomponent for performing matrix multiplication calculation on a second group of batch historical view features.

9. The reinforcement learning parallel processing accelerator according to claim 8, characterized in that: The first flattening and splicing subassembly, the second flattening and splicing subassembly and the third flattening and splicing subassembly each include a first input port, a second input port, a channel control input portion, a first data selection controller, a first multiplexer and a first output port; The first input port is connected to the channel control input unit and is used to read the characteristic data of each channel group one by one through the channel control input unit; The first data selection controller is connected to the first multiplexer and is used to rearrange the characteristic data of each channel group and transmit the rearranged characteristic data to the first multiplexer; The second input port is connected to the first multiplexer and is used to read the state vector and transmit it to the first multiplexer; The first output port is used to output the feature data and state vector selected by the first multiplexer.

10. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The computing component further includes: a stack parallel component connected to the instruction load distribution component and the cache cross selection component respectively; The stacked parallel component includes a third input port, a fourth input port, a fifth input port, a second data selection controller, and a second output port; The third input port is used to read the state vectors corresponding to at least two groups of historical view features; The fourth input port is used to read the action sequences corresponding to at least two groups of historical view features; The fifth input port is used to read reward sequences corresponding to at least two groups of historical view features; The second data selection controller is used to rearrange the data read from the third input port, the fourth input port, and the fifth input port, and output the rearranged data from the second output port.

11. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The computing component further includes: a query key value splitting component connected to the instruction load distribution component and the cache cross selection component respectively; The query key value splitting component includes a sixth input port, a branching controller, a third output port, a fourth output port and a fifth output port; The sixth input port is used to read the hidden layer features to be split into query keys; The branch controller is used to sequentially divide the feature data read by the sixth input port into query cache feature data, key cache feature data and value cache feature data; The third output port is used to output the query cache feature data; The fourth output port is used to output the key cache feature data; The fifth output port is used to output the value cache characteristic data.

12. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The computing component further includes: a splitting and splicing component connected to the instruction loading and distributing component and the cache cross selection component respectively; The splitting and splicing component includes a seventh input port, an eighth input port, a third data selection controller, a second multiplexer, and a sixth output port; The seventh input port is connected to the second multiplexer, and is used to read the state sequence corresponding to the current view feature, and transmit the state sequence corresponding to the current view feature to the second multiplexer; The eighth input port is connected to the third data selection controller, and is used to group the input hidden layer features into reward sequences, state sequences and action sequences through the third data selection controller, and transmit them to the second multiplexer; The sixth output port is used to output the data selected by the second multiplexer.

13. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The calculation component further includes: a normalized index component connected to the instruction load distribution component and the cache cross selection component respectively; The normalized index component is used to perform a normalized index operation on the read data.

14. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: The computing component further includes: a layer normalization component connected to the instruction load distribution component and the cache cross selection component respectively; The layer normalization component is used to receive weight data and bias data, and perform normalization processing in combination with the read data.

15. The reinforcement learning parallel processing accelerator according to claim 1, characterized in that: The two groups of memories are used as ping-pong buffers to interactively store calculation results and provide feature input for the next layer.

16. The reinforcement learning parallel processing accelerator according to claim 4, characterized in that: Also includes: The first selector is connected to the computing component, the cache cross selection module and the first-in-first-out queue.

17. The reinforcement learning parallel processing accelerator according to claim 6, characterized in that: Also includes: The second selector is connected to the instruction load distribution component, the matrix multiplication calculation component, the cache cross selection component, and a parameter cache for storing weight data, bias data, and embedding data.

18. The reinforcement learning parallel processing accelerator according to claim 4, characterized in that: The controller is also used to read the data obtained after the FIFO queue conversion from the memory for data comparison.

19. The reinforcement learning parallel processing accelerator according to claim 1, characterized in that: Also includes: An instruction control component is connected to the controller and the instruction loading and distributing component, and is used to send a start command to the instruction loading and distributing component.

20. The reinforcement learning parallel processing accelerator according to claim 19, characterized in that: The instruction control component is also used to notify the host computer in an interrupt manner after all layers of calculation are completed; or, after all layers of calculation are completed, when receiving a query instruction from the host computer, the operation completion status is fed back to the host computer.

21. A method for accelerating a reinforcement learning parallel processing accelerator, characterized in that: The reinforcement learning parallel processing accelerator is the reinforcement learning parallel processing accelerator according to any one of claims 1 to 20, and the acceleration method comprises: The controller writes the current view features, at least two sets of batch history view features and the instruction sequence into corresponding memories respectively; The instruction loading and distributing component reads the instruction sequence in the memory, decodes the read instruction sequence, and distributes the obtained memory access parameters, calculation parameters and start calculation instructions to the calculation component; The data loading control component selects required feature data from the memory according to the parameters of each layer and loads it into the corresponding feature cache; After receiving the start calculation instruction, the calculation component simultaneously reads the current view feature data and at least two groups of batch historical view feature data in the feature cache, combines the corresponding memory access parameters and calculation parameters, performs parallel processing, obtains the intermediate layer result data and the final result data and returns them to the controller.

22. An electronic device, characterized in that: Comprising the reinforcement learning parallel processing accelerator as described in any one of claims 1 to 20.

Images