Data processing device and method and vehicle

Abstract

The embodiment of the invention provides a data processing device and method and a vehicle. The device is a storage and calculation integrated device for calculating and storing data by adopting a memory, and is used for sequentially calculating feature maps of a plurality of pieces of to-be-processed data to obtain respective corresponding query features; the device comprises a pre-calculation module used for carrying out weight transformation on a feature map of current to-be-processed data to obtain and cache a pre-calculation result; the sampling module is used for sampling a target query feature to obtain a sampling result, and the target query feature comprises a query feature which is calculated by the query generation module and corresponds to any to-be-processed data before the current to-be-processed data; the attention weight module is used for carrying out attention weight calculation on the target query features to obtain an attention result; and the query generation module is used for performing calculation according to the attention result, the sampling result and the pre-calculation result to obtain and cache query features corresponding to the to-be-processed data. The device can improve the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing device, method and vehicle. Background Art

[0002] With the continuous development of artificial intelligence technology, various artificial intelligence models are applied in many practical scenarios, among which artificial intelligence models based on attention mechanism are widely used. For example, in the scene of vehicle autonomous driving, artificial intelligence models based on attention mechanism such as Bird's Eye View (BEV) model (Former) are used to process the data collected by the vehicle during the autonomous driving process to obtain data processing results, based on which environmental perception, path planning and motion control decision-making can be achieved.

[0003] In practical applications, the functional realization of artificial intelligence models is based on hardware devices such as data processing devices. For example, whether in the training stage or the application stage, the operation of artificial intelligence models usually stores data through storage devices and processes the data through processors. Therefore, the performance of hardware devices is crucial to the performance of artificial intelligence models.

[0004] When the existing data processing device runs an artificial intelligence model based on the attention mechanism, the speed of data calculation and storage is slow, and the amount of data processed per unit time is small. Therefore, when the existing data processing device runs an artificial intelligence model based on the attention mechanism, the data processing efficiency is low. Summary of the invention

[0005] The embodiments of the present application provide a data processing device, method and vehicle to achieve the effect of improving the data processing efficiency when running an artificial intelligence model based on an attention mechanism.

[0006] In a first aspect, an embodiment of the present application provides a data processing device, which is a storage-computation integrated device that uses a memory to calculate and store data, and is used to sequentially calculate feature graphs of multiple data to be processed to obtain respective corresponding query features; the data processing device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module;

[0007] The pre-calculation module is used to perform weight transformation on the feature map of the current data to be processed, obtain a pre-calculation result, and cache the pre-calculation result;

[0008] The sampling module is used to sample the target query feature to obtain a sampling result, wherein the target query feature includes a query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module;

[0009] The attention weight module is used to calculate the attention weight of the target query feature to obtain an attention result;

[0010] The query generation module is used to perform calculations based on the attention results, the sampling results and the pre-calculation results to obtain query features corresponding to the data to be processed, and cache the query features.

[0011] In a possible implementation, the query generation module includes an adaptive selector, the adaptive selector includes at least two groups of weight comparison units consisting of a comparator and a first-in-first-out queue, and the query generation module is specifically used to:

[0012] Comparing the attention weight in the attention result with a preset attention threshold through a comparator in the weight comparison unit, and caching the attention weight greater than the attention threshold in a corresponding first-in-first-out queue;

[0013] Obtaining a filtering result corresponding to the attention result according to the attention weight output by the first-in-first-out queue;

[0014] Calculation is performed according to the filtering result, the sampling result and the pre-calculation result to obtain query features corresponding to the data to be processed, and the query features are cached.

[0015] In a possible implementation, the query generation module is specifically used to:

[0016] Perform multiplication and accumulation operations on the filtering result, the sampling result and the pre-calculation result to obtain an accumulation result;

[0017] The query feature corresponding to the data to be processed is obtained according to the result of comparison performed by a comparison circuit according to the accumulated result, and the query feature is cached.

[0018] In a possible implementation, the pre-calculation module includes a pre-calculation engine and a pre-calculation buffer, and the pre-calculation module is specifically used to:

[0019] The feature map of the data to be processed is multiplied according to the pre-calculation weights pre-stored in the pre-calculation engine to obtain the pre-calculation result, and the pre-calculation result is cached in the pre-calculation buffer.

[0020] In a possible implementation, the pre-calculation engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing units include a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array, the storage array pre-stores a first weight value of the pre-calculated weight; the pre-calculation module is specifically used to:

[0021] Acquire first characteristic data of the characteristic graph of the data to be processed through the write driver;

[0022] Performing a multiplication operation on the first feature data and the first weight value by the multiplication logic circuit to obtain a first product;

[0023] Performing an addition operation on the first product by the addition tree circuit to obtain a first sum value;

[0024] The first sum is accumulated and calculated by the accumulating unit to obtain a second sum, and the second sum is transmitted to the pre-calculation buffer for caching.

[0025] In one possible implementation, the sampling module includes a sampling engine and a sampling buffer, wherein the sampling engine is used to calculate a corresponding offset and bilinear interpolation parameters according to the target query feature, and calculate a sampling result corresponding to a reference point after the offset based on the offset and the bilinear interpolation parameters, and transmit the sampling result to the sampling buffer for caching.

[0026] In a possible implementation, the attention weight module includes an attention engine and an attention buffer, the attention engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a function activation unit and a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing unit includes a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array consisting of a plurality of storage elements, the storage array pre-stores a second weight value of the attention weight; the attention weight module is specifically used for:

[0027] Acquire second feature data in the target query feature through the write driver;

[0028] performing a multiplication operation on the second feature data and the second weight value by the multiplication logic circuit to obtain a second product;

[0029] Performing an addition operation on the second product by the addition tree circuit to obtain a third sum value;

[0030] Accumulating the third sum value by the accumulating unit to obtain a fourth sum value;

[0031] The fourth sum value is input into the function activation unit to obtain an activated fourth sum value, and the activated fourth sum value is transmitted to the attention buffer for caching.

[0032] In a possible implementation manner, the data processing device further includes a dynamic execution unit, which is connected to the query generation module; the dynamic execution unit is used to:

[0033] Acquire the query features corresponding to the current data to be processed obtained by the query generation module;

[0034] The similarity calculation circuit in the dynamic execution unit performs similarity calculation on the query feature corresponding to the current data to be processed and the previously obtained query feature to obtain a similarity value;

[0035] The similarity value is compared with a preset similarity threshold by a comparison circuit in the dynamic execution unit to obtain a similarity comparison result, and the similarity comparison result is used to indicate whether the query feature corresponding to the current data to be processed is determined as the target query feature corresponding to data processing for the next data to be processed.

[0036] In a possible implementation, the data processing device further includes a fully connected layer calculation module, which is connected to the query generation module and is used to perform linear transformation calculation on the query features obtained by the query generation module to obtain linearly transformed query features.

[0037] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a storage-computing integrated device that uses a memory to calculate and store data, and the storage-computing integrated device is used to sequentially calculate feature graphs of multiple data to be processed to obtain respective corresponding query features, and the storage-computing integrated device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module; the method includes:

[0038] Based on the pre-calculation module, weight transformation is performed on the feature map of the current data to be processed to obtain a pre-calculation result, and the pre-calculation result is cached;

[0039] Sampling the target query feature based on the sampling module to obtain a sampling result, wherein the target query feature includes a query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module;

[0040] Calculate the attention weight of the target query feature based on the attention weight module to obtain an attention result;

[0041] Based on the query generation module, calculation is performed according to the attention result, the sampling result and the pre-calculation result to obtain query features corresponding to the data to be processed, and the query features are cached.

[0042] In a third aspect, an embodiment of the present application provides a vehicle, comprising the data processing device as described in the first aspect above, wherein the vehicle processes the data to be processed of the bird's-eye view model through the data processing device to obtain query features of the bird's-eye view model.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement various possible implementation methods of the second aspect as above.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements various possible implementation methods of the second aspect above.

[0045] The data processing device, method and vehicle provided in the embodiment of the present application are a storage-computation integrated device that uses a memory to calculate and store data, and is used to calculate the feature graphs of multiple data to be processed in sequence to obtain the corresponding query features. The data processing device includes a pre-calculation module, a sampling module, an attention weight module and a query generation module. Each module is a hardware module in the memory and is mapped to each operator of the attention mechanism respectively. The input data is calculated and stored in each module, and the data calculation process can be performed in the memory, and the operation based on the attention mechanism is realized. Since the calculation process occurs at a position close to the storage array, the dependence on the intermediate links of data access such as the system bus can be eliminated. By directly processing data in the memory, the transmission time of data between the computing component and the memory can be greatly reduced, effectively overcoming the limitations brought by the von Neumann architecture, reducing the pipeline overhead and reducing the data access overhead, and increasing the data processing volume per unit time. Therefore, through the data processing device, the effect of improving the data processing efficiency when running an artificial intelligence model based on the attention mechanism can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of the von Neumann architecture provided for this application;

[0048] Figure 2A schematic diagram of the bandwidth of an on-chip network based on the von Neumann architecture provided in this application;

[0049] Figure 3 A schematic diagram of data movement energy consumption based on the von Neumann architecture provided for this application;

[0050] Figure 4 Schematic diagram of the overall architecture of BEVFormer provided for this application;

[0051] Figure 5 The structure of the data processing device provided in this application is shown in FIG. Figure 1 ;

[0052] Figure 6 The structure of the data processing device provided in this application is shown in FIG. Figure 2 ;

[0053] Figure 7 A schematic diagram of the structure of the query generation engine and dynamic execution unit provided in this application;

[0054] Figure 8 A schematic diagram of the module and operator mapping provided for this application;

[0055] Fig. 9 A schematic diagram showing the correspondence between the modules provided in this application and the conventional attention model;

[0056] Fig.10 Schematic diagram of the speedup ratio of SCADE and the baseline on the BEV model provided for this application;

[0057] Fig.11 Schematic diagram of energy savings of SCADE and baseline on the BEV model provided for this application;

[0058] Fig.12 A schematic diagram of the acceleration ratio decomposition of the BEV basic model and the BEV segmentation model provided in this application;

[0059] Fig.13 Schematic diagram of the trade-off between FPS and accuracy loss under different dynamic filtering rates provided by this application;

[0060] Fig.14 A schematic diagram showing the trade-off between FPS and accuracy loss under different dynamic execution ratios provided for this application;

[0061] Fig.15 A flowchart of the data processing method provided in this application.

[0062] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0063] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application.

[0064] Figure 1 The schematic diagram of the von Neumann architecture provided in this application is that in the existing data processing device of the artificial intelligence model, the von Neumann architecture is used to store and calculate data. Figure 1 As shown in FIG. 1 , the von Neumann architecture includes a computing component, a memory, and a system bus. The computing component is the main device for realizing data computing, such as a central processing unit (CPU); the memory is the main device for realizing data storage, such as a memory or a hard disk. A system bus is provided between the computing component and the memory, and the computing component and the memory transmit data through the system bus, wherein the system bus can also be called a global bus. Due to the different manufacturing processes of the memory and the CPU, there may be a large gap in performance between the two. The traditional von Neumann architecture has the characteristics of storage and computing separation, that is, the storage and computing processes are independently run by different components. The storage and computing separation will lead to more serious storage wall problems and power consumption wall problems when data processing is performed based on this architecture, resulting in low data processing efficiency.

[0065] Figure 2 The schematic diagram of the bandwidth of the on-chip network based on the von Neumann architecture provided for this application is as follows: Figure 1 and Figure 2As shown, the computing component includes multiple computing arrays, which can also be called systolic arrays. The maximum bandwidth of a single computing array can be 92TB / s. The bandwidth of the cache system in the memory can be 1TB / s; the bandwidth of the 20MB static random access memory (SRAM) in the memory can be 328TB / s. The bandwidth of the system bus can be 167GB / s. It can be seen that the bandwidths of the computing component, the cache system, and the system bus differ in order of magnitude. The bandwidth of the computing component is 564 times that of the system bus, and the performance of data processing is greatly limited by the data access bandwidth of the system bus.

[0066] Figure 3 The data movement energy consumption diagram based on the von Neumann architecture provided in this application is as follows: Figures 1 to 3 It can be seen that the energy consumption of the system bus, the network energy consumption in the cache system, the energy consumption when reading the SRAM array (SRAM macro), and the energy consumption of data operations in the computing unit also have orders of magnitude differences. Among them, the energy consumption of the system bus is about 420 times that of the computing unit. The energy consumption of the system bus dominates the energy efficiency of the entire data processing device, which greatly limits the endurance of low-power terminal devices based on the von Neumann architecture.

[0067] In summary, the bottleneck of the von Neumann architecture is a limitation that cannot be ignored. To overcome this limitation, the multi-level cache design added to the existing CPU architecture not only fully utilizes the temporal or spatial locality of memory access, but also Figure 2 It can be seen that the bandwidth of a 20MB SRAM can reach more than 3.5 times the bandwidth of a computing component, while its energy consumption for accessing memory is only 1 / 16 of that of a computing component. In some high-performance computing components, about 40% to 60% of the area is occupied by SRAM for storage.

[0068] Although adding some cache to the computing components can alleviate the bottleneck of the von Neumann architecture, it still cannot significantly improve the application performance of the model for AI model application scenarios that require high data processing timeliness. For example, in the field of autonomous driving, it is necessary to efficiently process a large amount of data collected in real time. If the data processing efficiency is low, it cannot meet the real-time requirements of autonomous driving.

[0069] For example, the vehicle's autonomous driving system can be divided into multiple levels from L0 (no automation) to L5 (full automation) according to the degree of automation. As the level increases, the real-time and reliability requirements for the on-board computing system are also getting higher and higher. For example, at level L3 (conditional automation) and above, the system needs to respond in milliseconds to ensure safety. Therefore, higher requirements are placed on the on-board computing system, which not only requires higher processing power to analyze complex environmental data in real time, but also requires lower latency to ensure instant response.

[0070] Some models in the field of autonomous driving, such as BEVFormer, use a bird's-eye view-based approach to improve the understanding of the environment. Although such models perform well in environmental perception, they also bring huge computing power and storage requirements. For example, BEVFormer needs to process a large amount of data from multiple sensors and generate a high-resolution BEV representation in a short time, which places high demands on the computing power and storage speed of the on-board computing system.

[0071] Figure 4 The overall architecture diagram of BEVFormer provided for this application is as follows Figure 4 As shown in the figure, BEVFormer includes a temporal self-attention module, a spatial cross-attention module, a hierarchical norm, and a feed-forward neural network (FFN), where the feed-forward neural network is the fully connected layer of the forward propagation in the figure. The hierarchical norm and fully connected layer structure are similar to the structure of the standard transformer model (Transformer), but BEVFormer has larger input and output feature sizes, so it requires higher computing power. The temporal self-attention module uses a self-attention mechanism to fuse the current BEV query features (current BEV B t ) and historical BEV query features (historical BEV B t-1 ). The deformable attention mechanism needs to get data from the corresponding position in the data cache, which involves large matrix multiplications (such as 512*128) and operations on a large number of matrices, making it more complex than the self-attention of the standard Transformer. The spatial cross attention module transforms tensors from one perspective view (PV) space to another BEV space, which is more complex than the self-attention in the standard Transformer model.

[0072] For example, in the process of fusing the information of the feature maps in the spatial cross attention layer, it is queried whether there is a hit angle v in the feature maps corresponding to multiple images. hit , the hitting angle v hitIt can be understood as a feature point related to the information update of the query point. hit By fusing the information of the query point (x′, y′) in the z dimension, we can get the information of the query point (x′, y′) in the z dimension, that is, (x′, y′, z j ′). In the process of fusing and generating BEV query Q (query feature) in the temporal self-attention layer, the historical query features (historical BEV B t-1 ) and the hit angle v hit Update the feature information of the query point (x, y) based on the information.

[0073] It can be seen that the model complexity of perception tasks in autonomous driving is higher than that of traditional Transformers and traditional Convolutional Neural Networks (CNN), and the demand for acceleration and chip storage is increased. BEVFormer involves a large number of vector-matrix multiplications, and the attention mechanism occupies the main calculation of BEVFormer. Therefore, handing over the core vector-matrix multiplication and vector-vector multiplication to the In-Memory Computing (CIM) architecture can make full use of the high parallel characteristics of its architecture and effectively improve data processing efficiency.

[0074] The existing data processing devices have low data processing efficiency due to the von Neumann bottleneck problem and cannot meet the operating requirements of artificial intelligence models based on the attention mechanism. Therefore, there is a need for a data processing device that can process data efficiently to meet the operating requirements of artificial intelligence models based on the attention mechanism.

[0075] Based on this, the present application provides a data processing device, which is a storage-computation integrated device that uses a memory to calculate and store data, and is used to calculate the feature graphs of multiple data to be processed in sequence to obtain the corresponding query features. The data processing device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module. Each module is a hardware module in the memory and is respectively mapped to each operator of the attention mechanism. In each module, the input data is calculated and stored, and the data calculation process can be performed in the memory, and the operation based on the attention mechanism is realized. Since the calculation process occurs at a position close to the storage array, the dependence on the data access intermediate link such as the system bus can be eliminated. By directly processing data in the memory, the transmission time of data between the computing component and the memory can be greatly reduced, effectively overcoming the limitations brought by the von Neumann architecture, reducing the pipeline overhead and reducing the data access overhead, and increasing the data processing volume per unit time. Therefore, through the data processing device, the effect of improving the data processing efficiency when running an artificial intelligence model based on the attention mechanism can be achieved.

[0076] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0077] Figure 5 The structure of the data processing device provided in this application is shown in FIG. Figure 1 The data processing device is a storage-computation integrated device that uses a memory to perform operations and storage on data, and is used to sequentially calculate the feature graphs of multiple data to be processed to obtain the corresponding query features. Figure 5 As shown, the data processing device includes a pre-calculation module 501, a sampling module 502, an attention weight module 503 and a query generation module 504.

[0078] The pre-calculation module 501 is used to perform weight transformation on the feature map of the current data to be processed, obtain the pre-calculation result, and cache the pre-calculation result.

[0079] Exemplarily, the data to be processed may be any type of data to be processed by an artificial intelligence model in an application scenario, for example, the data to be processed may be image data, point cloud data, or position data, etc. The current data to be processed is the data to be processed at the current moment among the multiple data to be processed.

[0080] Feature maps can be understood as representation information used by artificial intelligence models to understand and process data. For example, feature maps can play an information description role in tasks such as image classification, target detection, and semantic segmentation. Through feature maps, artificial intelligence models can capture and represent important information in the data to be processed, thereby making effective decisions and predictions. The feature map of the data to be processed can be feature representation information obtained after feature extraction of the data to be processed using feature extraction methods such as convolution operations. The feature map can be represented by tensor data such as vectors or matrices. For example, when the data to be processed is point cloud data, the point cloud data can be projected into a two-dimensional image, and the corresponding feature map can be obtained by extracting features from the projected two-dimensional image.

[0081] Taking the application scenario where the data processing device is used to execute the BEVFormer reasoning process as an example, the data to be processed can be images collected by the image acquisition device of the autonomous driving vehicle. The collected images are processed by feature extraction such as convolution operations to obtain feature maps corresponding to the images.

[0082] like Figure 4In the figure, the backbone network can be a network model for extracting features from images in an image acquisition device or an on-board computing system, such as a convolutional neural network. The vehicle's multiple cameras capture images 1 to 6 respectively, that is, multiple perspective inputs are obtained, and each image is data to be processed. These images are input into the backbone network for feature extraction, and the feature maps corresponding to each image can be obtained, that is, the multiple camera features Ft corresponding to the multiple perspective inputs are obtained. After obtaining the feature maps of each data to be processed, each feature map is processed in turn by a data processing device, and feature fusion based on temporal self-attention and spatial cross-attention can be achieved, that is, the feature information in each feature map is fused to obtain the updated current BEV B t Based on the current BEV B t It can perform tasks such as detection and / or segmentation.

[0083] Exemplarily, the pre-calculation module is a hardware module in the memory, and a hardware circuit for implementing vector-matrix multiplication (VMM) can be set in the pre-calculation module to perform weight transformation on the feature map of the current data to be processed. For example, hardware circuits such as multipliers, adders, registers, control units, clock circuits and / or accumulators are arranged in the memory, and vector-matrix multiplication operations can be performed on the data input to the pre-calculation module, thereby achieving weight transformation of the feature map of the current data to be processed, obtaining a pre-calculation result, and caching the pre-calculation result in the memory, which can be a matrix or a vector.

[0084] For example, the pre-calculation module stores the pre-calculation weights W obtained through model training, and performs VMM operation on the pre-calculation weights W and the feature map x of the current data to be processed to obtain the pre-calculation result Wq. The pre-calculation result Wq is stored in the cache of the pre-calculation module and used to calculate with the results of other modules to obtain the query features corresponding to the current data to be processed.

[0085] The sampling module 502 is used to sample the target query feature to obtain a sampling result. The target query feature includes the query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module.

[0086] Exemplarily, the query feature (query, q) is a vector used to retrieve attention-related information. The query feature can be used to perform similarity calculations with the key feature (key, k) to determine which value features (value, v) should be focused and weighted.

[0087] When the feature graphs of multiple data to be processed are processed in sequence, multiple query features can be obtained. Before the moment of processing the current data to be processed, if at least one query feature has been obtained, then one query feature can be determined as the target query feature among these query features, and the data of the target query feature is input into the sampling module 502 for sampling, and a sampling result can be obtained, which can be a vector. If the current data to be processed is the data processed at the first moment, and the sampling module 502 has not obtained the query feature before this, the pre-calculated result corresponding to the data to be processed can be determined as the target query feature for sampling.

[0088] Exemplarily, the sampling module is also a hardware module in the memory, and can perform VMM operations by arranging hardware circuits similar to the pre-calculation module. In addition, the sampling module can also include other (OT) digital circuits for calculating reference points and sampling parameters to complete sampling. Among them, the digital circuit can be, for example, an addition tree circuit, etc.; the sampling parameters can be, for example, linear interpolation parameters, etc.

[0089] When a data processing device executes an artificial intelligence model including a deformable attention mechanism, a sampling module can be used to sample target query features to reduce the computational complexity of data processing. The deformable attention mechanism is an improved attention mechanism designed to more flexibly process information at different spatial locations. Sampling in the deformable attention mechanism can be understood as determining which locations to extract feature information from by selecting query points in the target query features.

[0090] The attention weight module 503 is used to calculate the attention weight of the target query feature to obtain the attention result.

[0091] Exemplarily, the attention weight module 503 is also a hardware module in the memory, and can perform VMM operations by laying out hardware circuits similar to the pre-calculation module. For example, the pre-calculation module stores the attention weight parameters obtained through model training, and performs VMM operations on the attention weight parameters and the target query feature q to obtain the attention weight A ij , that is, the attention result is obtained, and the attention weight A can be ij Stored in the cache of the attention weight module 503, it is used to calculate with the results of other modules to obtain the query features corresponding to the current data to be processed.

[0092] The query generation module 504 is used to perform calculations based on the attention results, sampling results and pre-calculation results to obtain query features corresponding to the data to be processed, and cache the query features.

[0093] Exemplarily, the query generation module 504 is also a hardware module in the memory, which can arrange hardware circuits to perform query feature-related operators on the attention results, sampling results and pre-calculated results, thereby obtaining the query features corresponding to the data to be processed, and can cache them.

[0094] Taking the data processing device executing an artificial intelligence model including a deformable attention mechanism as an example, the underlying operators corresponding to the query generation module 504 include scalar vector multiplication (SM), vector addition (VA) and comparison circuit (CC).

[0095] For example, the attention results, sampling results and pre-calculation results are read into the query generation module to perform SM, VA and CC operations, the results obtained after the operations are determined as the query features corresponding to the current data to be processed, and the query features are cached.

[0096] The data processing device provided in this embodiment is a storage-computation integrated device that uses a memory to calculate and store data, and is used to calculate the feature graphs of multiple data to be processed in sequence to obtain the corresponding query features. The data processing device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module. Each module is a hardware module in the memory and is mapped to each operator of the attention mechanism respectively. The input data is calculated and stored in each module, and the data calculation process can be performed in the memory, and the operation based on the attention mechanism is realized. Since the calculation process occurs at a position close to the storage array, the dependence on the data access intermediate link such as the system bus can be eliminated. By directly processing data in the memory, the transmission time of data between the computing component and the memory can be greatly reduced, effectively overcoming the limitations brought by the von Neumann architecture, reducing the pipeline overhead and reducing the data access overhead, and increasing the data processing volume per unit time. Therefore, through the data processing device, the effect of improving the data processing efficiency when running an artificial intelligence model based on the attention mechanism can be achieved.

[0097] In addition, the data processing device of the present application uses an integrated storage and computing architecture to process the data of the artificial intelligence model, thereby avoiding processing the data through the system bus and computing components. Therefore, it can save the energy consumption of the system bus and computing components, thereby improving the overall energy efficiency during data processing.

[0098] In some embodiments, the query generation module includes an adaptive selector, the adaptive selector includes at least two groups of weight comparison units consisting of a comparator and a first-in-first-out queue, and the query generation module is specifically used to:

[0099] The attention weight in the attention result is compared with the preset attention threshold by the comparator in the weight comparison unit, and the attention weight greater than the attention threshold is cached in the corresponding first-in-first-out queue; according to the attention weight output by the first-in-first-out queue, the filtering result corresponding to the attention result is obtained; according to the filtering result, the sampling result and the pre-calculated result, the query feature corresponding to the data to be processed is obtained, and the query feature is cached.

[0100] Specifically, the adaptive selector can be a filter for dynamically optimizing the attention weights in the attention results. Among them, the comparator can be a hardware circuit or component for comparing two input values, and the input value can be a scalar or a vector. The first-in first-out (FIFO) queue can be a structure that implements data caching through units such as storage units, read pointers, write pointers, and control logic. The weight comparison unit composed of the comparator and the first-in first-out queue constitutes a ping-pong-like structure to support dynamic attention weight filtering and selection. The attention threshold can be a pre-set value for comparing and judging the attention weight, and the attention threshold can be an arbitrary value set according to the requirements of the application scenario.

[0101] Figure 6 The structure of the data processing device provided in this application is shown in FIG. Figure 2 , Figure 6 The data processing device shown can be used to execute an artificial intelligence model including a deformable attention operator (DeformAttn) or an artificial intelligence model including a conventional attention mechanism. Figure 7 This is a schematic diagram of the structure of the query generation engine and dynamic execution unit provided in this application. Figure 6 and Figure 7 As shown, the query generation engine in the data processing device is a query generation module, and the query generation engine includes an adaptive selector, a scalar multiplication, a vector multiplication, and an output query buffer. The adaptive selector includes two weight comparison units, each of which includes a comparator and a FIFO queue, such as Figure 7 The query generation engine can read the attention results in the attention buffer and assign each attention weight A in the attention result to the FIFO#0 or FIFO#1. ij The weights are input into the comparator of a weight comparison unit and compared with the attention threshold δ. ij When the absolute value of is greater than the attention threshold δ, the attention weight A ij It is cached in the FIFO queue of the corresponding weight comparison unit. ij If the absolute value of is less than or equal to the attention threshold δ, it can be discarded. By collecting the attention weights A selected in each FIFO queue ijA filtering result corresponding to the attention result can be obtained, and the filtering result can be understood as a result of dynamically optimizing and filtering the attention result.

[0102] In some embodiments, the query generation module is specifically used to: perform multiplication and accumulation operations on the filtering results, sampling results and pre-calculated results to obtain an accumulated result; obtain the query features corresponding to the data to be processed based on the result of comparison through a comparison circuit according to the accumulated result, and cache the query features.

[0103] Figure 8 A schematic diagram of the module and operator mapping provided for this application, Figure 8 The VMM, SM, VA, etc. operators shown in Figure 6 The operation process of each module of the data processing device is mapped. Figures 6 to 8 As shown in , after collecting the selected attention weights to obtain the filtering results, the filtering results can be multiplied by the sampling results obtained from the sampling buffer, and then the multiplication results can be multiplied by the pre-calculated results obtained from the pre-calculated buffer, and the results after the multiplication with the pre-calculated results can be accumulated to obtain the accumulated results. The accumulated results are compared by the comparison circuit to obtain the query features corresponding to the data to be processed. Figure 8 In the example, the input of the query generation engine can be understood as the target query feature q, and after the SM, VA and CC operations, the updated target query feature is output. The updated target query feature is cached through the output query buffer.

[0104] In this embodiment, the query generation module filters and screens the attention weights in the attention results through an adaptive selector, which can filter out irrelevant or noise features, helping the model to focus on important features more accurately, thereby improving the overall performance of the model. In addition, the weight comparison unit composed of a comparator and a first-in-first-out queue constitutes a ping-pong-like structure, which can support dynamic attention weight filtering and selection, and improve the effectiveness and filtering efficiency of dynamic optimization filtering. In addition, through dynamic optimization filtering, the number of features that need to be processed can also be reduced, thereby reducing the computational complexity and improving the response speed of the model. Further, the filtering results, sampling results, and pre-calculated results are multiplied and added to obtain a cumulative result; the query feature corresponding to the data to be processed is obtained according to the result of the comparison by the comparison circuit according to the cumulative result, which can effectively realize the query feature calculation through the hardware module query generation module in the memory, and realize the query feature update generation of the storage and computing integrated architecture.

[0105] In some embodiments, the pre-calculation module includes a pre-calculation engine and a pre-calculation buffer. The pre-calculation module is specifically used to: multiply the feature map of the data to be processed according to the pre-calculation weights pre-stored in the pre-calculation engine, obtain the pre-calculation result, and cache the pre-calculation result in the pre-calculation buffer.

[0106] Take the data processing device executing the data processing of BEVFormer as an example, Figures 6 to 8 As shown, the pre-calculation module in the data processing device includes a pre-calculation engine for calculating data and a pre-calculation buffer for caching pre-calculation results. The pre-calculation engine and the pre-calculation buffer can be used to implement pre-calculation optimization in BEVFormer. Specifically, the pre-calculation engine receives the feature map x as input, and performs a transformation with a pre-calculation weight W on it, that is, multiplying the feature map of the processed data, and this operation process corresponds to the matrix-vector multiplication operator VMM. Among them, the feature map x may come from the backbone network or the BEV query feature of the previous moment according to the temporal self-attention or spatial cross-attention mechanism; the pre-calculation weight W can be the weight matrix W i W i ′, where W i ′ represents the matrix W i The pre-calculated weights can be weight parameters obtained through model training. The pre-calculated results calculated by the pre-calculation engine will be output to the pre-calculation buffer for temporary storage.

[0107] Based on this, feature extraction can be performed on the feature graph through calculations by the pre-computation engine, thereby providing necessary feature information for updating query features and obtaining query features corresponding to the data to be processed of the feature graph.

[0108] In some embodiments, the pre-calculation engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing units include a write driver, a multiplication logic circuit, an addition tree circuit and a storage array, and the storage array pre-stores a first weight value of the pre-calculated weight; the pre-calculation module is specifically used for:

[0109] The first characteristic data of the characteristic graph of the data to be processed is obtained by writing to the driver; the first characteristic data and the first weight value are multiplied by the multiplication logic circuit to obtain a first product; the first product is added by the addition tree circuit to obtain a first sum; the first sum is accumulated by the accumulation unit to obtain a second sum, and the second sum is transmitted to the pre-calculation buffer for caching.

[0110] Exemplarily, the storage-computation integrated array can be understood as a hardware circuit array for implementing pre-computation, the storage-computation integrated processing unit is a computing unit that performs the pre-computation process, and the storage-computation integrated processing unit includes a storage array, which can be understood as an array composed of multiple storage elements. The storage element can be any storage element, for example, a 6T SRAM with 6 transistors or an 8T SRAM with 8 transistors. The first weight value can be a numerical value in the pre-computation weight, which can be understood as an element value in the weight matrix of the pre-computation weight. The first feature data can be a numerical value in the feature map, which can be understood as an element value in the matrix corresponding to the feature map.

[0111] The storage-computation integrated processing unit also includes a write driver, a multiplication logic circuit, and an addition tree circuit. The write driver may be a driving circuit for performing data reading and writing. The multiplication logic circuit may be a circuit for performing multiplication operations, such as a parallel multiplier, a serial multiplier, a shift and add multiplier, an array multiplier, etc. The addition tree circuit may be a tree structure composed of adders, which may be understood as an adder tree or an addition tree.

[0112] For example, Figure 6 As shown, the pre-computation engine includes multiple storage-computation integrated arrays, such as CIM Array#0, CIMArray#1, etc. Each storage-computation integrated array includes multiple computing units, and the computing units include storage-computation integrated processing units, such as CIMPE#0, and accumulation units. Optionally, the storage-computation integrated processing unit CIM PE is equipped with an 8TSRAM array of 256 columns × 64 rows. Each CIM PE can store 2048 8-bit weights, and in each cycle, a multiplication-accumulation (MAC) operation can be performed on 32 8-bit weights and inputs.

[0113] For example, Figure 6 and Figure 8 As shown, the pre-calculation module can obtain the first feature data of the feature map of the data to be processed in the input buffer by writing to the driver. The first feature data will be input into the multiplication logic circuit and multiplied with the first weight value stored in the 8TSRAM array, that is, the VMM operation is performed on the input feature map x to obtain the first product. The first product is added by the addition tree circuit to obtain the first sum value; the first sum value is accumulated by the accumulation unit to obtain the second sum value, which can be understood as an element in the pre-calculation result. The second sum value is transmitted to the pre-calculation buffer for caching, that is, the feature map pre-calculation result is cached.

[0114] In this embodiment, the pre-calculation engine includes multiple storage and calculation integrated arrays, which include a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array; the first feature data is obtained through the write driver and a multiplication operation is performed through the multiplication logic circuit, the first product is added through the addition tree circuit, and the first sum is accumulated and calculated through the accumulation unit to obtain the second sum, thereby obtaining the pre-calculation result. Based on this, the storage and calculation integrated pre-calculation module VMM operation is realized, which can improve the efficiency of the model in pre-calculation processing.

[0115] In some embodiments, the sampling module includes a sampling engine and a sampling buffer. The sampling engine is used to calculate the corresponding offset and bilinear interpolation parameters according to the target query characteristics, and calculate the sampling results corresponding to the offset reference point based on the offset and bilinear interpolation parameters, and transmit the sampling results to the sampling buffer for caching.

[0116] Specifically, the sampling engine is the part of the sampling module that performs data calculations, and the sampling buffer is the part of the sampling module that performs data caching. The offset can be understood as the difference between the reference point and the offset reference point. The offset reference point can be understood as the corresponding point after the offset adjustment based on the reference point. The offset can be a hyperparameter obtained through model training. The bilinear interpolation method can be used to quickly and effectively obtain the feature information of the offset reference point in the target query feature. The bilinear interpolation parameters can be understood as the values ​​of the four points around the offset reference point used to calculate the value of the offset reference point.

[0117] For example, Figure 6 and Figure 8 As shown, the sampling module in the data processing device includes a sampling engine for calculating data and a sampling buffer for caching sampling results. The sampling engine can calculate the corresponding offset Δp according to the input target query feature q ij And for the reference point p+Δp after adding the offset adjustment ij The input of the sampling engine is q, and the output is p+Δp ij The operations in the sampling engine include matrix-vector multiplication operators VMM and other basic digital circuit operations OT required for calculating reference points and interpolation parameters. The sampling engine may include multiple storage-computation-in-one arrays, and the structure of the storage-computation-in-one array may be similar to the storage-computation-in-one array in the pre-computation engine in the above embodiment.

[0118] Exemplarily, the following formula (1) and formula (2) are calculation formulas corresponding to the deformable attention operator:

[0119]

[0120] Among them, q represents the query feature; p represents the reference point; x represents the input feature; Δp ij Indicates the offset; W i represents the weight matrix, W i ′ represents the weight matrix W i The transposed matrix of ij represents the attention weight; N head represents the number of attention heads; N key represents the number of reference points sampled by each attention head; t i,j,k represents the interpolation parameter, i represents the i-th attention head, j represents the j-th reference point, and k represents the k-th point around the reference point.

[0121] For example, the reference point p can be obtained by random sampling, and after calculating the offset, the reference point p+Δp ij When the corresponding sampling result is obtained, the offset Δp corresponding to the reference point p can be calculated by other basic digital circuit operations such as addition tree OT ij and the interpolation parameter t i,j,k The VMM operation is performed by the storage and computing integrated array in the sampling engine to obtain the offset reference points p+Δp corresponding to the target query feature q ij , that is, obtaining the sampling result, and caching the sampling result in the sampling buffer.

[0122] Based on this, the sampling module can realize the integrated storage and computing of target query feature sampling, and the sampling operation in the deformable attention operator is realized based on the hardware module, which reduces the amount of calculation when updating query features when applying models such as BEV and improves the updating efficiency of query features.

[0123] In some embodiments, the attention weight module includes an attention engine and an attention buffer, the attention engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a function activation unit and a plurality of computing units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing unit includes a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array consisting of a plurality of storage elements, the storage array pre-stores a second weight value of the attention weight; the attention weight module is specifically used for:

[0124] The second feature data in the target query feature is obtained by writing to the driver; the second feature data and the second weight value are multiplied by the multiplication logic circuit to obtain a second product; the second product is added by the addition tree circuit to obtain a third sum; the third sum is accumulated by the accumulation unit to obtain a fourth sum; the fourth sum is input into the function activation unit to obtain the activated fourth sum, and the activated fourth sum is transmitted to the attention buffer for caching.

[0125] Specifically, the attention engine is the part that performs data calculation in the attention weight module, and the attention buffer is the part that performs data caching in the attention weight module. In the attention engine, the computing unit of the storage-computation integrated array is similar to the computing unit in the pre-computation engine in the above embodiment. The storage-computation integrated array of the attention engine also includes a function activation unit, which can be a unit that implements activation function operations such as a rectified linear unit (RELU), a Gaussian error linear unit (GELU), and a normalized exponential function (Softmax Function, Softmax). The function activation unit can be used to perform nonlinear transformations on the output results of the computing unit to increase the model's ability to characterize nonlinearity.

[0126] The second weight value may be a value in the pre-calculated weight, which may be understood as an element value in the weight matrix of the attention weight. The second feature data may be a value in the target query feature, which may be understood as an element value in the matrix corresponding to the target query feature.

[0127] For example, when the data processing device executes the BEVFormer application, Figure 6 As shown, the data processing device also includes a global BEV query buffer for storing the obtained query features. The global BEV query buffer can transmit the target query features to the input buffer through a data bus in the memory.

[0128] like Figure 6 and Figure 8 As shown, the attention engine can obtain the second feature data in the target query feature in the input buffer by writing to the driver. The second feature data will be input into the multiplication logic circuit and multiplied with the second weight value stored in the 8T SRAM array, that is, the VMM operation is performed on the target query feature q to obtain the second product. The second product is added by the addition tree circuit to obtain the third sum; the third sum is accumulated by the accumulation unit to obtain the fourth sum. For example, the fourth sum is input into the function activation unit to perform a Softmax operation for normalizing the attention weight. The fourth sum is input into the function activation unit for activation to obtain the fourth sum after activation. Each fourth sum after activation can be understood as the attention weight Aij of the corresponding query point in the target query feature, and the fourth sum is transmitted to the pre-calculation buffer for caching, that is, the attention result corresponding to the target query feature is cached.

[0129] Based on this, the attention weight module can realize the coarse attention weight calculation of the target query feature with integrated storage and calculation. The attention engine can determine the attention weight for each query point in the target query feature, which is used to weight the calculation results in the subsequent calculation stage.

[0130] In some embodiments, the data processing device further includes a dynamic execution unit, which is connected to the query generation module; the dynamic execution unit is used to:

[0131] The query feature corresponding to the current data to be processed obtained by the query generation module is obtained; the similarity calculation circuit in the dynamic execution unit performs similarity calculation on the query feature corresponding to the current data to be processed and the query feature obtained previously, and obtains a similarity value; the comparison circuit in the dynamic execution unit compares the similarity value with a preset similarity threshold value, and obtains a similarity comparison result, and the similarity comparison result is used to indicate whether the query feature corresponding to the current data to be processed is determined as the target query feature corresponding to the data processing of the next data to be processed.

[0132] Exemplarily, after the query generation module obtains the query feature corresponding to the current data to be processed, the query feature can be identified and processed by the dynamic execution unit to determine whether the current query feature is similar to the query feature of the previous data to be processed. If the two are similar, it can be understood that the feature changes of each query point of the current query feature are small. At this time, if the attention weight result and sampling result obtained by calculating the previous query feature are used to calculate the query feature of the next data to be processed, it can avoid inputting the current query feature into the sampling module and the attention weight module for calculation, which can reduce the occupation of computing resources, reduce the amount of calculation, and improve the update efficiency of the query feature.

[0133] Exemplarily, the similarity calculation circuit may be, for example, a circuit for calculating cosine similarity. The comparison circuit may be a circuit capable of comparing input values. The preset similarity threshold may be any value preset according to actual requirements of the application scenario.

[0134] For example, Figures 6 to 8 As shown, the dynamic execution unit may include a similarity calculation circuit and a comparison circuit, and the dynamic execution unit may be used to control the dynamic update of each query feature obtained. Figure 7 The new vector in can be understood as the query feature corresponding to the current data to be processed, and the old vector can be understood as the query feature obtained before, for example, it can be the query feature corresponding to the previous feature to be processed or the query feature corresponding to the previous feature to be processed.

[0135] The dynamic execution unit may also include a mask circuit for generating a mask. The dynamic execution unit performs similarity calculation on the query feature new q corresponding to the input current data to be processed and the query feature q obtained previously to obtain a similarity value, that is, performs OT operation and CC operation. After comparing the similarity, the dynamic execution unit can output a mask for the query feature corresponding to the current data to be processed. For example, the mask can be a binary mask, in which 1 indicates that the query feature corresponding to the current data to be processed is not similar to the query feature obtained previously, and 0 indicates that the query feature corresponding to the current data to be processed is similar to the query feature obtained previously. The output mask can indicate whether to use the query feature corresponding to the current data to be processed as the target query feature for query feature update in the next update cycle.

[0136] In this embodiment, a mechanism for similarity identification of query features corresponding to the current data to be processed can be implemented through the dynamic execution unit. This mechanism can skip unnecessary calculations for redundant or unimportant queries, thereby effectively utilizing computing resources and improving data processing efficiency.

[0137] In some embodiments, the data processing device also includes a fully connected layer calculation module, which is connected to the query generation module and is used to perform linear transformation calculation on the query features obtained by the query generation module to obtain linearly transformed query features.

[0138] Specifically, the fully connected layer calculation module is used to perform linear layer transformation on the query features, and the fully connected layer calculation module may include the calculation units in the above embodiments.

[0139] For example, Figures 6 to 8 As shown, the fully connected layer computing module may include a fully connected layer computing engine and a fully connected layer buffer, and the fully connected layer computing engine includes a fully digital storage and computing integrated computing unit based on SRAM, which can be used to perform the operation of the VMM operator. Through the fully connected layer computing engine, the input query feature q and the preset fully connected weight W can be calculated and processed to obtain the fully connected calculation result Wq.

[0140] Based on this, the query features obtained can be effectively summarized and optimized through the fully connected layer computing module to facilitate the subsequent detection, segmentation or other application tasks.

[0141] For example, take the application scenario of a data processing device executing BEVFormer as an example. Figure 6As shown, the data processing device may also include a controller and a layer norm unit. The layer norm unit is connected to the global BEV query buffer and is used to perform feature normalization processing on the BEV query features. The controller is respectively connected to the pre-calculation module, the sampling module, the attention weight module, the query generation module, the fully connected layer calculation module, the dynamic execution unit, etc., and is used to control the connected modules. The controller can be a storage controller in the memory, which can generate control signals to control each module, and the control signals can instruct the corresponding module to perform operations such as data reading, data writing, and data transmission.

[0142] When the data processing device calculates and updates the graph features corresponding to the image input to BEVFormer, the data flow is as follows: Figure 6 and Figure 7 As shown. The global BEV query buffer records the calculation results of each layer (temporal self-attention layer / spatial cross-attention layer) of each BEV block each time, that is, the BEV query features. Before the calculation of the temporal / spatial attention mechanism layer, the BEV query features are input into the pre-calculation engine as feature maps, and the results of the query generation engine are stored back in the global BEV query buffer after the calculation is completed. When calculating the fully connected layer, the BEV query features are input into the fully connected layer calculation engine, and the calculation results are stored back in the global BEV query buffer after the calculation is completed.

[0143] For the calculation of the temporal self-attention mechanism / spatial cross-attention layer, the pre-computation engine first transforms the input feature map x to obtain the pre-computation result x′ and stores it in the pre-computation buffer. After the pre-computation is completed, the sampling engine calculates the offset Δp based on the input BEV query feature and position p ij , and calculate the reference point p+Δp based on the offset ij And the corresponding bilinear interpolation parameter t i,j,k , and outputs the calculation results to the sampling buffer. At the same time, the attention engine calculates the attention weight A based on the input BEV query features ij And input into the attention buffer of the query generation module.

[0144] The query generation engine takes the pre-computed results as input and generates updated BEV query features. These updated BEV query features are stored back in the global BEV query buffer. The adaptive selector in the query generation engine filters out attention weights below the attention threshold based on the size of the attention weights. The filtered results are sent to the scalar multiplication unit together with the cached sampling results and pre-computed results to complete the calculation process. This data flow can effectively handle the calculation and update of BEV query features in the BEVFormer model.

[0145] Exemplarily, the BEV query features stored in the global BEV query buffer can be sent to the fully connected layer calculation engine for fully connected layer calculation. The resulting BEV query features are stored back in the global BEV query buffer again. The mask bits from the dynamic execution unit can be appended to the BEV query features and stored back in the global BEV query buffer. When obtaining the BEV query features for updating, if the corresponding mask bit of the BEV query feature is 0, the update of the BEV query feature will be skipped.

[0146] Exemplarily, the above-mentioned data processing device can also be used to perform data computing tasks of artificial intelligence models that include conventional attention mechanisms. Fig. 9 The schematic diagram of the modules provided in this application corresponding to the conventional attention model is as follows: Fig. 9 As shown, the pre-computation array can correspond to the pre-computation module, the local calculation unit can correspond to the sampling module, the query generation unit can correspond to the query generation module, the attention weight calculation unit can correspond to the attention weight module, and the fully connected layer array can correspond to the fully connected layer calculation module.

[0147] like Fig. 9 As shown, since the processing units of each engine and computing unit of the data processing device are basically based on a fully digital SRAM storage and computing unit that supports vector matrix multiplication acceleration, the data processing device provided by the embodiment of the present application can be mapped to a conventional attention computing mechanism architecture. Specifically, the pre-calculated array is mapped to complete the Q×K T The engine for this matrix calculation uses pre-calculated memory to cache the calculation result S / P. The local calculation unit and the attention weight calculation unit can be used to complete the multiplication calculation of S / P and the matrix V. The fully connected layer array remains unchanged to continue to support the fully connected layers of the model based on the attention mechanism architecture. In addition, except for the pruned filters that are not mapped, the vector scalar multiplication unit and the vector accumulation unit can be used in the network architecture based on the conventional attention mechanism.

[0148] By way of example, in order to intuitively understand the performance of the data processing device provided in the embodiments of the present application in terms of data processing, the performance evaluation of the SRAM-based data processing device is described below.

[0149] The data processing device based on the SRAM storage and computing integrated architecture can efficiently accelerate the BEVFormer model. By integrating the storage and computing units together and adopting high-bandwidth, low-latency SRAM storage units, storage optimization is achieved, access to external memory is reduced, and memory access efficiency is improved. Through comprehensive performance evaluation and tuning, a superior acceleration effect is obtained on the BEVFormer model. The following performance evaluation experiment describes the comparison between the data processing device of the present application and a graphics processor (Graphics Processing Unit, GPU), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), etc., to verify the performance of the data processing device of the present application in terms of acceleration ratio, energy consumption, filtering rate and dynamic execution ratio.

[0150] First, the evaluation setup is introduced. Since there is currently no specific accelerator design for the BEV model, the data processing device of the present application can be compared with a GPU, a spatial attention (SpAtten) circuit, and a spatially adaptive denormalization (SPADE) circuit. SpAtten supports dynamic attention pruning, and SPADE supports BEV-like pillar modeling of point clouds, which have certain similarities with the BEV model calculations. Therefore, the execution process of the BEV model is mapped to these two ASIC designs for comparison. Table 1 is a static data comparison table of the three accelerators provided in this application.

[0151] Table 1

[0152]

[0153] As shown in Table 1, SCADE represents the data processing device of the present application. SCADE can be implemented using Verifying Logic (Verilog), and a design compiler (Design Compiler) is used to synthesize the register transfer level (RTL) using a semiconductor manufacturer's 28nm standard library at a frequency of 400MHz to evaluate the area and power consumption of the data processing device. For SRAM-based CIM PE, measurements can be made on the chip brought out. For example, the measured voltage is 0.9V, the frequency is 400MHz, the size is 64×256, the single write power consumption is 7.6nJ, and the single read power consumption is 6.6nJ. Table 2 is a SCADE configuration parameter table provided by the present application, and the specific configuration of SCADE is shown in Table 2.

[0154] Table 2

[0155]

[0156] A cycle-accurate simulator can be built from the measured data to simulate microarchitectural behavior. For on-chip SRAM, the Cache Access and Cycle Time Interface (CACTI) 7.0 can be used to model area and power consumption.

[0157] Fig.10 The schematic diagram of the speedup ratio of SCADE and the baseline on the BEV model provided for this application is as follows: Fig.10 As shown in Figure 3, SCADE achieves substantial speedups compared to GPU and accelerator baselines. On average, SCADE is 15.2 times faster than GPUs and 4.17 times faster than ASIC accelerators. Specifically, SCADE achieves nearly 3.50 times better performance than the leading accelerator SPADE. This efficiency stems from SCADE’s ability to reduce computational complexity by eliminating redundant temporal and spatial attention computations through precomputation. Dynamic filtering and execution minimizes unnecessary computations, while pipelined execution fully utilizes on-chip resources for fast inference.

[0158] Fig.11 A schematic diagram of energy savings of SCADE and baseline on the BEV model provided for this application is shown in Fig.11 As shown in Figure 2, the energy saving comparison results of SCADE and the baseline on the BEV model show that SCADE achieves significant energy saving compared with other accelerators. In multiple 3D perception tasks, SCADE saves an average of 37.6× energy. It is worth noting that when performing 3D object detection tasks using the BEV lightweight (BEV-tiny) model, SCADE achieves even higher energy saving of 5206.5×. Compared with SpAtten and SPADE, SCADE's energy consumption is reduced by 6.54× and 5.22×, respectively. This is due to its low-power CIM design that minimizes data migration, dynamic filtering and execution mechanisms, and customized query generation engines and dynamic execution units.

[0159] Fig.12 The schematic diagram of the acceleration ratio decomposition of the BEV basic model and the BEV segmentation model provided in this application is as follows: Fig.12As shown in the figure, BEV-base represents the BEV base model; BEV-seg represents the BEV segmentation model, which can be a BEV model specifically used for semantic segmentation tasks; Pc, Fi, and Dy represent pre-computation, dynamic filtering, and dynamic execution, respectively. Details of the speed improvement of SCADE relative to V100GPU in 3D object detection and map segmentation tasks. Compared with GPUs with a large number of computing cores, SCADE can still achieve an average performance improvement of 6.36 times. It is worth noting that on the segmentation task, SCADE is 9.80 times faster than the GPU. Specifically, SCADE (pc) optimization further improves the acceleration and achieves a speed improvement of 4.31×. This is mainly due to the significant reduction in the amount of calculation through pre-computation optimization. In addition, compared with the dynamic filtering mechanism, the acceleration effect of the dynamic execution mechanism is better. This is because the dynamic execution mechanism allows unnecessary updates to be skipped in a coarse-grained manner, further improving efficiency.

[0160] Fig.13 The trade-off diagram between FPS and accuracy loss under different dynamic filtering rates provided by this application is shown in Figure 1. The dynamic filtering mechanism of the query generation module in the data processing device of this application is verified by adjusting the filtering rate. The dynamic filtering rate can be changed from 5% to 50%, and the corresponding frames per second (FPS) and mean average precision (mAP) are evaluated. The results are shown in Figure 1. Fig.13 As shown. As the filtering rate increases, the mAP remains relatively stable, which indicates that the dynamic filtering method of the data processing device of the present application effectively discards information with unimportant attention weights without affecting the accuracy of reasoning. In addition, at some filtering rates, the mAP of SCADE is even higher than that of the unfiltered version, which indicates that the filtering process may help eliminate noise information. At a filtering rate of 50%, SCADE improves FPS by 1.36 times by reducing vector operations.

[0161] Fig.14 A schematic diagram of the trade-off between FPS and accuracy loss under different dynamic execution ratios provided in this application. For the dynamic filtering mechanism of the dynamic execution unit in the data processing device of this application, this part of the design space exploration is verified by adjusting the dynamic execution ratio. Fig.14 The effect of dynamic execution ratio on FPS and mAP of nuScenes dataset is shown. Fig.14As shown, as the percentage of dynamic execution vectors increases, mAP fluctuates, but these fluctuations are still within a tolerable range. Specifically, when using the BEVbase model, a 50% dynamic execution rate significantly improves SCADE's FPS by 1.36 times. This proves the effectiveness of the dynamic execution mechanism in improving the performance of the base model. In addition, when configured with a dynamic execution rate of 25%, SCADE's FPS using the BEV small model (BEV small) reached 100 times. This further highlights the performance improvement brought by dynamic execution. It is worth noting that at 25%, mAP is even improved by 1.06 due to avoiding noise interference.

[0162] Fig.15 A flow chart of the data processing method provided in this application, such as Fig.15 As shown, the data processing method provided in this embodiment is applied to a storage-computation integrated device that uses a memory to calculate and store data. The storage-computation integrated device is used to sequentially calculate feature graphs of multiple data to be processed to obtain respective corresponding query features. The storage-computation integrated device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module; the method includes:

[0163] Step 1501, performing weight transformation on the feature map of the current data to be processed based on the pre-calculation module, obtaining a pre-calculation result, and caching the pre-calculation result;

[0164] Step 1502, sampling the target query feature based on the sampling module to obtain a sampling result, where the target query feature includes a query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module;

[0165] Step 1503, calculating the attention weight of the target query feature based on the attention weight module to obtain an attention result;

[0166] Step 1504, based on the query generation module, calculations are performed according to the attention results, sampling results and pre-calculation results to obtain query features corresponding to the data to be processed, and the query features are cached.

[0167] In some embodiments, the query generation module includes an adaptive selector, which includes at least two groups of weight comparison units consisting of comparators and first-in-first-out queues, and the query generation module calculates according to the attention results, the sampling results and the pre-calculation results to obtain the query features corresponding to the data to be processed, and caches the query features, including:

[0168] The comparator in the weight comparison unit compares the attention weight in the attention result with the preset attention threshold, and caches the attention weight greater than the attention threshold in the corresponding first-in-first-out queue;

[0169] According to the attention weights output by the first-in-first-out queue, the filtering results corresponding to the attention results are obtained;

[0170] Calculation is performed based on the filtering results, sampling results, and pre-calculation results to obtain the query features corresponding to the data to be processed, and the query features are cached.

[0171] In some embodiments, calculations are performed based on the filtering results, the sampling results, and the pre-calculation results to obtain query features corresponding to the data to be processed, and the query features are cached, including:

[0172] Perform multiplication and accumulation operations on the filtering result, the sampling result and the pre-calculation result to obtain an accumulation result;

[0173] The query feature corresponding to the data to be processed is obtained through comparison of the accumulated result through a comparison circuit, and the query feature is cached.

[0174] In some embodiments, the pre-calculation module includes a pre-calculation engine and a pre-calculation buffer, and the pre-calculation module performs weight transformation on the feature map of the current data to be processed to obtain a pre-calculation result, and caches the pre-calculation result, including:

[0175] The feature map of the data to be processed is multiplied according to the pre-calculated weights pre-stored in the pre-calculation engine to obtain a pre-calculated result, and the pre-calculated result is cached in the pre-calculation buffer.

[0176] In some embodiments, the pre-calculation engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing units include a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array, and the storage array pre-stores a first weight value of a pre-calculation weight; performing multiplication calculation on a feature map of the to-be-processed data according to the pre-calculation weight pre-stored in the pre-calculation engine to obtain a pre-calculation result, and caching the pre-calculation result in a pre-calculation buffer, including:

[0177] Acquire first characteristic data of a characteristic graph of data to be processed by writing to a driver;

[0178] Performing a multiplication operation on the first feature data and the first weight value through a multiplication logic circuit to obtain a first product;

[0179] Performing an addition operation on the first product through an addition tree circuit to obtain a first sum value;

[0180] The first sum is accumulated and calculated by an accumulation unit to obtain a second sum, and the second sum is transmitted to a pre-calculation buffer for caching.

[0181] In some embodiments, the sampling module includes a sampling engine and a sampling buffer. The sampling engine is used to calculate the corresponding offset and bilinear interpolation parameters according to the target query characteristics, and calculate the sampling results corresponding to the offset reference point based on the offset and bilinear interpolation parameters, and transmit the sampling results to the sampling buffer for caching.

[0182] In some embodiments, the attention weight module includes an attention engine and an attention buffer, the attention engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a function activation unit and a plurality of computing units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing unit includes a write driver, a multiplication logic circuit, an addition tree circuit, and a storage array consisting of a plurality of storage elements, and the storage array pre-stores a second weight value of the attention weight; based on the attention weight module, the attention weight is calculated for the target query feature to obtain an attention result, including:

[0183] Acquire second feature data in the target query feature by writing to the driver;

[0184] Performing a multiplication operation on the second feature data and the second weight value through a multiplication logic circuit to obtain a second product;

[0185] Performing an addition operation on the second product through an addition tree circuit to obtain a third sum value;

[0186] The third sum is accumulated and calculated by an accumulation unit to obtain a fourth sum;

[0187] The fourth sum value is input into the function activation unit to obtain the activated fourth sum value, and the activated fourth sum value is transmitted to the attention buffer for caching.

[0188] In some embodiments, the storage-computing integrated device further includes a dynamic execution unit, and the dynamic execution unit is connected to the query generation module; the method further includes:

[0189] Obtain the query features corresponding to the current data to be processed obtained by the query generation module;

[0190] The similarity calculation circuit in the dynamic execution unit calculates the similarity between the query feature corresponding to the current data to be processed and the previously obtained query feature to obtain a similarity value;

[0191] The similarity value is compared with a preset similarity threshold by a comparison circuit in the dynamic execution unit to obtain a similarity comparison result, which is used to indicate whether the query feature corresponding to the current data to be processed is determined as the target query feature corresponding to the data processing for the next data to be processed.

[0192] In some embodiments, the storage and computing integrated device also includes a fully connected layer computing module, which is connected to the query generation module and is used to perform linear transformation calculation on the query features obtained by the query generation module to obtain linearly transformed query features.

[0193] The data processing method provided in this embodiment may correspond to the data processing devices in the above embodiments, and the implementation principles and technical effects thereof are similar, so they will not be described in detail in this embodiment.

[0194] An embodiment of the present application further provides a vehicle, which includes the data processing device in any of the above embodiments, and the vehicle processes the data to be processed of the bird's-eye view model through the data processing device to obtain query features of the bird's-eye view model.

[0195] The data processing device may be an on-board chip in the vehicle or part of an on-board chip. For example, existing on-board chips have limited performance in raw data processing and model reasoning; in addition, existing on-board chips have not completely solved the von Neumann bottleneck problem, that is, the data transmission speed between the processor and the storage is still a bottleneck. Therefore, some existing on-board chips cannot fully meet the needs of more advanced autonomous driving models such as the BEVFormer model. Therefore, if a vehicle uses some existing on-board chips, even if advanced autonomous driving algorithms are used, the actual on-board system may not be able to achieve the ideal performance level.

[0196] The data processing device in the above embodiment integrates the storage and computing units together and adopts a high-bandwidth, low-latency SRAM storage unit to achieve storage optimization, reduce access to external memory, and improve memory access efficiency. The vehicle provided in this embodiment adopts the storage-computation integrated data processing device of the above embodiment, that is, an on-board chip with high-efficiency data processing. Based on the huge potential of the storage-computation integrated architecture in accelerating big data processing, the vehicle can efficiently accelerate the processing efficiency of the BEVFormer model based on the SRAM storage-computation integrated architecture on the basis of models such as BEVFormer, and can effectively improve the vehicle's autonomous driving performance.

[0197] In the above embodiments, it should be understood that the processor may be a central processing unit CPU, or other general-purpose processors, digital signal processors (English: Digital Signal Processor, referred to as: DSP), application-specific integrated circuits ASIC, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0198] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0199] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0200] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0201] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0202] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory SRAM, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0203] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit ASIC. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0204] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0208] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0209] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data processing device, characterized in that: The data processing device is a storage-computation integrated device that uses a memory to calculate and store data, and is used to sequentially calculate the feature graphs of multiple data to be processed to obtain the corresponding query features; the data processing device includes a pre-calculation module, a sampling module, an attention weight module and a query generation module; The pre-calculation module is used to perform weight transformation on the feature map of the current data to be processed, obtain a pre-calculation result, and cache the pre-calculation result; The sampling module is used to sample the target query feature to obtain a sampling result, wherein the target query feature includes a query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module; The attention weight module is used to calculate the attention weight of the target query feature to obtain an attention result; The query generation module is used to perform calculations based on the attention results, the sampling results and the pre-calculation results to obtain query features corresponding to the data to be processed, and cache the query features.

2. The device according to claim 1, characterized in that The query generation module includes an adaptive selector, and the adaptive selector includes at least two groups of weight comparison units consisting of a comparator and a first-in-first-out queue. The query generation module is specifically used for: Comparing the attention weight in the attention result with a preset attention threshold through a comparator in the weight comparison unit, and caching the attention weight greater than the attention threshold in a corresponding first-in-first-out queue; Obtaining a filtering result corresponding to the attention result according to the attention weight output by the first-in-first-out queue; Calculation is performed according to the filtering result, the sampling result and the pre-calculation result to obtain query features corresponding to the data to be processed, and the query features are cached.

3. The device according to claim 2, characterized in that The query generation module is specifically used for: Perform multiplication and accumulation operations on the filtering result, the sampling result and the pre-calculation result to obtain an accumulation result; The query feature corresponding to the data to be processed is obtained according to the result of comparison performed by a comparison circuit according to the accumulated result, and the query feature is cached.

4. The device according to any one of claims 1 to 3, characterized in that: The pre-calculation module includes a pre-calculation engine and a pre-calculation buffer, and the pre-calculation module is specifically used for: The feature map of the data to be processed is multiplied according to the pre-calculation weights pre-stored in the pre-calculation engine to obtain the pre-calculation result, and the pre-calculation result is cached in the pre-calculation buffer.

5. The device according to claim 4, characterized in that The pre-calculation engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing units include a write driver, a multiplication logic circuit, an addition tree circuit and a storage array, the storage array pre-stores the first weight value of the pre-calculated weight; the pre-calculation module is specifically used for: Acquire first characteristic data of the characteristic graph of the data to be processed through the write driver; Performing a multiplication operation on the first feature data and the first weight value by the multiplication logic circuit to obtain a first product; Performing an addition operation on the first product by the addition tree circuit to obtain a first sum value; The first sum is accumulated and calculated by the accumulating unit to obtain a second sum, and the second sum is transmitted to the pre-calculation buffer for caching.

6. The device according to any one of claims 1 to 3, characterized in that: The sampling module includes a sampling engine and a sampling buffer. The sampling engine is used to calculate the corresponding offset and bilinear interpolation parameters according to the target query characteristics, and calculate the sampling result corresponding to the offset reference point based on the offset and the bilinear interpolation parameters, and transmit the sampling result to the sampling buffer for caching.

7. The device according to any one of claims 1 to 3, characterized in that: The attention weight module includes an attention engine and an attention buffer, the attention engine includes a plurality of storage-computation integrated arrays, the storage-computation integrated arrays include a function activation unit and a plurality of calculation units consisting of storage-computation integrated processing units and accumulation units, the storage-computation integrated processing unit includes a write driver, a multiplication logic circuit, an addition tree circuit and a storage array consisting of a plurality of storage elements, the storage array pre-stores a second weight value of the attention weight; the attention weight module is specifically used for: Acquire second feature data in the target query feature through the write driver; performing a multiplication operation on the second feature data and the second weight value by the multiplication logic circuit to obtain a second product; Performing an addition operation on the second product by the addition tree circuit to obtain a third sum value; Accumulating the third sum value by the accumulating unit to obtain a fourth sum value; The fourth sum value is input into the function activation unit to obtain an activated fourth sum value, and the activated fourth sum value is transmitted to the attention buffer for caching.

8. The device according to any one of claims 1 to 3, characterized in that: The data processing device further comprises a dynamic execution unit, which is connected to the query generation module; the dynamic execution unit is used to: Acquire the query features corresponding to the current data to be processed obtained by the query generation module; The similarity calculation circuit in the dynamic execution unit performs similarity calculation on the query feature corresponding to the current data to be processed and the previously obtained query feature to obtain a similarity value; The similarity value is compared with a preset similarity threshold by a comparison circuit in the dynamic execution unit to obtain a similarity comparison result, and the similarity comparison result is used to indicate whether the query feature corresponding to the current data to be processed is determined as the target query feature corresponding to data processing for the next data to be processed.

9. The device according to any one of claims 1 to 3, characterized in that: The data processing device also includes a fully connected layer calculation module, which is connected to the query generation module and is used to perform linear transformation calculation on the query features obtained by the query generation module to obtain linearly transformed query features.

10. A data processing method, characterized in that: The data processing method is applied to a storage-computation integrated device that uses a memory to calculate and store data, and the storage-computation integrated device is used to sequentially calculate feature graphs of multiple data to be processed to obtain respective corresponding query features, and the storage-computation integrated device includes a pre-calculation module, a sampling module, an attention weight module, and a query generation module; the method includes: Based on the pre-calculation module, weight transformation is performed on the feature map of the current data to be processed to obtain a pre-calculation result, and the pre-calculation result is cached; Sampling the target query feature based on the sampling module to obtain a sampling result, wherein the target query feature includes a query feature corresponding to any to-be-processed data before the current to-be-processed data calculated by the query generation module; Calculate the attention weight of the target query feature based on the attention weight module to obtain an attention result; Based on the query generation module, calculation is performed according to the attention result, the sampling result and the pre-calculation result to obtain query features corresponding to the data to be processed, and the query features are cached.

11. A vehicle, characterized in that: The vehicle includes the data processing device according to any one of claims 1 to 9, and the vehicle processes the data to be processed of the bird's-eye view model through the data processing device to obtain the query features of the bird's-eye view model.

Images