Data processing methods, apparatus, storage media and electronic devices

Abstract

This application provides a data processing method, apparatus, storage medium, and electronic device. This application can directly obtain resident data from the on-chip cache without having to read the input data from the off-chip storage into the on-chip cache for a second calculation. Furthermore, since the bandwidth required to read data from the on-chip cache is much greater than the bandwidth required to read data from the off-chip storage, it is not necessary to load data from the off-chip storage multiple times, which is beneficial to improving computing performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] During the training of deep neural networks, due to the complexity of the computation process or the need for multiple calculations, data often needs to be read multiple times. For example, BatchNorm is an important regularization technique that can make the intermediate results between network layers more stable and speed up the convergence of the training process. However, when using BatchNorm for forward or backward computation, input data needs to be read multiple times.

[0003] However, with multiple data reads, multiple off-chip loadings are also required to ensure the smooth completion of the entire computation process, which will degrade the overall computing performance. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, and electronic device. By storing data in an on-chip cache, it eliminates the need to load data from off-chip storage multiple times, thereby improving overall computing performance.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a computing device, the computing device including an on-chip cache, the computing device being connected to off-chip storage, the off-chip storage containing multiple input data, the method comprising:

[0006] The multiple input data are read into the on-chip cache and a first calculation is performed to obtain a first calculation result;

[0007] The multiple input data reside in the on-chip cache;

[0008] A second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result to obtain the calculation result of the multiple input data.

[0009] Secondly, embodiments of this application provide a computing device, the device including: an on-chip cache and a processor, the device being connected to off-chip storage, the off-chip storage having a plurality of input data;

[0010] The processor is configured to read the plurality of input data into the on-chip cache and perform a first calculation to obtain a first calculation result;

[0011] The on-chip cache is used to store the plurality of input data; the processor is also used to perform a second calculation on the plurality of input data in the on-chip cache and the first calculation result to obtain the calculation result of the plurality of input data.

[0012] Thirdly, embodiments of this application provide a combined processing apparatus, which includes the computing device as described in the third aspect.

[0013] Fourthly, embodiments of this application provide an electronic device, which includes a memory, a processor, a communication bus, and a communication interface, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and when the processor executes the program stored in the memory, it implements some or all of the steps described in the first aspect above.

[0014] Fifthly, embodiments of this application provide a computer-readable storage medium, the computer-readable storage medium including a computer program for data exchange, the computer program being executed by a processor to implement some or all of the steps described in the first aspect of embodiments of this application.

[0015] Sixthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0016] This application provides a data processing method, apparatus, storage medium, and electronic device that can read multiple input data into an on-chip cache and perform a first calculation to obtain a first calculation result; store the multiple input data in the on-chip cache; and read the multiple input data and the first calculation result from the on-chip cache to perform a second calculation to obtain a calculation result for the multiple input data. Thus, during the second calculation, the stored data can be directly retrieved from the on-chip cache, eliminating the need to read the input data from external storage into the on-chip cache again for the second calculation. Furthermore, since the bandwidth required to read data from the on-chip cache is much greater than the bandwidth required to read data from external storage, multiple loadings from external storage are unnecessary, which improves computational performance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of a circuit board provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of a combined processing device provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the internal structure of a computing device provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the internal structure of a processor core provided in an embodiment of this application;

[0022] Figure 5a This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0023] Figure 5b This is a schematic diagram of an on-chip cache structure provided in an embodiment of this application;

[0024] Figure 5c This is a schematic diagram of the structure of multidimensional tensor data provided in an embodiment of this application;

[0025] Figure 6 This is a functional unit block diagram of a computing device provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] The present application will now be described in detail with reference to specific embodiments.

[0030] Please see Figure 1, Figure 1 This is a schematic diagram of the structure of a board 10 provided in an embodiment of this application, as shown below. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0031] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0032] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0033] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and off-chip storage 204.

[0034] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform relevant calculations in the deep learning or machine learning training process. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0035] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, processing device 203 can obtain input data from off-chip storage, and then computing device 201 can obtain input data from processing device 203 via interface device 202, or computing device 201 can directly obtain input data from off-chip storage 204, and perform a first calculation on the input data through the processor in computing device 201 to obtain a first calculation result, which can be written to the on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them into the on-chip control cache of computing device 201. According to the control commands, computing device 201 stores the input data in the on-chip control cache of computing device 201 (i.e., the on-chip cache described in the embodiments of this application). Finally, the processor in computing device 201 can perform a second calculation on the input data and the first calculation result to obtain the second calculation result corresponding to the input data. Alternatively or optionally, the interface device 202 may also read data such as the first calculation result, the second calculation result, and input data from the storage device of the computing device 201 and transmit them to the processing device 203.

[0036] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0037] The off-chip storage 204 is used to store input data. It can be DDR memory, typically 16G or larger, and is used to store data of the computing device 201 and / or other processing devices. The data can be input data or first calculation results that cannot be fully stored in the internal or on-chip storage of the computing device 201 or other processing devices.

[0038] Figure 3 A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple clusters, and each cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0039] From the perspective of system-on-a-chip hierarchy, such as Figure 3 As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0040] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2 The computing device 201 is connected to the off-chip storage 204, allowing it to read or write multiple input data from or to the external storage. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305, enabling the transmission of different input data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201; four are shown exemplary in the figure. With hardware development, the computing device 201 disclosed herein may also include eight, sixteen, sixty-four, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0041] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0042] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43, which is an on-chip cache. The processor core 306 can be used to read multiple input data from off-chip storage 204 to the storage module 43 for a first calculation and obtain a first calculation result. It can also store multiple input data in the storage module and store the first calculation result in the storage module 43. Alternatively, it can be used to perform a second calculation on the multiple input data and the first calculation result stored in the storage module 43 and obtain a calculation result for the multiple input data.

[0043] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0044] The computation module 42 includes a vector computation unit 421 and a matrix computation unit 422. The vector computation unit 421 performs vector operations, supporting complex operations such as vector multiplication, addition, and nonlinear transformations. The matrix computation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution. This computation module is used to perform the aforementioned first and / or second computations.

[0045] The storage module 43 is used to store or move related data (e.g., multiple input data read from the off-chip cache 204, or the first calculation result obtained by the processor core 306 performing the first calculation, or the second calculation result obtained by the processor core 306 performing the second calculation, etc.), including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access (IODMA) module 433, and a move direct memory access (MVDMA) module 434. NRAM 431 is used to store multiple input data for the processor core 306 to use for the first calculation, and / or the first calculation result and multiple input data for the second calculation, etc.; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and off-chip memory 204 through broadcast bus 309; MVDMA 434 is used to control the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0046] It should be noted that, in the embodiments of this application, the above-mentioned storage module 43 may include one or more on-chip caches (e.g., NRAM, WRAM, etc.) described in the embodiments of this application.

[0047] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data (e.g., multiple input data) or intermediate results (e.g., a first calculation result, a second calculation result, etc.) among the processor cores 306, and for executing communication between the cluster 305 and the off-chip storage 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0048] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from the off-chip storage 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0049] Furthermore, in this embodiment, the SRAM308 can also be used as an on-chip cache to store multiple input data, first calculation results, etc.

[0050] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between clusters 305 and off-chip memory 204, respectively. These will be explained below.

[0051] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0052] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201. Figure 3 This diagram illustrates the operation of CDMA310 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0053] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0054] In one embodiment, return Figure 3 The GDMA 311 works in conjunction with the external memory controller 301 to control accesses from the on-chip cache SRAM 308 of the cluster 305 to the off-chip memory 204, or to read input data from the off-chip memory 204 into the SRAM 308. As described above, communication between the off-chip memory 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is through IODAM 433 directly connecting the off-chip memory 204 with the on-chip cache NRAM 431 or WRAM 432; the second channel involves first using GDMA 311 to transfer data between the off-chip memory 204 and the on-chip cache SRAM 308, and then using MVDMA 434 to transfer input data between the on-chip cache SRAM 308 and the on-chip cache NRAM 431 or WRAM 432, thus achieving the residency of input data. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between off-chip storage 204 and on-chip cache NRAM 431 or on-chip cache WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein can select the data transmission channel based on their hardware capabilities.

[0055] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure.

[0056] Please see Figure 5a , Figure 5aThis is a flowchart illustrating a data processing method provided in an embodiment of this application, applied to the aforementioned computing device 201. The computing device includes an on-chip cache (corresponding to...). Figure 4 The computing device 201 is connected to the external storage 204 (storage module 43), and the external storage 204 has multiple input data, such as... Figure 5a As shown, the method includes the following steps:

[0057] S510. Read the multiple input data into the on-chip cache and perform a first calculation to obtain a first calculation result.

[0058] When processing data, the input data is first read from off-chip storage into the on-chip cache, and then the input data in the on-chip cache is calculated.

[0059] The aforementioned input data can be image data, voice data, video data, etc., depending on the different application scenarios, and this application does not impose any restrictions on this. Optionally, the data can be one-dimensional or multi-dimensional tensor data. When the data is image data, the input data can be four-dimensional tensor data x[n,h,w,c], where N represents the number, C represents the image channels, H represents the height, and W represents the width.

[0060] The aforementioned first calculation may refer to the initial calculation process in which the computing device performs calculations on multiple input data. For some calculations, it is necessary to read the input data multiple times for calculation, such as normalize and BatchNorm calculations, which require two calculations on the input data. The first calculation refers to the first calculation performed on the input data.

[0061] S520. The plurality of input data resides in the on-chip cache.

[0062] Since the bandwidth for a computing device to read data from off-chip storage is much smaller than the bandwidth for reading data from on-chip cache, the aforementioned multiple input data can reside in the on-chip cache. This helps reduce the amount of data read from off-chip storage during subsequent computations, thereby reducing the amount of input data loaded. In this embodiment, "residence" refers to reading input data from off-chip storage and storing it in the on-chip cache, maintaining its original state and preventing modification until the data becomes invalid.

[0063] S530: Read multiple input data and the first calculation result from the on-chip cache and perform a second calculation to obtain the calculation result of the multiple input data.

[0064] For certain calculations that require reading input data multiple times, such as normalize and BatchNorm, the input data needs to be read twice for the second calculation. The second calculation requires the input data and the result of the first calculation as input for the second calculation.

[0065] To better understand the solutions of the embodiments of this application, the following uses BatchNorm calculation as an example to introduce the embodiments of this application.

[0066] In the training of deep neural networks, BatchNorm is an important regularization technique that makes the intermediate results between network layers more stable and accelerates the convergence speed of the training process. During deep learning network training, for a 4-dimensional tensor X of (N, H, W, C), BatchNorm includes forward and backward operations. The forward calculation of BatchNorm is as follows:

[0067]

[0068]

[0069]

[0070] The reverse calculation of BatchNorm is as follows:

[0071]

[0072]

[0073]

[0074] Where N represents the quantity, C represents the image channel, H represents the height, and W represents the width; M = N * H * W, representing the total number of elements in each image channel in a batch of data (called a mini-batch); γ c It can refer to the scaling variable, β c It can refer to the scaling variable, γ c and β c The normalized result can be scaled or shifted to a certain extent; μ c It can refer to the mean value obtained with respect to the input data x[n,h,w,c]. It can refer to the variance obtained with respect to the input data x[n,h,w,c]; ∈ can refer to a coefficient close to 0 to prevent... The entire denominator is 0. It can refer to the gradient at x[n,h,w,c].

[0075] For forward computation, as shown in (1) to (3) above, y[n,h,w,c] = batchnorm_forward(x[n,h,w,c],gamma[c],beta[c]). During the computation process, x[n,h,w,c] needs to be loaded onto the chip twice. The first time is used to calculate the mean and variance of each channel, and the second time is used to calculate the value of the element at the corresponding position of the output tensor.

[0076] If this application is used for BatchNorm calculation, the BatchNorm calculation includes forward calculation, and the first calculation result includes: the mean and variance of the multiple input data.

[0077] Specifically, during the BatchNorm forward computation process, the processor first reads the input data from off-chip memory into the on-chip cache. Then, it performs a first calculation on the read input data, namely, calculating the mean and variance. This input data is then stored in the on-chip cache. Next, the processor reads the multiple input data stored in the on-chip cache, along with the mean and variance obtained from the first calculation. It calculates the difference between the input data and the mean, and then normalizes the input data by dividing the difference by the square of the variance and the square root of the coefficient ε. Finally, it uses γ... c and β c The normalized result is scaled or shifted to a certain extent to complete the entire second calculation process and obtain the calculation results corresponding to all the input data in the off-chip storage.

[0078] During the forward computation of BatchNorm, after calculating the mean and variance of the input data (i.e., the first computation), the input data resides in the on-chip cache. When performing normalization computation on the input data (i.e., the second computation), the input data can be directly retrieved from the on-chip cache for computation, without having to read the data from off-chip storage into the on-chip cache again, thus improving computational performance.

[0079] Regarding the reverse calculation, as can be seen from the above reverse formula, we know from the calculation formula that diff_x[n,h,w,c]=batchnorm_backward(x[n,h,w,c],diff_y[n,h,w,c],gamma[c],beta[c]). During the calculation process, x[n,h,w,c] also needs to be loaded onto the chip twice. The first time is used to calculate the gradients at gamma[c] and beta[c], and the second time is used to calculate the gradient at x[n,h,w,c].

[0080] If this application is used for BatchNorm calculation, the BatchNorm calculation further includes inverse calculation, and the first calculation result includes: a first differential result and a second differential result.

[0081] Specifically, during the BatchNorm reverse computation process, the processor first reads the input data from off-chip memory into on-chip memory. This input data includes two parts: input tensor data and output error data (mean and variance) from the forward computation process. Then, the processor performs a first calculation on the read input data, which corresponds to the first and second differential calculations in formulas (5) and (6) above, respectively, to obtain the first and second differential results corresponding to multiple input data. The input data is then stored in the on-chip cache. Next, the processor can first read the multiple input data stored in the on-chip cache and the parameter γ obtained through the first calculation. c The first differential result and the result with respect to parameter β c The second differential result, wherein the aforementioned input data may include input tensor data and data such as mean and variance from the forward computation process, and then, based on the differential operation performed on the input tensor data, the gradient at the input tensor data is obtained. This is to complete the entire second calculation process and obtain the calculation results corresponding to all the input data in the off-chip storage. The second calculation may refer to the calculation described in the above formula (6) for the input data and its corresponding parameter γ. c The first differential result and with respect to parameter β c The second calculation result is a series of calculation processes.

[0082] In the reverse computation process of BatchNorm, after differentiating the input data (i.e., the first computation), the input data resides in the on-chip cache. When performing the second computation on the input data, the input data can be directly retrieved from the on-chip cache for computation, without having to read the data from off-chip storage into the on-chip cache again, thus improving computational performance.

[0083] As mentioned above, whether BatchNorm performs forward or reverse computation, the input tensor x[n,h,w,c] needs to be used twice during the computation process. Therefore, it needs to be loaded from off-chip memory to the on-chip cache twice. Compared to existing technologies, in the prior art, the time consumed by BatchNorm forward computation at optimal computational resource performance is the sum of the time required to read the input tensor x[n,h,w,c] into the on-chip cache twice and to write the output tensor Y once; while the time consumed by BatchNorm reverse computation at optimal computational resource performance is the sum of the time required to read the input tensor x[n,h,w,c] and Diffff_Y twice and to write the input tensor Y once. Thus, the need to load from off-chip memory to the on-chip cache multiple times results in poor computational performance of BatchNorm. This application improves computational performance by storing the input data in an on-chip cache after the first calculation. When the input data needs to be read again, it is not necessary to read the input data from off-chip memory again, but can directly read the stored input data from the on-chip cache.

[0084] It should be noted that the current example only considers two calculation steps (first calculation and second calculation). When three or more calculation steps are involved, the method described in this application is equally applicable. For example, if three calculation steps are involved (which may include first calculation, second calculation, and third calculation, and the three calculations have the same input data), after the first calculation is completed, multiple input data can be stored in on-chip storage. During the second and third calculations, the stored multiple input data can be directly read from the on-chip cache to obtain the final desired calculation result.

[0085] As can be seen, in this application, when performing subsequent calculations on multiple input data, multiple input data can be read from the on-chip cache. Compared with reading data from off-chip storage, the bandwidth of reading data from the on-chip cache is much greater than that of reading data from off-chip storage, which helps to reduce the amount of data that needs to be read from off-chip storage when the calculation results are obtained.

[0086] In one possible example, the cache space size corresponding to the on-chip cache is the first cache space; the required cache space size corresponding to the multiple input data is the second cache space; before residing the multiple input data in the on-chip cache, the above method further includes the following steps: determining whether the first cache space is smaller than the second cache space; if the first cache space is smaller than the second cache space, then residing a portion of the input data with the same size as the first cache space in the on-chip cache.

[0087] Specifically, since the on-chip cache has limited storage space, or it may contain other input data besides the multiple input data used in this calculation, or the amount of multiple input data is too large, it is necessary to determine whether to store all or part of the multiple input data in the on-chip cache. When the amount of input data is large and the on-chip cache space is insufficient to store multiple input data, some of the multiple input data can be stored in the on-chip cache, and the remaining input data can be stored in off-chip storage; when the on-chip cache space is sufficient to store the aforementioned multiple input data, all input data can be stored in the on-chip cache.

[0088] In other words, by comparing the sizes of the first cache space and the second cache space, it can be determined whether to keep all the multiple input data in the on-chip cache. If the first cache space is smaller than the second cache space, it means that there is not enough storage space in the on-chip cache to store the multiple input data. The input data with the same size as the first cache space will be kept in the on-chip cache, and the remaining input data will be left in off-chip storage. Conversely, if the first cache space is greater than or equal to the second cache space, it means that there is enough storage space in the on-chip storage to temporarily store the multiple input data. All the multiple input data will be kept in the on-chip cache.

[0089] As can be seen, in this application, multiple input data can be allocated according to the size of the input data and the storage situation of the on-chip cache. All or part of the above multiple input data can reside in the on-chip cache, which is beneficial to improve data residency efficiency and success rate. This avoids the situation where all input data is read from off-chip storage but the on-chip cache cannot store it, and helps to reduce the amount of input data loaded in such cases.

[0090] Furthermore, when the multiple input data residing in the on-chip cache constitute all the input data, that is, the storage space of the on-chip cache is sufficient to store all the input data in the off-chip cache, the processor directly reads all the residing input data from the on-chip cache, performs a second calculation with the first calculation result, and obtains the final calculation result of the multiple input data.

[0091] In one possible example, if not all the input data is required for the second calculation, then when selecting input data to reside in the on-chip cache, only the portion of the input data that needs to be calculated a second time can be selected to reside in the on-chip cache. The specific selection method depends on the calculation method applicable to the embodiments of this application and is not limited here.

[0092] When the multiple input data residing in the on-chip cache are only partial input data, meaning the remaining input data can be stored in off-chip storage, the multiple input data can be read from the on-chip cache and off-chip storage according to the order in which the multiple input data reside in the multiple cache spaces and the cache spaces in off-chip storage. Then, combined with the first calculation result, a subsequent second calculation is performed to obtain the calculation result corresponding to the multiple input data.

[0093] The above method may further include the following steps: performing a second calculation on the portion of input data residing in the on-chip cache and the first calculation result to obtain a second calculation result; reading the remaining input data (excluding the portion of input data residing in the on-chip cache) from the off-chip memory and performing a second calculation on the first calculation result to obtain a third calculation result; and obtaining the calculation result of the plurality of input data based on the second calculation result and the third calculation result.

[0094] It should be noted that, in this embodiment of the application, the on-chip cache has insufficient storage space to store all the multiple input data. Therefore, some of the input data can be stored in the on-chip cache.

[0095] For example, if the method described in the embodiments of this application is applied to the forward calculation of BatchNorm, then the second calculation mentioned above may refer to the calculation process described in the above formula (1) of normalizing the input data and its mean and variance; in this case, the processor can first read part of the input data resident in the on-chip cache and calculate the difference between the part of the input data and the mean, and then normalize the part of the input data by dividing the difference between the part of the input data and the mean by the square root of the variance and the coefficient ε, and then normalize the part of the input data by γ. c and β c The normalized result is scaled or shifted by a certain margin to obtain a second calculation result corresponding to a portion of the input data. Then, the remaining input data stored in off-chip memory is read into the on-chip cache and subjected to a series of calculations as described above, along with the mean and variance obtained in the first calculation, to achieve normalization of the remaining input data. Furthermore, through γ... c and β c The normalized result is scaled or shifted by a certain margin to obtain the third calculation result corresponding to the remaining input data. Finally, the second and third calculation results can be concatenated to obtain the calculation results corresponding to all input data (multiple input data).

[0096] For example, if the method described in the embodiments of this application is applied to the reverse calculation of BatchNorm, the processing method is the same as that corresponding to the forward calculation of BatchNorm, and will not be repeated here.

[0097] In one possible example, when concatenating the second and third calculation results as described above, the following steps may be included: determining a first storage order corresponding to a portion of the input data residing in one or more cache spaces in the on-chip cache; determining a second storage order between the portion of the input data and the remaining input data; and concatenating the second calculation result corresponding to the portion of the input data and the third calculation result corresponding to the remaining input data according to the second storage order to obtain the calculation result of multiple input data.

[0098] Specifically, the second storage order can be used to determine which data is preceding and which is following the partial input data. If the partial input data is preceding the partial input data, the second calculation result is appended to the preceding position of the third calculation result; otherwise, the second calculation result is appended to the following position of the third calculation result.

[0099] In one possible example, on-chip caching may include one or more on-chip cache spaces, which can be, for example, Figure 3 One or more of the on-chip caches such as NRAM, SRAM, and WRAM shown can also be other on-chip cache spaces, and this application does not limit this.

[0100] For example, such as Figure 5b The diagram shown is a schematic of an on-chip cache, which may include one SRAM, four NRAMs and four WRAMs.

[0101] When the on-chip cache includes multiple cache spaces, multiple input data and a first calculation result are read from the on-chip cache to perform a second calculation, so as to obtain the calculation result of the multiple input data, including:

[0102] The order in which the multiple input data resides in the multiple cache spaces is determined; the multiple input data in the on-chip cache and the first calculation result are read according to the order to perform a second calculation to obtain the calculation result of the multiple input data.

[0103] Specifically, the order in which input data resides in the on-chip cache space can be arbitrary, or it can be stored sequentially in multiple on-chip cache spaces according to a pre-defined order. This pre-defined order can be determined by the size of the on-chip cache space, or it can be user-defined or a system default; there are no restrictions here. For example, input data can be stored in a front-to-back order, or linked using pointers, etc.

[0104] Since the input data may be correlated or ordered in pairs, when reading multiple input data from the on-chip cache, they can be read according to their order of residence in the on-chip cache. Specifically, if all the input data involved in the second calculation resides in the on-chip cache, the order in which the multiple input data resides in the multiple cache spaces corresponding to the multiple on-chip caches can be determined. The input data is then read from the on-chip cache according to this order and combined with the first calculation result to perform the second calculation, thus obtaining the calculation result for the multiple input data. This approach preserves the order of the multiple input data, which helps improve the accuracy of the calculation result.

[0105] In one possible example, when the on-chip cache includes NRAM, input data is preferentially stored in NRAM. Specifically, due to the different calling methods of computation instructions (e.g., instructions for calling BatchNorm computation), when computation is performed on input data, if data needs to be read from the on-chip cache, it is often necessary to move or allocate input data stored in other on-chip caches (e.g., SRAM and / or WRAM) to NRAM before the computation instructions corresponding to the actual computation can be called through NRAM. Therefore, when storing input data in the on-chip cache, it is often necessary to prioritize storing input data in the NRAM cache space. Only when the NRAM cache space is insufficient to store all input data will the input data be stored in other on-chip cache spaces.

[0106] In a specific implementation, if the on-chip cache includes NRAM, SRAM, and WRAM, then the input data read from the off-chip storage will first reside in NRAM. Only when the NRAM cache space is insufficient to store all the input data will the input data be stored in SRAM and / or WRAM.

[0107] As can be seen, in this application, when the NRAM cache space is insufficient to store all the input data, the input data can be stored in other on-chip cache spaces. In this way, when the input data needs to be read later, the input data residing in the SRAM and / or WRAM can be read into the NRAM. The bandwidth of reading from one on-chip cache to another on-chip cache is much greater than the bandwidth of reading data from off-chip storage, which is beneficial to improving data reading efficiency.

[0108] In one possible example, if the on-chip cache includes NRAM, SRAM, and WRAM, the following steps can be included when storing the multiple input data in the on-chip cache: Since the input data needs to be retrieved from NRAM when performing subsequent calculations on the input data, and the actual cache space of NRAM is unknown, the maximum NRAM cache space that can be stored when no data is stored in NRAM can be determined; if the NRAM cache space is greater than or equal to the second cache space, then all multiple input data are stored in NRAM; if the NRAM cache space is less than the second cache space, then after storing the input data in NRAM, the remaining input data are stored in SRAM and / or WRAM.

[0109] As can be seen, in this application, the dwelling method of multiple input data can be planned according to the calling method of the calculation instructions in the specific calculation process and the subsequent specific calculation method, which helps to reduce the failure of dwelling data; in subsequent calculations, it helps to reduce the occurrence of input data reading errors caused by the calling method of calculation instructions.

[0110] In one possible embodiment, taking BatchNorm computation as an example, if the on-chip cache includes at least two of NRAM, SRAM, and WRAM, i.e., includes multiple cache spaces, and if the instruction used to call BatchNorm computation needs to be called through NRAM, then the input data residing in SRAM and / or WRAM can be read and concatenated with the input data residing in NRAM according to the first storage order described above to obtain partial input data, which is then residing in NRAM; then, the subsequent second computation can be performed as described above, etc., which will not be elaborated here.

[0111] In one possible embodiment, if the on-chip cache includes NRAM and SRAM, and if the instruction for calling BatchNorm calculation needs to be called through NRAM, if the current cache space in NRAM is insufficient to store the input data residing in SRAM, then second input data of the same size as the current cache space of SRAM can be read from SRAM. A third storage order between the second input data and the third input data residing in SRAM (excluding the second input data) is recorded, and this third input data is concatenated with the first input data in NRAM according to the first storage order. This concatenated first and second input data can then be used in a second calculation with the first calculation result to obtain a fourth calculation result. Furthermore, the remaining third input data residing in SRAM can be moved or transferred to NRAM and used in a second calculation with the first calculation result to obtain a fifth calculation result. Finally, the fourth and fifth calculation results are concatenated according to the third storage order to obtain the second result data corresponding to the aforementioned partial input data.

[0112] It should be noted that the number of on-chip caches mentioned above is only an example. If there are multiple on-chip caches, the above method can also be used, which will not be elaborated here.

[0113] As can be seen, in this application, regardless of the number of on-chip storage devices, the second calculation can be performed first. After the calculation is completed, the calculation results are concatenated according to the storage order of the data to obtain the final calculation result. Furthermore, since the bandwidth required for data transfer or movement between multiple on-chip caches is much greater than the bandwidth required for reading from off-chip storage, the calculation efficiency is higher than that from off-chip storage when performing calculations on multiple input data, as it is not necessary to read data from off-chip storage. This is beneficial for improving computing performance.

[0114] In one possible example, when reading the remaining input data from off-chip storage, the offset of the remaining input data in all the data in the off-chip storage can be determined; and the starting position of the remaining input data in the off-chip storage, i.e. the first input data in the remaining input data, can be determined based on the offset; and the remaining input data can be read from the off-chip storage from this starting position until the position corresponding to the last input data of the remaining input data.

[0115] In one possible example, the input data is multidimensional tensor data, including four dimensions: N, H, W, and C. When the amount of multidimensional tensor data is greater than or equal to a preset threshold, the following steps may also be included: splitting the multidimensional tensor data into multiple segments along the C dimension, and then using the computing device to perform calculations on each segment of the split multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data.

[0116] In this approach, dimension C is considered a low dimension, while the other three dimensions N, H, and W are treated as high dimensions without distinction. Since dimension C exists independently during computation, it can be split into its own dimensions.

[0117] The aforementioned preset threshold can be set by the user or be a system default, and is not limited here. For example, the C-dimensional input data can be split according to its size. When the input data is very large, the bandwidth required for loading increases, affecting computational efficiency. This preset threshold can be used to define small-scale and large-scale data; data exceeding this preset threshold can be defined as large-scale data. For example, the preset threshold can be defined as 10M; multiple input data exceeding 10M can be defined as large-scale data, and data less than 10M can be defined as small-scale data. Optionally, the input data can also be split according to the computational unit, dividing the C-dimensional input data according to the data size that the computational unit can process, which can prevent the waste of computational units, and so on.

[0118] It should be noted that in this embodiment of the application, the main focus is on large-scale input data. The above-mentioned multidimensional data may include multidimensional tensor data, such as tensor data X in formulas (1)-(6) above. The multidimensional tensor data includes four dimensions, namely, NHWC four dimensions.

[0119] In practice, since the tensor data is multi-dimensional, if the method described in the above embodiments is used, most of the data may not be able to reside in the on-chip cache. Therefore, the multi-dimensional tensor data can be split into multiple segments.

[0120] Furthermore, such as Figure 5cThe diagram shows the structure of multidimensional tensor data. As shown, the input data is multidimensional tensor data. Taking BatchNorm calculation as an example, the process of splitting the input data is explained. Since the entire BatchNorm calculation is independent in the C dimension, each segment of multidimensional tensor data after splitting can be iteratively calculated in the C dimension, as shown in the diagram. Each Task corresponds to one BatchNorm calculation. The overall data is split into multiple segments in the C dimension (corresponding to Job1, Job2, ..., Jobn in the diagram). Before each second calculation, this segment of multidimensional tensor data can reside in an on-chip cache. Then, calculations can be performed in the C dimension using a iterative calculation method (the specific calculation process for each segment is consistent with that described in methods S510-S530 above) (i.e., Task1, Task2, ..., Taskn in the diagram). Finally, the results can be concatenated in the C dimension to obtain the calculation result corresponding to the entire BatchNorm calculation.

[0121] As can be seen, in this application, a large number of multidimensional tensors can be split along the C dimension to reduce the data processing process, i.e., the process of reading multidimensional tensor data from off-chip storage, which is beneficial to improving the overall data retention rate.

[0122] In one possible example, the above method may further include the following steps: determining the jump step size and loading segment length of the split based on the data bandwidth; determining an adjustment coefficient based on the jump step size and the loading segment length; and determining the number of segments of the multidimensional data split in the C dimension based on the adjustment coefficient.

[0123] In particular, considering that splitting multidimensional tensor data along the C dimension will reduce the bandwidth for subsequent data loading, it is necessary to find a balance point that optimizes overall computational performance.

[0124] Here, the aforementioned jump step size S can refer to the number of jumps required between the first segment and the next segment, and the aforementioned loading segment length can refer to the segment length corresponding to loading a segment of multidimensional tensor data for calculation.

[0125] In the specific implementation, the above adjustment coefficient m can refer to the quotient between the jump step size S and the loading segment length L, that is, the adjustment coefficient m = jump step size S / loading segment length L. When the adjustment coefficient m is larger, the bandwidth of the loading data will be smaller. Therefore, the bandwidth of the above loading data can be tested according to the adjustment coefficient to determine a balance point, that is, to determine the number of segments of the multidimensional tensor data in the C dimension, so as to achieve the best overall performance.

[0126] As can be seen, the data processing method described in this application embodiment can read multiple input data into the on-chip cache and perform a first calculation to obtain a first calculation result; store the multiple input data in the on-chip cache; and read the multiple input data and the first calculation result from the on-chip cache to perform a second calculation to obtain a calculation result for the multiple input data. Thus, when performing the second calculation, the stored data can be directly retrieved from the on-chip cache, eliminating the need to read the input data from external storage into the on-chip cache again for the second calculation. Furthermore, since the bandwidth required to read data from the on-chip cache is much greater than the bandwidth required to read data from external storage, multiple loadings from external storage are unnecessary, which is beneficial for improving computational performance.

[0127] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0129] Please see Figure 6 , Figure 6 This is a functional unit block diagram of a computing device 600 provided in an embodiment of this application, such as... Figure 6 As shown, the computing device 600 includes an on-chip cache 610 and a processor 620, the device being connected to off-chip memory, the off-chip memory containing multiple input data;

[0130] The processor 620 is configured to read the plurality of input data into the on-chip cache and perform a first calculation to obtain a first calculation result;

[0131] The on-chip cache 610 is used to store the plurality of input data; the processor 620 is also used to perform a second calculation on the plurality of input data in the on-chip cache and the first calculation result to obtain the calculation result of the plurality of input data.

[0132] As can be seen, this application provides a computing device that can read multiple input data into the on-chip cache and perform a first calculation to obtain a first calculation result; store the multiple input data in the on-chip cache; and read the multiple input data and the first calculation result from the on-chip cache to perform a second calculation to obtain a calculation result for the multiple input data. Thus, when performing the second calculation, the stored data can be directly retrieved from the on-chip cache, eliminating the need to read the input data from external storage into the on-chip cache again for the second calculation. Furthermore, since the bandwidth required to read data from the on-chip cache is much greater than the bandwidth required to read data from external storage, multiple loadings from external storage are unnecessary, which is beneficial for improving computing performance.

[0133] Optionally, the on-chip cache includes multiple cache spaces, and the second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result to obtain the calculation result of the multiple input data; the processor 610 is specifically used for:

[0134] Determine the order in which the multiple input data reside in the multiple cache spaces;

[0135] The second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result in the order described, so as to obtain the calculation result of the multiple input data.

[0136] Optionally, the cache space size corresponding to the on-chip cache is a first cache space; the required cache space size corresponding to the multiple input data is a second cache space; before storing the multiple input data in the on-chip cache, the processor 610 is further configured to:

[0137] Determine whether the first cache space is smaller than the second cache space;

[0138] If the first cache space is smaller than the second cache space, then a portion of the input data of the same size as the first cache space will reside in the on-chip cache.

[0139] Optionally, the processor 610 is further configured to:

[0140] A second calculation is performed on the portion of input data residing in the on-chip cache and the first calculation result to obtain a second calculation result;

[0141] The remaining input data, excluding the portion of input data residing in the on-chip cache, is read from the off-chip memory, and the first calculation result is used to perform a second calculation to obtain a third calculation result;

[0142] The calculation results of the multiple input data are obtained based on the second calculation result and the third calculation result.

[0143] Optionally, the input data is multidimensional tensor data, including four dimensions: N, H, W, and C;

[0144] When the amount of the multidimensional tensor data is greater than or equal to a preset threshold, the processor 610 is further configured to:

[0145] The multidimensional tensor data is split into multiple segments along the C dimension, and the computing device is used to perform calculations on each segment of the multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data.

[0146] Optionally, the processor 610 is further configured to:

[0147] The jump step size and loading segment length are determined based on the data bandwidth;

[0148] The adjustment coefficient is determined based on the jump step size and the loading segment length;

[0149] Based on the adjustment coefficient, the number of segments in the C-dimensional tensor data is determined.

[0150] It is understood that the functions of each program module of the computing device in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0151] This application also provides a computer storage medium that stores a computer program for data processing, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0152] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. This computer program product can be a software installation package.

[0153] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7As shown, the electronic device includes a memory, input devices, output devices, and a processor. The electronic device may also include a communication bus, allowing the processor, input devices, output devices, and memory to be interconnected via the bus. Optionally, the electronic device may further include an instruction storage unit located adjacent to the processor. More preferably, the instruction storage unit is integrated with the processor, meaning it is an on-chip memory unit of the processor. Thus, when the processor needs to execute a program from memory, the electronic device first loads the program from memory into the instruction storage unit. Then, the processor can access the instruction storage unit to execute the program within it.

[0154] When the processor executes the program stored in the memory, it performs the following steps:

[0155] The multiple input data are read into the on-chip cache and a first calculation is performed to obtain a first calculation result;

[0156] The multiple input data reside in the on-chip cache;

[0157] A second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result to obtain the calculation result of the multiple input data.

[0158] Furthermore, the aforementioned processor can be a Central Processing Unit (CPU), an Intelligence Processing Unit (NPU), a Graphics Processing Unit (GPU), or an Image Processing Unit; this application does not limit this to any particular processor. Depending on the processor, the method for determining hardware performance proposed in this application can be applied to artificial intelligence application fields such as image recognition processing, deep learning processing, computer vision processing, intelligent robot processing, and natural language processing, executing complex functional programs in the field of artificial intelligence.

[0159] Furthermore, depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, computing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0160] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0161] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0162] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0163] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0164] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0165] The foregoing can be better understood in accordance with the following terms:

[0166] Clause A1. A data processing method applied to a computing device, the computing device including an on-chip cache, the computing device being connected to off-chip memory, the off-chip memory having a plurality of input data, including:

[0167] The multiple input data are read into the on-chip cache and a first calculation is performed to obtain a first calculation result;

[0168] The multiple input data reside in the on-chip cache;

[0169] A second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result to obtain the calculation result of the multiple input data.

[0170] A2. According to the method described in A1, the on-chip cache includes multiple cache spaces, and the step of reading multiple input data from the on-chip cache and the first calculation result to perform a second calculation to obtain the calculation result of the multiple input data includes:

[0171] Determine the order in which the multiple input data reside in the multiple cache spaces;

[0172] The second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result in the order described, so as to obtain the calculation result of the multiple input data.

[0173] A3. According to the method described in A1 or A2, the cache space size corresponding to the on-chip cache is the first cache space; the required cache space size corresponding to the multiple input data is the second cache space; before residing the multiple input data in the on-chip cache, the method further includes:

[0174] Determine whether the first cache space is smaller than the second cache space;

[0175] If the first cache space is smaller than the second cache space, then a portion of the input data of the same size as the first cache space will reside in the on-chip cache.

[0176] A4. The method described in A3 further includes:

[0177] A second calculation is performed on the portion of input data residing in the on-chip cache and the first calculation result to obtain a second calculation result;

[0178] The remaining input data, excluding the portion of input data residing in the on-chip cache, is read from the off-chip memory, and the first calculation result is used to perform a second calculation to obtain a third calculation result;

[0179] The calculation results of the multiple input data are obtained based on the second calculation result and the third calculation result.

[0180] A5. According to the method described in A1 or A2, the input data is multidimensional tensor data, including four dimensions: N, H, W, and C;

[0181] When the amount of data in the multidimensional tensor is greater than or equal to a preset threshold, the method further includes:

[0182] The multidimensional tensor data is split into multiple segments along the C dimension, and the computing device is used to perform calculations on each segment of the multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data.

[0183] A6. The method described in A5 further includes:

[0184] The jump step size and loading segment length are determined based on the data bandwidth;

[0185] The adjustment coefficient is determined based on the jump step size and the loading segment length;

[0186] Based on the adjustment coefficient, the number of segments in the C-dimensional tensor data is determined.

[0187] A7. The method according to any one of A1-A6, wherein the on-chip cache includes one or more of NRAM, SRAM, and WRAM.

[0188] A8. The method according to A1 further includes: the method is used for BatchNorm calculation, the BatchNorm calculation includes forward calculation; the first calculation result includes: the mean of the plurality of input data and the variance of the plurality of input data.

[0189] A9. According to the method described in A1-A8, the method is used for BatchNorm calculation, the BatchNorm calculation includes reverse calculation; the first calculation result includes: a first differential result and a second differential result.

[0190] A10. A computing device, the device comprising: an on-chip cache and a processor, the device being connected to off-chip memory, the off-chip memory having a plurality of input data;

[0191] The processor is configured to read the plurality of input data into the on-chip cache and perform a first calculation to obtain a first calculation result;

[0192] The on-chip cache is used to store the plurality of input data; the processor is also used to perform a second calculation on the plurality of input data in the on-chip cache and the first calculation result to obtain the calculation result of the plurality of input data.

[0193] A11. According to the apparatus described in A10, the step of reading multiple input data from the on-chip cache and performing a second calculation on the first calculation result to obtain a calculation result of the multiple input data; the processor is specifically used for:

[0194] Determine the order in which the multiple input data reside in the on-chip cache;

[0195] The second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result in the order described, so as to obtain the calculation result of the multiple input data.

[0196] A12. In the apparatus according to A10 or A11, the cache space size corresponding to the on-chip cache is a first cache space; the required cache space size corresponding to the plurality of input data is a second cache space; before residing the plurality of input data in the on-chip cache, the processor is further configured to:

[0197] Determine whether the first cache space is smaller than the second cache space;

[0198] If the first cache space is smaller than the second cache space, then a portion of the input data of the same size as the first cache space will reside in the on-chip cache.

[0199] A13. According to the apparatus of A12, the processor is further configured to:

[0200] A second calculation is performed on the portion of input data residing in the on-chip cache and the first calculation result to obtain a second calculation result;

[0201] The remaining input data, excluding the portion of input data residing in the on-chip cache, is read from the off-chip memory, and the first calculation result is used to perform a second calculation to obtain a third calculation result;

[0202] The calculation results of the multiple input data are obtained based on the second calculation result and the third calculation result.

[0203] A14. According to the apparatus described in A10 or A11, the input data is multidimensional tensor data, including four dimensions: N, H, W, and C;

[0204] When the amount of the multidimensional tensor data is greater than or equal to a preset threshold, the processor is further configured to:

[0205] The multidimensional tensor data is split into multiple segments along the C dimension, and the computing device is used to perform calculations on each segment of the multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data.

[0206] A15. According to the apparatus of A14, the processor is further configured to:

[0207] The jump step size and loading segment length are determined based on the data bandwidth;

[0208] The adjustment coefficient is determined based on the jump step size and the loading segment length;

[0209] Based on the adjustment coefficient, the number of segments in the C-dimensional tensor data is determined.

[0210] B1. A neural network chip comprising a method for performing any one of the terms A1-A9.

[0211] C1. A computer-readable storage medium comprising storing a computer program for data exchange, wherein the computer program, when executed by a processor, implements the method as described in any one of clauses A1-A9.

[0212] D1. An electronic device, comprising:

[0213] processor;

[0214] Memory used to store processor-executable instructions;

[0215] The processor is configured to invoke instructions stored in the memory to perform the method described in any one of clauses A1-A9.

[0216] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that the embodiments described above are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A data processing method, characterized in that, Applied to a computing device, the computing device including an on-chip cache, the computing device being connected to off-chip memory, the off-chip memory having multiple input data, the method includes: The multiple input data are read into the on-chip cache and a first calculation is performed to obtain a first calculation result; The multiple input data reside in the on-chip cache; Read multiple input data from the on-chip cache and the first calculation result to perform a second calculation to obtain the calculation result of the multiple input data; The input data is multidimensional tensor data, including four dimensions: N, H, W, and C. When the amount of multidimensional tensor data is greater than or equal to a preset threshold, the method further includes: The multidimensional tensor data is split into multiple segments along the C dimension, and the computing device is used to calculate each segment of the multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data. The method further includes: The jump step size and loading segment length are determined based on the data bandwidth; The adjustment coefficient is determined based on the jump step size and the loading segment length; Based on the adjustment coefficient, the number of segments in the C-dimensional tensor data is determined.

2. The method according to claim 1, characterized in that, The on-chip cache includes multiple cache spaces. The step of reading multiple input data from the on-chip cache and performing a second calculation on the first calculation result to obtain the calculation result of the multiple input data includes: Determine the order in which the multiple input data reside in the multiple cache spaces; The second calculation is performed by reading multiple input data from the on-chip cache and the first calculation result in the order described, so as to obtain the calculation result of the multiple input data.

3. The method according to claim 1 or 2, characterized in that, The cache space size corresponding to the on-chip cache is the first cache space; the required cache space size corresponding to the multiple input data is the second cache space. Before residing the plurality of input data in the on-chip cache, the method further includes: Determine whether the first cache space is smaller than the second cache space; If the first cache space is smaller than the second cache space, then a portion of the input data of the same size as the first cache space will reside in the on-chip cache.

4. The method according to claim 3, characterized in that, The method further includes: A second calculation is performed on the portion of input data residing in the on-chip cache and the first calculation result to obtain a second calculation result; The remaining input data, excluding the portion of input data residing in the on-chip cache, is read from the off-chip storage, and the first calculation result is used to perform a second calculation to obtain a third calculation result; The calculation results of the multiple input data are obtained based on the second calculation result and the third calculation result.

5. The method according to claim 1, characterized in that, The on-chip cache includes one or more of NRAM, SRAM, and WRAM.

6. The method according to claim 5, characterized in that, The method is used for BatchNorm calculation, which includes forward calculation; the first calculation result includes: the mean and variance of the multiple input data.

7. The method according to claim 6, characterized in that, The method is used for BatchNorm calculation, which includes inverse calculation; the first calculation result includes: a first differential result and a second differential result.

8. A computing device, characterized in that, The device includes: an on-chip cache and a processor, the device being connected to off-chip memory, the off-chip memory having multiple input data; The processor is configured to read the plurality of input data into the on-chip cache and perform a first calculation to obtain a first calculation result; The on-chip cache is used to store the plurality of input data; the processor is also used to perform a second calculation on the plurality of input data in the on-chip cache and the first calculation result to obtain the calculation result of the plurality of input data; The input data is multidimensional tensor data, including four dimensions: N, H, W, and C; when the amount of the multidimensional tensor data is greater than or equal to a preset threshold, the processor is further configured to: The multidimensional tensor data is split into multiple segments along the C dimension, and the computing device is used to calculate each segment of the multidimensional tensor data to obtain the calculation result corresponding to the multidimensional tensor data. The processor is further specifically used for: The jump step size and loading segment length are determined based on the data bandwidth; The adjustment coefficient is determined based on the jump step size and the loading segment length; Based on the adjustment coefficient, the number of segments in the C-dimensional tensor data is determined.

9. A combined processing apparatus, characterized in that, The device includes the computing device as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program for data exchange, which, when executed by a processor, implements the method as described in any one of claims 1-7.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method described in any one of claims 1-7.

Images