A method for performing pooling calculation operations on a simulator

Abstract

An embodiment of the present invention provides a method for performing pooling calculation operations on a simulator, comprising: obtaining the scale of image data to be pooled and resource information of designated processing units on the simulator that can be used for this pooling, wherein the resource information includes at least the number of designated processing units; adjusting the scale of the image data according to the resource information of the designated processing units on the simulator that can be used for this pooling to obtain adjusted image data, so that the pooling calculation operations on the adjusted image data can be evenly distributed to the designated processing units; and evenly distributing the adjusted image data to the designated processing units on the simulator to perform the pooling calculation operations.

Description

Technical Field

[0001] The present invention relates to the field of computer architecture, in particular to the technical field of accelerating computing using simulators, and more particularly to a method for performing pooled computing operations on a simulator. Background Art

[0002] Simulators play a crucial role in the design and development of computer systems. In the early stages of design, simulators can be used to perform coarse-grained simulations of various design options, selecting the optimal design by comparing simulation results. During product development, simulators are used to evaluate various microarchitectural designs and make trade-offs. In the later stages of product development, simulators are primarily used for system software development of the target system, enabling simultaneous hardware and software development and accelerating system development. After the system is complete, simulators can generate rich trace information, enabling bottleneck analysis and performance optimization. Due to these important roles, a large number of simulators have been developed in both academia and industry. For example, a DPU (Deep Learning Processor Unit) simulator for deep learning applications has a hierarchical structure primarily consisting of a framework and a dynamic link library (DLL) for the DPU module. Application programmers primarily use upper-layer application-related data and instructions, including data to be processed, ARM programs, and data flow programs. The data and instructions related to the upper-level application are loaded into the memory when the DPU simulator is initialized. Then the ARM core starts to execute the ARM program from a specific address in the memory. During the execution of the ARM program, the microcontroller of the DMA and data flow array is configured to copy and execute data, results, and data flow programs to achieve the simulation purpose.

[0003] Pooling layers are the most common data processing layers in deep learning. They have a distinct purpose: reducing the size of feature maps, which in turn reduces computational effort and required video memory. Pooling layers reduce the size of spatial information, improving computational efficiency. This reduction in spatial information also reduces the number of parameters, reducing the risk of overfitting and ensuring spatial transformation invariance. Four common pooling operations are used: average pooling, max pooling, random pooling, and global average pooling. For example, average pooling is a pooling technique used in convolutional neural networks for image recognition. It works by sliding a window over a local region of features, such as pixels, and then taking the average of all values ​​in the window as the output. This compresses the input representation into a lower-dimensional representation.

[0004] In the process of implementing the pooling layer using the DPU simulator, the inventor discovered that the current implementation of the pooling layer for the DPU simulator is a direct allocation based on the original data format of the task to be assigned. Since the tasks to be assigned are not well planned according to the processing resources of the current DPU simulator, the processing resources are not fully utilized and / or the resources are unevenly distributed. For example, there is no reasonable data allocation method to mobilize all the processing units to work together to realize the computing function. For another example, the data functions allocated to each processing unit are not equal, resulting in some processing units working more and some processing units working less, and unbalanced resource allocation. Therefore, it is necessary to improve the existing technology. Summary of the Invention

[0005] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a method for performing pooling calculation operations on a simulator.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a method for performing a pooling calculation operation on a simulator is provided, comprising: obtaining the scale of image data to be pooled and resource information of designated processing units on the simulator that can be used for this pooling, wherein the resource information includes at least the number of designated processing units; adjusting the scale of the image data according to the resource information of the designated processing units on the simulator that can be used for this pooling to obtain adjusted image data, so that the pooling calculation operation on the adjusted image data can be evenly distributed to the designated processing units; and evenly distributing the adjusted image data to the designated processing units on the simulator to perform the pooling calculation operation.

[0008] In some embodiments of the present invention, the scale adjustment includes adjusting the pixel size or the number of channels of the image data so that the image data can be evenly divided by the designated processing units at the pixel level and / or the channel level.

[0009] In some embodiments of the present invention, the pixel size of the image data is adjusted by adding all-zero pixels to the pixel edges of the image data; or the number of channels of the image data is adjusted by adding all-zero channels to the image data.

[0010] In some embodiments of the present invention, the method further includes: determining whether the image data can be evenly divided by the designated processing unit at at least one of the pixel level and the channel level based on resource information of the designated processing unit on the simulator that can be used for this pooling; if not, rescaling the image data at at least one of the pixel level and the channel level.

[0011] In some embodiments of the present invention, the step of evenly distributing the adjusted image data to designated processing units on the simulator to perform pooling calculation operations includes: evenly distributing the adjusted image data to designated processing units on the simulator to perform pooling calculation operations at the adjusted scale level.

[0012] In some embodiments of the present invention, the resource information also includes the number of data used for one operation of the SIMD instruction for this pooling in the specified processing unit, wherein the scale adjustment includes: adding all-zero channels to the image data based on the resource information of the specified processing unit that can be used for this pooling on the simulator and the number of data used for one operation of the SIMD instruction for this pooling in the specified processing unit, so that the number of channels of the adjusted image data is an integer multiple of the product of the number of specified processing units and the number of data used for one operation of the SIMD instruction for this pooling.

[0013] In some embodiments of the present invention, the method further includes: determining, based on resource information of a designated processing unit on the simulator that can be used for this pooling, whether the number of channels of the image data is equal to the number of data operated on by the SIMD instruction used for this pooling at one time; and if so, rescaling the image data at the pixel level; and, at the pixel level, evenly distributing the adjusted image data to the designated processing units on the simulator to perform pooling calculation operations.

[0014] In some embodiments of the present invention, the method further includes: obtaining an intermediate pooling result obtained by performing a pooling calculation operation on the adjusted image data, and extracting a pooling result of the image data from the intermediate pooling result.

[0015] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions.

[0016] Compared with the prior art, the advantages of the present invention are:

[0017] This application obtains the scale of the image data to be pooled and the resource information of the designated processing units on the simulator that can be used for this pooling. The image data is rescaled based on the resource information of the designated processing units on the simulator that can be used for this pooling, so that the rescaled image data is evenly distributed to the designated processing units on the simulator to perform the pooling calculation operation. In this way, the available computing resources on the simulator are fully utilized to improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a flow chart of a method for performing a pooling calculation operation on a simulator according to an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the principle of scaling at the channel level according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of data arrangement of image data after resizing at the channel level in an on-chip memory of a simulator according to an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of constant transmission according to an example of the present invention;

[0023] Figure 5 A schematic diagram of a pooling process according to an example of the present invention;

[0024] Figure 6 Schematic diagram of the principle of scaling at the pixel level according to an example of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] As mentioned in the background technology section, in the process of implementing the pooling layer using the DPU simulator, the inventors found that the current implementation of the pooling layer for the DPU simulator is a direct allocation based on the original data format of the task to be assigned. Since the tasks to be assigned are not well planned according to the processing resources of the current DPU simulator, the processing resources are not fully utilized and / or the resources are unevenly distributed. Therefore, the present application obtains the scale of the image data that needs to be pooled and the resource information of the designated processing unit on the simulator that can be used for this pooling, and adjusts the scale of the image data according to the resource information of the designated processing unit on the simulator that can be used for this pooling, so as to evenly distribute the adjusted image data to the designated processing unit on the simulator to perform pooling calculation operations. In this way, the computing resources available on the simulator are fully utilized to improve resource utilization.

[0027] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained:

[0028] Image data refers to the intermediate output result (Feature Map) of the neural network, which needs to be pooled.

[0029] The scale of image data refers to the pixel scale and channel scale of image data. The pixel scale includes width and height, and the channel scale includes the number of channels.

[0030] According to one embodiment of the present invention, see Figure 1 , provides a method for performing pooling calculation operations on a simulator, including steps S1, S2, S3, and S4. In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0031] Step S1: obtaining the scale of the image data to be pooled and resource information of the designated processing units on the simulator that can be used for this pooling, wherein the resource information at least includes the number of designated processing units.

[0032] According to one embodiment of the present invention, assuming that a user is performing performance analysis on a designed neural network model, in order to complete the design work faster, the neural network model can be run on a general-purpose computer, and the general-purpose computer is connected to a simulator that executes the method of the present application to accelerate the operation process of the neural network model. For example, the general-purpose computer is configured to send at least the image data that needs to be pooled to the simulator for auxiliary processing, which involves data transmission and coordination between the DDR of the general-purpose computer and the on-chip memory SPM (Scratched Pad Memory) of the simulator. For example, the data scale (for example, transmission in bytes) and transmission method for transmission between the DDR and the on-chip memory SPM (Scratched Pad Memory) are configured by configuring the DMA channel, including the constants required for calculation, the input data from the DDR to the SPM (i.e., image data), and the output data from the SPM to the DDR (i.e., pooling results). It should be understood that the simulator of the present application can be a DPU (Deep Learning Processor Unit) simulator, a GPDPU (General Purpose Deep Learning Processor Unit) simulator, or any other simulator that can implement the method of the present application, and the present invention does not impose any limitation on this.

[0033] According to one embodiment of the present invention, in the image data to be pooled, the scale of the image data includes width, height, and number of channels. This application will subsequently express the scale of the image data in the form of: width × height × number of channels. For example, 13 × 13 × 128 represents image data with a width and height of 13 pixels and 128 channels. It should be understood that this is for illustrative purposes only and is not intended to limit the present invention in any way.

[0034] At present, the use of simulators is often to directly divide the image data that needs to be processed into multiple processing units, and the distribution of image data is not reasonably planned in advance according to the number of available processing units, resulting in uneven distribution of computing resources in the simulator and low resource utilization. In order to maximize the use of simulator resources, reduce the time for users to develop models, and improve resource utilization, according to one embodiment of the present invention, before allocating image data, the present invention first obtains the resource information of the designated processing unit (Process Element, abbreviated as PE) on the simulator that can be used for this pooling, so as to better distribute image data according to the resource information. Preferably, the resource information includes at least the number of designated processing units. For example, the number of designated processing units on the simulator that can be used for this pooling is 16, 32 or 128. It should be understood that this is only for illustration and the present invention does not impose any limitation on this.

[0035] In addition, it should be noted that the method of the present application is applicable to universal application fields, such as classification models in the medical field, recognition models in the target recognition field, etc. The pooling process of any neural network model that requires pooling can adopt the method of the present application.

[0036] Step S2: rescaling the image data according to the resource information of the designated processing unit on the simulator that can be used for this pooling to obtain the resized image data, so that the pooling calculation operation on the resized image data can be evenly distributed to the designated processing unit.

[0037] According to one embodiment of the present invention, the scale adjustment includes adjusting the pixel size or the number of channels of the image data so that the image data can be evenly divided by the designated processing unit at the pixel level and / or at the channel level. For example, the pixel size of the image data is adjusted by adding all-zero pixels at the pixel edges of the image data. For another example, the number of channels of the image data is adjusted by adding all-zero channels in the image data. Preferably, in the present application, the designated processing unit can be represented or replaced by a specified number of processing units. In order to ensure that the pooling results corresponding to the image data can be associated subsequently, a mapping relationship between the image data and the adjusted image data is established during the scale adjustment, and the mapping relationship includes an indication of the position of each element in the pooling result of the image data in the intermediate pooling result of the adjusted image data. For example, the mapping relationship is indicated by the offset (width, height and / or offset on the channel) of each element in the pooling result of the image data in the intermediate pooling result of the adjusted image data. For example, assuming that in the pooling result of the image data before adjustment, the position of an element (pixel) is at (width, height, channel) (2, 2, 9), and assuming that 24 all-zero channels are inserted between the original channels 8 and 9, and the scale of the inserted image data is adjusted, the offset of the element is (0, 0, 24), then the value of the element in the intermediate pooling result of the adjusted image data is (2, 2, 33), and the pooling value of the element can be extracted. In this way, the pooling values ​​of other elements can be extracted from the intermediate pooling result of the image data, thereby restoring the pooling result of the image data.

[0038] In some cases, the image data may just be able to be evenly divided. Therefore, in actual applications, it can be selectively processed according to actual conditions to improve resource utilization. According to one embodiment of the present invention, the method further includes: determining whether the image data can be evenly divided by the designated processing unit at at least one of the pixel level and the channel level based on the resource information of the designated processing unit that can be used for this pooling on the simulator; if not, resizing the image data at at least one of the pixel level and the channel level; if so, not actually resizing the image data. For example, when the image is resized, the supplementary numbers for width, height, and number of channels are all 0. According to one embodiment of the present invention, when the image data is resized at the channel level, supplementary all-zero channels are intermittently inserted into the original channels of the image data. Thereby, the original data of the image data is more evenly distributed.

[0039] According to one embodiment of the present invention, the scale adjustment can be performed based on the number of data used for the SIMD instruction of this pooling in a designated processing unit to improve resource utilization. According to one embodiment of the present invention, the resource information also includes the number of data used for the SIMD instruction of this pooling in a designated processing unit. The scale adjustment includes: based on the resource information of the designated processing unit available for this pooling on the simulator and the number of data used for the SIMD instruction of this pooling in a designated processing unit, adding all-zero channels to the image data so that the number of channels of the adjusted image data is an integer multiple of the product of the number of designated processing units and the number of data used for the SIMD instruction of this pooling. Preferably, the integer multiple is a minimum integer multiple. For example, if the number of specified processing units is 2, and the SIMD instruction used for this pooling in the specified processing unit is SIMD32, indicating that the number of data operated on at one time by SIMD32 is 32, then the product of the number of specified processing units and the number of data operated on at one time by the SIMD instruction used for this pooling is: 2×32=64; assuming that the original number of channels of the image data is 48, the number of channels is adjusted through the scale adjustment in step S2, and the adjusted image data has 64 channels.

[0040] According to one embodiment of the present invention, the number of channels of the image data may be exactly equal to the number of data that the SIMD instruction used for this pooling in the specified processing unit operates on at one time. In this case, each processing unit can parallelly calculate all channels of each image data at one time. However, if it is only assigned to one processing unit, the other processing units in the specified processing unit will be idle, resulting in low resource utilization. Therefore, according to one embodiment of the present invention, the method further includes: determining whether the number of channels of the image data is equal to the number of data that the SIMD instruction used for this pooling operates on at one time based on the resource information of the specified processing unit available for this pooling on the simulator, and if so, rescaling the image data at the pixel level. For example, if the SIMD instruction used for this pooling in the specified processing unit is SIMD32, indicating that the number of data that SIMD32 operates on at one time is 32, assuming that the scale of the image data is 55×55×32, the number of channels of the image data (i.e., 32) is exactly equal to the number of data that the SIMD instruction used for this pooling in the specified processing unit operates on at one time (i.e., 32). At this time, to further improve resource utilization, the width or height of the image data can be adjusted to evenly distribute the image data to each processing unit at the width or height level. For example, the image can be adjusted to 65×55×32.

[0041] Step S3: Evenly distribute the adjusted image data to designated processing units on the simulator to perform pooling calculation operations.

[0042] In order to utilize the resources of the simulator more efficiently, according to one embodiment of the present invention, at the adjusted scale level, the adjusted image data is evenly distributed to the processing units specified on the simulator to perform pooling calculation operations. For example, if step S2 is to adjust the number of channels, the adjusted image data is evenly distributed to the processing units specified on the simulator at the channel level to perform pooling calculation operations. For example, the adjusted image data of the aforementioned embodiment (adjusted from 48 channels to 64 channels) is assigned to one processing unit for every 32 channels. For another example, if step S2 is to adjust the pixels, the adjusted image data is evenly distributed to the processing units specified on the simulator at the pixel level to perform pooling calculation operations.

[0043] According to one embodiment of the present invention, the step of evenly distributing the adjusted image data to the designated processing units on the simulator to perform pooling calculation operations includes: allocating computing resources and storage resources for the adjusted image data on the designated processing units, performing pooling calculation operations on the adjusted image data in parallel through SIMD instructions, and obtaining intermediate pooling results.

[0044] Step S4: obtaining an intermediate pooling result obtained by performing a pooling calculation operation on the adjusted image data, and extracting a pooling result of the image data from the intermediate pooling result.

[0045] According to one embodiment of the present invention, an intermediate pooling result obtained by performing a pooling calculation operation on the adjusted image data by the simulator is obtained, and the pooling result of the image data is extracted from the intermediate pooling result according to the mapping relationship. According to the mapping relationship, the position of each element in the pooling result corresponding to the image data in the intermediate pooling result can be found, and data can be extracted therefrom to form the pooling result of the image data. The pooling result of the image data will be transmitted to a device that performs the next calculation task of the neural network model (for example: a general-purpose computer or other processing unit of the simulator).

[0046] The following two examples are given from the perspectives of channel-level and pixel-level scaling to better illustrate the technical principles of this application. It should be understood that the scales of image data that need to be pooled in the intermediate output of different neural network models are different. The following examples are only for illustration and the present invention does not impose any limitations on this.

[0047] Example 1 (channel-level rescaling):

[0048] 1. Pooling Implementation for 13×13×128 Graphs

[0049] First, take an image data of size 13×13×128 (i.e. scale) as an example. Assume that 16 computing components (corresponding to the designated processing units on the simulator that can be used for this pooling) are available for pooling, using SIMD32 instructions; the size of the pooling layer of the neural network model is a 3×3 window with a step size of 2. The output result obtained by calculation is 6×6×128. Three computing tasks are designed to be used on the simulator, and each computing task performs two rows of pooling operations at the same time. In addition, because all 16 computing components should be used as much as possible to improve resource utilization, by dividing the 128 channels into 16 parts, each PE can process 8 channels. Because SIMD components are used for processing, 32 operations can be performed simultaneously at a time, but 8 channels are not enough, so the data needs to be expanded. See Figure 2 By inserting 24 sets of 13×13 data with all zeros between every 8 channels (corresponding to the 24 channels), the 13×13×128 image is expanded to a 13×13×512 image. In the figure, the data in the form of (a,b) represents the index of the pixel point. For example, (1,3) represents a pixel with a height value of 1 and a width value of 3. The rest is similar and will not be repeated here. Although many zero values ​​(null values) are added to the original data, these additions do not affect the calculation speed. Instead, they make the data planning more reasonable and the data allocation to each processing unit is the same.

[0050] In the specific implementation, the above solution generates the adjusted image data in DDR in data_generate.c in the simulator on-chip memory data conversion module (spm_data), and then transfers it to the on-chip memory SPM (Scratched Pad Memory) through SIMD transmission in the test file testarm.c, and obtains the following Figure 3 The data arrangement in SPM is shown.

[0051] At the end of the data transmission of the experimental process of this example, the applicant designed the implementation of the kernel of the simulator. For example, the implemented kernel instantiated 4×4=16 logical nodes (Node, corresponding to the specified processing unit), and each node cyclically calculated to obtain 2 rows of values ​​in the output image after the pooling calculation operation. The above-mentioned logical nodes are mapped on a 4×4 PE array (whose processing unit is the specified processing unit) and executed by the PE array. Considering that the current PE execution unit is SIMD32, each SIMD branch calculates the pooling calculation operation of an image, and each task completes the pooling partial results of 32 input images each time (2 rows of result data in 32 output feature maps (Output Feature)). Each task of the currently implemented pooling calculation operation kernel can be composed of 2 subtasks (Subtasks). Assuming that the pooling is average pooling, the implementation of these two subtasks is described in detail below.

[0052] Subtask 1: Load data (load_copy, corresponding to LDN)

[0053] The main function of subtask 1 is to load the constant required for average pooling (i.e., the reciprocal of the divisor, such as the x3 pooling layer, the divisor is 9, and the reciprocal of the divisor is 1 / 9) into the local register of the PE array.

[0054] like Figure 4 As shown, subtask 1 consists of three types of instruction templates (inst_block templates), namely csv0, csv1, and csv2. The functions of the three types of instruction templates are as follows:

[0055] Csv0: Loads the reciprocal of the divisor stored in the SPM (assuming it is data a) and copies the data to the downstream node.

[0056] Csv1: Copies (COPY) the data obtained from the upstream node to the downstream node.

[0057] Csv2: Similar to a no-op (no null instruction, replaced by the addition instruction ADD, but no specific operation is performed, as long as the operation does not affect the value in the register), belonging to a leaf node.

[0058] Subtask 2: Load data and calculate the average value (ld_copy_compute)

[0059] Subtask 2 loads the data in a pooling window, adds and averages the data, and stores the resulting data back into the SPM. In this example, the loop iterates six times to produce one row of results. The number of loops is (input image width – pooling layer convolution kernel width) / stride + 1. The input image is the image data; the pooling layer convolution kernel width is the size of the pooling kernel used for size calculation. For example, the convolution kernel width of a 3×3 pooling layer is 3.

[0060] like Figure 5 As shown, subtask 2 consists of one instruction template (inst_block template), and the functions of the template are as follows:

[0061] Csv3: Loads data from a pooling window (corresponding to LDN), adds and averages the data (corresponding to CMPT), and stores the resulting data back into the SPM (corresponding to ST). The 16 processing elements (PEs) work in parallel. Each PE in a task can simultaneously calculate two rows of output results. In a single task, the 16 PEs generate two rows of output results from 128 channels, and all calculations can be completed with just three tasks.

[0062] Of course, it is necessary to set the data fetching position for each task and the position of different processing units in a task when fetching data. For example, the first task fetches data from the 1-5th row of image data from the top to the bottom, the second task fetches data from the 5-9th row of image data from the top to the bottom, and the third task fetches data from the 9-13th row of image data from the top to the bottom; processing units 1-16 fetch data from the 1-32nd, 33-64th, 65-96th, ..., 480-512th channel positions respectively. It is also necessary to set the data fetched in each cycle of the same processing unit in a task and the address where the calculation results are saved. That is, the address where each element of the intermediate pooling result is saved, such as the address yyyy, so that the pooling result of the image data can be restored later according to the mapping relationship. For example, assuming that the result of the element at position (2, 2, 33) in the intermediate pooling result of the adjusted image data is stored at address yyyy, and the corresponding offset is (0, 0, 24), then when restoring the value of the position (2, 2, 9) in the pooling result of the image data before adjustment, the pooling value of the element can be taken from yyyy. In this way, the pooling values ​​of other elements can be extracted from the intermediate pooling result of the image data, thereby restoring the pooling result of the image data.

[0063] After obtaining the calculated results in SPM, a simple single-channel transmission mode is used to transfer the results from SPM back to DDR. After the DPU simulator program completes execution, the ARM module is responsible for copying the 12MB of data starting at SPM_DDR_ADDR from the simulator DDR module (the copy size is temporarily fixed to 12MB and can be modified and adjusted according to actual conditions), generating the spu_out file. The simulator execution result data is extracted from this file and compared with the theoretical output results. If the results are correct, a result is obtained, such as the computing resource utilization of the pooling solution in Example 1 shown in Table 1.

[0064] Table 1

[0065] APP(0) Initialization time (575) Execution time (1538) Idle time (0) PE Computing resource utilization = 21.12% DMA First start time = 50194 DMA Final end time = 80848 DMA Total conversion time = 3875 Arm Total execution time = 78870 END TIME 86540 Over!

[0066] This example shows that for pooling of 13×13×128 image data, the resource utilization rate of the present invention's method can reach 21.12%. The resource utilization rate of the existing method used by the applicant for pooling image data of the same size is about 15%, so it can be seen that the resource utilization rate of the present invention's method is improved.

[0067] Example 2 (pixel-level resizing):

[0068] 2. Pooling Implementation of 55×55×32 Image

[0069] This example uses image data of size 55×55×32 as an example. Assume that 16 processing units (corresponding to the processing units designated for this pooling on the simulator) are available for pooling, using SIMD32 instructions. The pooling layer of the neural network model has a size of 3×3 and a stride of 2. The resulting output is 27×27×32. Since there are 32 channels, a single processing unit (PE) can perform computations on all channels in parallel. However, to maximize resource utilization, a balanced distribution can be achieved at the pixel level. However, since the 27×27×32 output is not easily evenly distributed across the 16 processing units, a more balanced output of 32×27×32 is considered. Based on this idea, a solution is to expand the original 55×55×32 data to 65×55×32. This data expansion method differs slightly from the previous one in that it expands the data on a per-channel basis (increasing the pixel size). To this end, the pixel size of the input image data can be expanded from 55×55 to 65×55, because (65-3) / 2+1=32, (55-3) / 2+1=27, and the excess part is padded with 0. Figure 6 shown.

[0070] Based on the idea of ​​aligning data to fill in blank values ​​and evenly dividing the processed data to each processing unit, the design uses one task. In one task, each PE performs two rows of pooling operations simultaneously. Because all 16 computing components must be used to improve resource utilization, SIMD instructions are used for processing. Therefore, 32 images can be directly processed at a time, looping 27 times. 16 tasks can directly complete all pooling operations.

[0071] The implementation of the simulator kernel is similar to that of Example 1. The only difference is the different offset addresses for storing and retrieving data for each PE within a task (i.e., the mapping relationship). This is not detailed here. Table 2 shows the computing resource utilization under the pooling scheme of Example 2.

[0072] Table 2

[0073] APP(0) Initialization time (279) Execution time (4438) Idle time (0) PE Computing resource utilization = 10.85% DMA First start time = 15893 DMA Last end time = 32590 DMA Total conversion time = 4785 Arm Total execution time = 27071 END TIME 38300 Over!

[0074] This example shows that for pooling calculations on 55×55×32 image data, the resource utilization rate of the present invention's method can reach 10.85%. In contrast, the resource utilization rate of the applicant's existing method for the same-sized image is less than 10%. Although resource utilization rates vary for different image sizes, for images of the same size, the present invention's solution has improved component utilization, as can be seen in the results of running the solution on the simulator.

[0075] Based on the above two examples, this application achieves the improvement of resource utilization of pooling operations on 55×55×32 graphs and 13×13×128 graphs on the simulator through two data expansion methods. However, this method can be extended to the use of graphs of any size. This application has contributed to the efficient calculation of the pooling layer on actual hardware.

[0076] The above two examples mainly reproduce and organize the average pooling layer, propose a pooling scheme for data alignment and filling in gaps, and a scheme for rationally allocating data to processing units. The main function of the tasks (underlying library functions) written is to complete the addition and averaging operations. However, it should be understood that other pooling methods such as maximum pooling and minimum pooling can also be implemented based on the above examples with slight adjustments, and the present invention does not impose any restrictions on this.

[0077] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0078] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0079] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0080] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for performing a pooling calculation operation on a simulator, characterized in that: include: Obtaining the scale of the image data to be pooled and resource information of the designated processing units on the simulator that can be used for this pooling, wherein the simulator is a DPU simulator or a GPDPU simulator, and the resource information includes at least the number of designated processing units and the amount of data in the designated processing units that can be used for a single SIMD instruction operation for this pooling; The image data is scaled according to the resource information of the designated processing unit that can be used for this pooling on the simulator to obtain the adjusted image data, so that the pooling calculation operation on the adjusted image data can be evenly distributed to the designated processing unit, wherein the scale adjustment includes: according to the resource information of the designated processing unit that can be used for this pooling on the simulator and the number of data used for the SIMD instruction of this pooling in one operation, adding all-zero channels to the image data so that the number of channels of the adjusted image data is an integer multiple of the product of the number of designated processing units and the number of data used for the SIMD instruction of this pooling; the method also includes: according to the resource information of the designated processing unit that can be used for this pooling on the simulator, determining whether the number of channels of the image data is equal to the number of data used for the SIMD instruction of this pooling in one operation, and if so, scaling the image data at the pixel level to evenly distribute the image data to each processing unit at the width or height level, and evenly distributing the adjusted image data to the designated processing units on the simulator at the pixel level to perform the pooling calculation operation; The adjusted image data is evenly distributed to the designated processing units on the simulator to perform pooling calculation operations.

2. The method according to claim 1, characterized in that The scale adjustment includes adjusting the pixel size or the number of channels of the image data so that the image data can be evenly divided by the designated processing units at the pixel level and / or the channel level.

3. The method according to claim 2, characterized in that The pixel size of the image data is adjusted by adding all-zero pixels at the pixel edges of the image data; or the number of channels of the image data is adjusted by adding all-zero channels in the image data.

4. The method according to claim 2, characterized in that The method further comprises: Based on the resource information of the designated processing unit that can be used for this pooling on the simulator, determine whether the image data can be evenly divided by the designated processing unit at at least one of the pixel level and the channel level; if not, resize the image data at at least one of the pixel level and the channel level.

5. The method according to claim 4, characterized in that The step of evenly distributing the adjusted image data to designated processing units on the simulator to perform pooling calculation operations includes: At the adjusted scale level, the adjusted image data is evenly distributed to the designated processing units on the simulator to perform pooling calculation operations.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtain an intermediate pooling result obtained by performing a pooling calculation operation on the adjusted image data, and extract a pooling result of the image data from the intermediate pooling result.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 6 by executing the executable instructions.

Images