Convolution optimization operation method and device, computer device and storage medium

Abstract

This application relates to a convolution optimization operation method, apparatus, computer device, and storage medium. The method includes: acquiring feature map data in global memory; inputting the current original feature map into a DepthWise convolution model for a first core operation to obtain a first intermediate feature map, and loading the first intermediate feature map into shared memory; splitting the first intermediate feature map to obtain multiple second intermediate feature maps, and loading each second intermediate feature map into register memory; inputting each second intermediate feature map into a PointWise convolution model for a second core operation to obtain the current result feature map data, and loading the current result feature map data into global memory; and iteratively processing the next original feature map adjacent to the current original feature map as the current original feature map until all result feature map data is obtained. Using the embodiments of this application, separable convolution operations can be accelerated.

Description

Technical Field

[0001] This application relates to the field of neural network hardware acceleration technology, and in particular to a convolution optimization operation method, apparatus, computer device and storage medium. Background Technology

[0002] Recently, researchers have begun to focus on high-performance inference for deep neural networks, and have developed various acceleration engines and related components for different hardware, algorithms, and application scenarios to minimize the overall network runtime on the given hardware. NVIDIA GPUs are widely recognized as the hardware acceleration platform for efficiently running deep learning models, with hundreds and thousands of models deployed on GPUs to serve daily life and work. Currently, the acceleration of convolution operations on GPUs mainly relies on high-performance computing libraries such as CuDNN (CUDA Deep Neural Network) or CuBLAS (CUDA Basic Linear Algebra Subprograms). These libraries can merge multiple element-wise operators in a neural network into a large computing core, thereby reducing the number of hardware instructions issued and accelerating the running speed of the core on the GPU.

[0003] However, the inventors discovered during the implementation process that, compared to models dominated by separable convolution, existing high-performance computing libraries' optimization schemes are difficult to fully cover due to the finer-grained operators and fusion opportunities, resulting in lower applicability. Summary of the Invention

[0004] This application provides a convolution optimization operation method, apparatus, computer device, and storage medium, which can accelerate separable convolution operations.

[0005] Firstly, this application provides a convolution optimization operation method, including:

[0006] Retrieve feature map data from global memory; the feature map data includes multiple raw feature maps.

[0007] The current original feature map is input into the DepthWise convolutional model for the first core operation to obtain the first intermediate feature map, and the first intermediate feature map is loaded into shared memory;

[0008] The first intermediate feature map is split to obtain multiple second intermediate feature maps, and each second intermediate feature map is loaded into the register memory;

[0009] Each second intermediate feature map is input into the PointWise convolutional model for the second core operation to obtain the result feature map data, which is then loaded into global memory. The second core operation is performed by calling Tensor Core.

[0010] The next original feature map adjacent to the current original feature map is used as the current original feature map and processed cyclically until all the result feature map data is obtained.

[0011] In some embodiments, loading the first intermediate feature map into shared memory includes:

[0012] The memory addresses loaded from global memory into shared memory are arranged in alternating odd and even order to obtain the sorted memory addresses;

[0013] Store the first intermediate feature map into the corresponding sorted memory address.

[0014] In some embodiments, the first intermediate feature map is split to obtain multiple second intermediate feature maps, including:

[0015] Based on the width of the first intermediate feature map, the first intermediate feature map is split to obtain multiple second intermediate feature maps; each second intermediate feature map has the same number of channels as the first intermediate feature map.

[0016] The number of second intermediate feature maps is the result of the width of the first intermediate feature map being evenly divided.

[0017] In some embodiments, each second intermediate feature map is input into the PointWise convolutional model for a second core operation to obtain the resulting feature map data, including:

[0018] Obtain the data type of each second intermediate feature map;

[0019] Determine the smallest matrix multiplication and addition unit based on the data type;

[0020] The minimum matrix multiplication and addition unit is used to perform the second core operation on each second intermediate feature map to obtain the result feature map data.

[0021] In some embodiments, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained, including:

[0022] The number of processing steps required to obtain all the result feature map data is determined based on the number of original feature maps and the number of second intermediate feature maps.

[0023] Based on the number of processing iterations, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained.

[0024] In some embodiments, the number of processing steps required to obtain all the resulting feature map data is determined based on the number of original feature maps and the number of second intermediate feature maps, including:

[0025] Calculate the product of the number of original feature maps and the number of second intermediate feature maps. The product is the number of processing steps required to obtain all the resulting feature map data.

[0026] In some embodiments, the first core operation is performed by calling CUDA Core.

[0027] Secondly, this application also provides a convolution optimization operation apparatus, comprising:

[0028] The acquisition module is used to acquire feature map data in global memory; the feature map data includes multiple raw feature maps.

[0029] The first operation module is used to input the current original feature map into the DepthWise convolutional model to perform the first core operation, obtain the first intermediate feature map, and load the first intermediate feature map into shared memory;

[0030] The splitting module is used to split the first intermediate feature map to obtain multiple second intermediate feature maps, and load each second intermediate feature map into the register memory;

[0031] The second computation module is used to input each second intermediate feature map into the PointWise convolutional model for the second core computation to obtain the result feature map data for this operation, and load the result feature map data for this operation into global memory; the second core computation is completed by calling the Tensor Core.

[0032] The results module is used to iteratively process the next original feature map adjacent to the current original feature map as the current original feature map until all result feature map data is obtained.

[0033] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0034] Retrieve feature map data from global memory; the feature map data includes multiple raw feature maps.

[0035] The current original feature map is input into the DepthWise convolutional model for the first core operation to obtain the first intermediate feature map, and the first intermediate feature map is loaded into shared memory;

[0036] The first intermediate feature map is split to obtain multiple second intermediate feature maps, and each second intermediate feature map is loaded into the register memory;

[0037] Each second intermediate feature map is input into the PointWise convolutional model for the second core operation to obtain the result feature map data, which is then loaded into global memory. The second core operation is performed by calling Tensor Core.

[0038] The next original feature map adjacent to the current original feature map is used as the current original feature map and processed cyclically until all the result feature map data is obtained.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0040] Retrieve feature map data from global memory; the feature map data includes multiple raw feature maps.

[0041] The current original feature map is input into the DepthWise convolutional model for the first core operation to obtain the first intermediate feature map, and the first intermediate feature map is loaded into shared memory;

[0042] The first intermediate feature map is split to obtain multiple second intermediate feature maps, and each second intermediate feature map is loaded into the register memory;

[0043] Each second intermediate feature map is input into the PointWise convolutional model for the second core operation to obtain the result feature map data, which is then loaded into global memory. The second core operation is performed by calling Tensor Core.

[0044] The next original feature map adjacent to the current original feature map is used as the current original feature map and processed cyclically until all the result feature map data is obtained.

[0045] The aforementioned convolution optimization operation method, apparatus, computer device, and storage medium obtain the original feature map from global memory, input the current original feature map into the DepthWise convolution model for the first core operation to obtain the first intermediate feature map, load the first intermediate feature map into shared memory, then split the first intermediate feature map to obtain multiple second intermediate feature maps, load all second intermediate feature maps into register memory, and call Tensor Core to input the second intermediate feature maps into the PointWise convolution model for the second core operation to obtain the current result feature map data, which is then loaded into global memory. This process is repeated for the next original feature map until all result feature map data is obtained. This application fully utilizes the multi-level caching structure of the Turing architecture GPU, optimizes data reuse among global memory, shared memory, and register memory, and leverages Tensor Core for reuse among multiple convolution operators, thereby accelerating separable convolution operations. Attached Figure Description

[0046] Figure 1 A flowchart illustrating a convolution optimization operation method provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram illustrating the address order of memory blocks loaded from global memory into shared memory, provided in an embodiment of this application;

[0048] Figure 3 A schematic diagram illustrating the independent operation process and the fusion optimization operation process of DepthWise convolution and PointWise convolution provided in an embodiment of this application;

[0049] Figure 4 This application provides a schematic diagram illustrating the data reuse process in a multi-level cache structure of a Turing architecture GPU, as illustrated in an embodiment of the present application.

[0050] Figure 5 A structural block diagram of a convolution optimization computing device provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0052] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be more thorough and complete.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0054] Separable convolutions can be standalone depthwise convolutions, standalone pointwise convolutions, or convolutions that combine the two types of convolutions in some way. For example, a two-dimensional separable convolution (i.e., Separable Conv2D) is constructed by concatenating a 3x3 (kernel size) depthwise convolution and a 1x1 (kernel size) pointwise convolution, or it can be a standalone 3x3 depthwise convolution or a standalone 1x1 pointwise convolution.

[0055] Since DepthWise convolution and PointWise convolution are two independent processes, the memory communication cost between the two convolution operators is relatively high. Currently, the implementation of separable convolution on GPUs is accomplished by relying on CUDA (Compute Unified Device Architecture) Cores, and a fixed template has been designed for CUDA Cores to accelerate operators of specific sizes. Since the existing acceleration libraries only accelerate these two convolution operators separately, many intermediate results between these two operators cannot be optimally reused, and the multi-level cache structure of the GPU cannot be fully utilized, resulting in relatively low efficiency.

[0056] The convolution optimization operation method provided in this application can improve the reuse among multiple convolution operators, perform reasonable tuning among multi-level memory in Turing architecture GPUs, and leverage the powerful computing capabilities of Tensor Cores to accelerate the operation of separable convolutions.

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In some embodiments, such as Figure 1 As shown, a convolution optimization operation method is provided. This embodiment illustrates the application of this method to a computer device, and the method includes the following steps:

[0059] Step S110: Obtain feature map data in global memory; the feature map data includes multiple original feature maps;

[0060] Specifically, the scale of the feature map data includes height (represented by H), width (represented by W), and number of channels (represented by C). The feature map data can be decomposed into multiple original feature maps, each corresponding to a channel input into the DepthWise convolutional model. In some examples, the height of each original feature map output can be 1, in which case the scale of each original feature map is represented as 1*W0*C0, where the width W0 is the same as the width W of the feature map data, the number of channels C0 is the same as the number of channels C of the feature map data, and the number of original feature maps obtained from the decomposition is equal to the height H of the feature map data.

[0061] Step S120: Input the current original feature map into the DepthWise convolutional model for the first core operation to obtain the first intermediate feature map, and load the first intermediate feature map into shared memory;

[0062] Specifically, in the DepthWise convolution operation, convolution is performed layer by layer on each channel of the input feature map. This operator has a simpler computational cost and complexity compared to traditional 2D convolution, allowing the result of the DepthWise convolution to be stored in shared memory. In some examples, when an original feature map with a scale of 1*W0*C0 is input into the DepthWise convolution model for the first core operation, the resulting first intermediate feature map has a scale of 1*W1*C1, where the scale of the first intermediate feature map is consistent with that of the original feature map (i.e., W1 = W0, C1 = C0). Each operation can load the first intermediate feature map with a scale of 1*W1*C1 into shared memory, facilitating efficient use of shared memory.

[0063] In some embodiments, the first core operation is performed by calling CUDA Core.

[0064] Specifically, DepthWise convolution operations in Turing architecture GPUs rely on calls to CUDA Cores.

[0065] In some embodiments, loading the first intermediate feature map into shared memory includes:

[0066] The memory addresses loaded from global memory into shared memory are arranged in alternating odd and even order to obtain the sorted memory addresses;

[0067] Store the first intermediate feature map into the corresponding sorted memory address.

[0068] Specifically, in some examples, to achieve high bandwidth, shared memory can physically comprise 32 memory blocks of the same size, and the memory address corresponding to each memory block can be encoded starting from T0, such as... Figure 2 As shown, T0-T31 represent the memory addresses corresponding to each memory block. In Turing architecture GPUs, each memory block is 4 bytes wide. Since shared memory is mapped to the memory storage in a linear manner, a memory conflict occurs when multiple threads within the same thread bundle attempt to access data at different levels of the same storage. Therefore, during the process of loading the first intermediate feature map into shared memory, the data from global memory (i.e., ...) can be changed. Figure 2 The order in which memory blocks are loaded from main memory into shared memory is arranged in an alternating pattern of odd and even addresses. Figure 2 As shown, the first intermediate feature map can be stored in the corresponding arranged memory address.

[0069] Step S130: The first intermediate feature map is split to obtain multiple second intermediate feature maps, and each second intermediate feature map is loaded into the register memory;

[0070] In some embodiments, the first intermediate feature map is split to obtain multiple second intermediate feature maps, including:

[0071] Based on the width of the first intermediate feature map, the first intermediate feature map is split to obtain multiple second intermediate feature maps; each second intermediate feature map has the same number of channels as the first intermediate feature map.

[0072] The number of second intermediate feature maps is the result of the width of the first intermediate feature map being evenly divided.

[0073] Specifically, for example, a first intermediate feature map with a scale of 1*W1*C1 is input into a feature map splitting model for processing. The scale of each second intermediate feature map is represented as 1*W2*C2. In a specific example, if the width W1 of the first intermediate feature map is evenly divided into... If you can get a portion, you will receive There are two second intermediate feature maps, where the width W2 of the second intermediate feature map is... The number of channels C2 in each second intermediate feature map is the same as the number of channels C1 in the first intermediate feature map.

[0074] Step S140: Each second intermediate feature map is input into the PointWise convolutional model for the second core operation to obtain the result feature map data for this operation, and the result feature map data for this operation is loaded into global memory; the second core operation is completed by calling Tensor Core.

[0075] Specifically, in some examples, a second intermediate feature map with a scale of 1*W2*C2 is input into the PointWise convolutional model for the second core operation, where the PointWise convolutional kernel has a scale of 1*1*C. 卷积通道数 (where C) 卷积通道数 The value is consistent with the number of channels C2 of the second intermediate feature map; the second core operation is completed with the help of Tensor Core. Tensor Core is a type of operation core that is different from CUDA Core. Tensor Core is an operation unit that can perform matrix multiplication and addition operations simultaneously according to wmma instructions on architectures such as Volta, Turing and Ampere. It can complete more operations with fewer instructions and can perform excellent acceleration operations for specific bit bandwidths.

[0076] Once the second core operation is completed, the result feature map data can be obtained, and then the result feature map data can be moved into global memory through corresponding instructions.

[0077] In some embodiments, each second intermediate feature map is input into the PointWise convolutional model for a second core operation to obtain the resulting feature map data, including:

[0078] Obtain the data type of each second intermediate feature map;

[0079] Determine the smallest matrix multiplication and addition unit based on the data type;

[0080] The minimum matrix multiplication and addition unit is used to perform the second core operation on each second intermediate feature map to obtain the result feature map data.

[0081] Specifically, in Turing architecture GPUs, matrix multiplication and addition (MMA) instruction sets can be configured for matrix multiplication and addition (MMA) units. For different data types, the matrix multiplication and addition (MMA) units can include multiple configurations. For example, when the data type being operated on is a 16-bit floating-point number (FP16), there are two scales: 8*8*4 and 16*8*8. When the data type being operated on is an 8-bit integer (INT8), a 4-bit integer (INT4), or a 1-bit integer (INT1), there are three scales: 8*8*16, 8*8*32, and 8*8*128.

[0082] When performing PointWise convolution operations, in a specific instance, when the data type of the second intermediate feature map is FP16, the running delays (i.e., running times) corresponding to matrix multiplication and addition operation units of two scales, 8*8*4 and 16*8*8, are obtained respectively. Then, the matrix multiplication and addition operation unit with the shortest running delay is taken as the minimum matrix multiplication and addition operation unit, and the obtained minimum matrix multiplication and addition operation unit is used to perform the second core operation on each second intermediate feature map, thereby obtaining the result feature map data for this time.

[0083] Step S150: The next original feature map adjacent to the current original feature map is used as the current original feature map and processed cyclically until all result feature map data is obtained.

[0084] Specifically, the next original feature map adjacent to the current original feature map is used as the current original feature map for the first core operation, splitting and second core operation mentioned above, until all original feature maps are processed, and the fusion operation of DepthWise convolution and PointWise convolution is completed to obtain all result feature map data.

[0085] In some embodiments, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained, including:

[0086] The number of processing steps required to obtain all the result feature map data is determined based on the number of original feature maps and the number of second intermediate feature maps.

[0087] Based on the number of processing iterations, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained.

[0088] Specifically, the above convolution optimization operation involves DepthWise convolution and PointWise convolution. Each original feature map is input into the DepthWise convolution model for the first core operation, and each second intermediate feature map is input into the PointWise convolution model for the second core operation. Therefore, based on the number of original feature maps and the number of second intermediate feature maps, the number of processing steps required to obtain all the result feature map data after the first core operation and the second core operation can be obtained.

[0089] In some embodiments, the number of processing steps required to obtain all the resulting feature map data is determined based on the number of original feature maps and the number of second intermediate feature maps, including:

[0090] Calculate the product of the number of original feature maps and the number of second intermediate feature maps. The product is the number of processing steps required to obtain all the resulting feature map data.

[0091] Specifically, multiplying the number of original feature maps by the number of second intermediate feature maps yields the number of processing steps required to obtain all the resulting feature map data. In a specific example, the number of processing steps is... Second-rate.

[0092] To better illustrate the technical solution of this application, a specific example is provided below:

[0093] like Figure 3 As shown, without fusion optimization, the operation of 3x3 scale DepthWise convolution and 1x1 scale PointWise convolution are two independent processes. The DepthWise convolution process consists of the following steps: loading the DepthWise kernel (i.e., loading 1), loading the DepthWise input feature map (i.e., reading input feature map information 1), loading the DepthWise weights (i.e., reading weight information 1), the DepthWise core operation process (i.e., core operation 1), and loading the DepthWise convolution calculation result (i.e., storing calculation result 1). The PointWise convolution process consists of the following steps: loading the PointWise kernel (i.e., loading 2), loading the PointWise input feature map (i.e., reading input feature map information 2), loading the PointWise weights (i.e., reading weight information 2), the PointWise core operation process (i.e., core operation 2), and loading the PointWise convolution calculation result (i.e., storing calculation result 2). When the convolution optimization operation method provided in this application is used to fuse and optimize the DepthWise convolution and the PointWise convolution, the resources required for the subsequent PointWise convolution can be loaded during the calculation of the first DepthWise convolution. That is, the weight information 2 can be read during the core operation 1, thereby completing the fusion operation of the DepthWise convolution and the PointWise convolution.

[0094] In the process of accelerating the fusion of DepthWise convolution and PointWise convolution, this application utilizes the multi-level cache structure in the Turing architecture GPU, such as... Figure 4As shown, data is first retrieved from global memory, and the CUDA Core is called to complete the first core operation of the DepthWise convolution. Then, the result of the first core operation is loaded into shared memory, and the data in shared memory is decomposed. The result of the decomposition is loaded into register memory. Finally, the data in the register is used by the Tensor Core to perform the second core operation in each core of the PointWise convolution. The above process is repeated continuously until all convolution operators are completed. It should be noted that because the accelerated operators on the GPU belong to the heterogeneous computing stage, the host needs to request the computation and memory allocation, and then send the content to be computed to the device for execution. After the execution is completed, the computation memory is released, and finally the computation result is returned to the host.

[0095] The above-mentioned convolution optimization operation method achieves maximum running efficiency on Tensor Cores by efficiently applying DepthWise convolution and PointWise convolution to Turing architecture-based GPUs, parsing and reusing reusable results between the two types of operators, and accelerating the fusion with the help of the GPU's multi-level cache structure.

[0096] Furthermore, the technical solution of this application can also be applied to accelerate the computation of standard convolutions. Specifically, the standard convolution can be decomposed into a convolution consisting of cascaded DepthWise and PointWise convolutions using Network Decoupling technology. Network Decoupling is a technique that can convert a standard convolutional layer into a separable convolution consisting of a 3x3 DepthWise convolution and a 1x1 PointWise convolution without training, and the final accuracy loss is negligible. Furthermore, after the convolution decomposition, the above-mentioned convolution optimization operation can be applied to indirectly accelerate the standard convolution by accelerating the separable convolution.

[0097] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0098] Based on the same inventive concept, this application also provides a convolution optimization operation device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in the embodiments of the convolution optimization operation device provided below can be found in the limitations of the convolution optimization operation method above, and will not be repeated here.

[0099] In one embodiment, such as Figure 5 As shown, a convolution optimization computing device is provided, comprising:

[0100] The acquisition module 510 is used to acquire feature map data in global memory; the feature map data includes multiple original feature maps.

[0101] The first operation module 520 is used to input the current original feature map into the DepthWise convolutional model to perform the first core operation, obtain the first intermediate feature map, and load the first intermediate feature map into shared memory;

[0102] The splitting module 530 is used to split the first intermediate feature map to obtain multiple second intermediate feature maps, and load each second intermediate feature map into the register memory;

[0103] The second operation module 540 is used to input each second intermediate feature map into the PointWise convolutional model for the second core operation, obtain the result feature map data for this operation, and load the result feature map data for this operation into global memory; the second core operation is completed by calling Tensor Core.

[0104] The result module 550 is used to iteratively process the next original feature map adjacent to the current original feature map as the current original feature map until all result feature map data is obtained.

[0105] In some embodiments, the first computation module 520 is specifically configured to: load the first intermediate feature map into shared memory.

[0106] The memory addresses loaded from global memory into shared memory are arranged in alternating odd and even order to obtain the sorted memory addresses;

[0107] Store the first intermediate feature map into the corresponding sorted memory address.

[0108] In some embodiments, in splitting the first intermediate feature map to obtain multiple second intermediate feature maps, the splitting module 530 is specifically used for:

[0109] Based on the width of the first intermediate feature map, the first intermediate feature map is split to obtain multiple second intermediate feature maps; each second intermediate feature map has the same number of channels as the first intermediate feature map.

[0110] The number of second intermediate feature maps is the result of the width of the first intermediate feature map being evenly divided.

[0111] In some embodiments, in terms of inputting each second intermediate feature map into the PointWise convolutional model for second core operations to obtain the current result feature map data, the second operation module 540 is specifically used for:

[0112] Obtain the data type of each second intermediate feature map;

[0113] Determine the smallest matrix multiplication and addition unit based on the data type;

[0114] The minimum matrix multiplication and addition unit is used to perform the second core operation on each second intermediate feature map to obtain the result feature map data.

[0115] In some embodiments, the result module 550 is specifically used for iterative processing of the next original feature map adjacent to the current original feature map as the current original feature map until all result feature map data is obtained:

[0116] The number of processing steps required to obtain all the result feature map data is determined based on the number of original feature maps and the number of second intermediate feature maps.

[0117] Based on the number of processing iterations, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained.

[0118] In some embodiments, the result module 550 is specifically configured to: determine the number of processing steps required to obtain all the resulting feature map data based on the number of original feature maps and the number of second intermediate feature maps;

[0119] Calculate the product of the number of original feature maps and the number of second intermediate feature maps. The product is the number of processing steps required to obtain all the resulting feature map data.

[0120] In some embodiments, the first core operation is performed by calling CUDA Core.

[0121] Each module in the aforementioned convolution optimization computing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0122] In one embodiment, a computer device is also provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0123] In one embodiment, such as Figure 6 As shown, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A convolution optimization operation method, characterized in that, include: Acquire feature map data from global memory; the feature map data includes multiple original feature maps. The current original feature map is input into the DepthWise convolutional model for the first core operation to obtain the first intermediate feature map, and the first intermediate feature map is loaded into shared memory; The first intermediate feature map is split to obtain multiple second intermediate feature maps, and each second intermediate feature map is loaded into the register memory; Each of the second intermediate feature maps is input into the PointWise convolutional model for the second core operation to obtain the current result feature map data, and the current result feature map data is loaded into the global memory; the second core operation is performed by calling Tensor Core; The next original feature map adjacent to the current original feature map is used as the current original feature map and processed cyclically until all result feature map data is obtained.

2. The method according to claim 1, characterized in that, Loading the first intermediate feature map into shared memory includes: The memory addresses loaded from the global memory into the shared memory are arranged in alternating odd and even order to obtain the arranged memory addresses; The first intermediate feature map is stored in the corresponding arranged memory address.

3. The method according to claim 2, characterized in that, The process of splitting the first intermediate feature map to obtain multiple second intermediate feature maps includes: Based on the width of the first intermediate feature map, the first intermediate feature map is split to obtain multiple second intermediate feature maps; each second intermediate feature map has the same number of channels as the first intermediate feature map. The number of the second intermediate feature maps is the result of the width of the first intermediate feature map being evenly divided.

4. The method according to claim 1, characterized in that, The process involves inputting each of the second intermediate feature maps into the PointWise convolutional model for the second core operation to obtain the resulting feature map data, including: Obtain the data type of each of the second intermediate feature maps; Based on the data type, determine the smallest matrix multiplication and addition unit; The minimum matrix multiplication and addition operation unit is used to perform the second core operation on each of the second intermediate feature maps to obtain the result feature map data for this operation.

5. The method according to any one of claims 1 to 4, characterized in that, The step of taking the next adjacent original feature map as the current original feature map and performing iterative processing until all result feature map data is obtained includes: Based on the number of original feature maps and the number of second intermediate feature maps, determine the number of processing steps required to obtain all the resulting feature map data; Based on the number of processing iterations, the next original feature map adjacent to the current original feature map is used as the current original feature map for iterative processing until all result feature map data is obtained.

6. The method according to claim 5, characterized in that, The step of determining the number of processing steps required to obtain all the result feature map data based on the number of original feature maps and the number of second intermediate feature maps includes: Calculate the product of the number of the original feature maps and the number of the second intermediate feature maps, where the product represents the number of processing steps required to obtain all the resulting feature map data.

7. The method according to claim 1, characterized in that, The first core operation is performed by calling CUDA Core.

8. A convolution optimization computation device, characterized in that, include: The acquisition module is used to acquire feature map data in global memory; the feature map data includes multiple original feature maps. The first operation module is used to input the current original feature map into the DepthWise convolution model to perform the first core operation, obtain the first intermediate feature map, and load the first intermediate feature map into shared memory; The splitting module is used to split the first intermediate feature map to obtain multiple second intermediate feature maps, and load each second intermediate feature map into register memory; The second operation module is used to input each of the second intermediate feature maps into the PointWise convolution model for the second core operation, to obtain the current result feature map data, and to load the current result feature map data into the global memory; The second core operation is performed by calling the Tensor Core; The result module is used to iteratively process the next original feature map adjacent to the current original feature map as the current original feature map until all result feature map data is obtained.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images