Method and apparatus for accelerating dense 3D packet convolution

Abstract

The invention relates to a method and apparatus for accelerating dense three-dimensional (3D) packet convolution. The method comprises the following steps: receiving an input feature map; generating a grouping label matrix of the input feature map, wherein the grouping label matrix is used for dividing the input feature map into a plurality of disjoint input groups; concurrently performing a convolution operation on the plurality of disjoint input packets based on the packet tag matrix; and directly generating an output feature map corresponding to the input feature map using the convolution results of the plurality of disjoint input packets.

Description

Technical Field

[0001] The present application relates to the field of neural networks, and more specifically, to a method and device for accelerating dense 3D grouped convolution. Background Art

[0002] Due to the popularity of deep learning, 3D convolutional neural networks (3D CNNs) are becoming the mainstream solution for 3D visual recognition tasks (e.g., INTEL® RealSense TM and MICROSOFT KINECT). However, in stark contrast to traditional 2D CNNs, 3D CNNs need to consider an additional spatial dimension in mathematical operations, resulting in a cubic growth problem in computational and memory requirements.

[0003] The RISC-V instruction set architecture (ISA) is now widely used in artificial intelligence (AI), such as Google, Meta, T-Head, Sophgo, etc. The main feature of the RISC-V ISA for AI is the RISC-V vector instructions (RVV), which is a highly flexible vector data processing instruction extension. RVV supports binary portability between cores from different vendors. RVV provides various types of arithmetic calculations and load / store operations, thus achieving high efficiency in AI workloads including convolution operations. Summary of the invention

[0004] The present application provides a new mechanism for accelerating dense 3D grouped convolutions and corresponding instructions, which linearly reduces the computational cost of convolution operations of 3D convolutional neural networks and realizes hardware acceleration of dense 3D grouped convolutions using RISC-V.

[0005] According to an embodiment of the present disclosure, a method for accelerating dense 3D grouped convolution is provided, comprising: receiving an input feature map; generating a grouped label matrix of the input feature map, the grouped label matrix being used to divide the input feature map into a plurality of non-intersecting input groups; performing convolution operations on the plurality of non-intersecting input groups simultaneously based on the grouped label matrix; and directly generating an output feature map corresponding to the input feature map using the convolution results of the plurality of non-intersecting input groups.

[0006] According to an embodiment of the present disclosure, a device for accelerating dense 3D grouped convolution is provided, comprising: a processor; and a memory, coupled to the processor and containing instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to perform the following operations: receiving an input feature map; generating a grouping label matrix of the input feature map to divide the input feature map into a plurality of non-intersecting input groups; performing convolution operations on the plurality of non-intersecting input groups simultaneously based on the grouping label matrix; and directly generating an output feature map corresponding to the input feature map using the convolution results of the plurality of non-intersecting input groups.

[0007] According to an embodiment of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored, wherein when the instructions are executed by a processor, the processor performs the above-mentioned method for accelerating dense 3D grouped convolution.

[0008] According to an embodiment of the present disclosure, a computer program product is provided, on which instructions are stored, wherein when the instructions are executed by a processor, the processor performs the above-mentioned method for accelerating dense 3D grouped convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present disclosure will be described by way of example and not limitation with reference to the figures of the accompanying drawings in which like reference numerals refer to like elements and in which: Figure 1 Illustrated is a block diagram of an example processor and / or SoC 100 that may have one or more cores and with an integrated memory controller.

[0010] Figure 2 A diagram of a process for performing dense 3D grouped convolution (D3GC) according to an embodiment of the present disclosure is shown.

[0011] Figure 3A and Figure 3B A diagram showing the efficiency comparison of the accelerated D3GC and traditional dense 3D convolution according to an embodiment of the present disclosure is shown.

[0012] Figure 4 It is a diagram of the overall process for accelerating D3GC according to an embodiment of the present disclosure.

[0013] Figure 5A is a diagram of an example of the convolution process of conventional D3GC.

[0014] Figure 5B is a diagram of an example of a convolution process of an accelerated D3GC according to an embodiment of the present disclosure.

[0015] Figure 6A flowchart of a method 600 for accelerating 3DGC according to an embodiment of the present disclosure is shown.

[0016] Figure 7 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium (eg, a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0017] The features and exemplary embodiments of various aspects of the application will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the application. However, it is obvious to those skilled in the art that the application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the application by illustrating the example of the application. The application is by no means limited to any specific configuration proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the spirit of the application. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid causing unnecessary ambiguity to the application.

[0018] Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.

[0019] The phrases "in an embodiment," "in one embodiment," and "in some embodiments" are used repeatedly herein. The phrase generally does not refer to the same embodiment; however, it may. The terms "comprising," "having," and "including" are synonymous unless the context dictates otherwise. The phrases "A or B" and "A / B" mean "(A), (B) or (A and B)."

[0020] Figure 1 A block diagram of an example processor and / or SoC 100 is illustrated, which may have one or more cores and have an integrated memory controller. The processor 100 illustrated by the solid line box has a single core 102(A), a system agent unit circuit 110, and a set of one or more interface controller unit circuits 116, while the optionally added dashed line box illustrates an alternative processor 100 as having multiple cores 102(A)-(N), a set of one or more integrated memory control unit circuits 114 in the system agent unit circuit 110, dedicated logic 108, and a set of one or more interface controller unit circuits 116.

[0021] Different implementations of processor 100 may include: 1) a CPU, where dedicated logic 108 is integrated graphics and / or scientific (throughput) logic (which may include one or more cores, not shown), and cores 102 (A)-(N) are one or more general-purpose cores (e.g., general-purpose in-order cores, general-purpose out-of-order cores, or a combination of the two); 2) a coprocessor, where cores 102 (A)-(N) are a large number of dedicated cores primarily for graphics and / or scientific (throughput) purposes; and 3) a coprocessor, where cores 102 (A)-(N) are a large number of general-purpose in-order cores. Thus, processor 100 may be a general-purpose processor, a coprocessor, or a dedicated processor, such as a network or communication processor, a compression engine, a graphics processor, a GPGPU (general-purpose graphics processing unit), a high-throughput many-in-one (MIC) coprocessor (including 30 or more cores), an embedded processor, etc. The processor may be implemented on one or more chips. Processor 100 may be part of and / or may be implemented on one or more substrates using any of a variety of process technologies, such as complementary metal oxide semiconductor (CMOS), bipolar CMOS (BiCMOS), P-type metal oxide semiconductor (PMOS), or N-type metal oxide semiconductor (NMOS).

[0022] The memory hierarchy includes one or more levels of cache unit circuits 104(A)-(N) within the cores 102(A)-(N), a set of one or more shared cache unit circuits 106, and external memory (not shown) coupled to the set of integrated memory controller unit circuits 114. The set of one or more shared cache unit circuits 106 may include one or more intermediate level caches, such as level 2 (L2), level 3 (L3), level 4 (4), or other levels of cache, such as last level cache (LLC), and / or combinations thereof. While in some examples an interface network circuit 112 (e.g., a ring interconnect) provides an interface to the dedicated logic 108 (e.g., integrated graphics logic), the set of shared cache unit circuits 106, and the system agent unit circuit 110, alternative examples use any number of well-known techniques to provide interfaces to these units. In some examples, coherency is maintained between one or more circuits in the shared cache unit circuits 106 and the cores 102(A)-(N). In some examples, interface controller unit circuitry 116 couples the cores to one or more other devices 118 , such as one or more I / O devices, storage, one or more communication devices (eg, wireless network, wired network, etc.), and the like.

[0023] In some examples, one or more of the cores 102(A)-(N) have multi-threading capabilities. The system agent unit circuit 110 includes those components that coordinate and operate the cores 102(A)-(N). The system agent unit circuit 110 may include, for example, a power control unit (PCU) circuit and / or a display unit circuit (not shown). The PCU may be (or may include) logic and components required to adjust the power state of the cores 102(A)-(N) and / or the dedicated logic 108 (e.g., integrated graphics logic). The display unit circuit is used to drive one or more externally connected displays.

[0024] The cores 102(A)-(N) may be homogeneous with respect to an instruction set architecture (ISA). Alternatively, the cores 102(A)-(N) may be heterogeneous with respect to an ISA; that is, a subset of the cores 102(A)-(N) may be capable of executing one ISA, while other cores may be capable of executing only a subset of the ISA or may be capable of executing another ISA. The processor cores 102(A)-(N) may employ the RISC-V instruction set architecture in whole or in part.

[0025] Before discussing the details of various embodiments, it is helpful to start with the basic concepts of conventional dense 3D convolution and traditional sparse 3D convolution. Given a 3D convolution layer of a CNN, the input feature map can be represented as X(W,H,D,M), the convolution kernel / filter can be represented as f(K^3,M,N) and the output feature map can be represented as X(W,H,D,N), where W, H and D represent the spatial size of the 3D data / input feature map, M is the number of input feature map channels, N is the number of output feature map channels, and K^3 is the spatial kernel / filter size (for the 2D convolution case with D=1, it can be set to K^2). For simplicity, in the various examples described herein, it is assumed that the input of the convolution layer (e.g., the input feature map) and the output from the convolution layer (e.g., the output feature map) have the same spatial size (it is understood by those skilled in the art that when the convolution stride is greater than 1, the spatial size may be different). For dense 3D convolution, f is applied to each feature position of X. In contrast, for conventional sparse 3D convolutions, f is only applied to active / valid (i.e., non-zero / non-empty) feature locations of X (e.g., feature locations stored in a hash table).

[0026] Various techniques have been proposed to address the computational issues associated with 3D CNNs. For example, patent application PCT / CN2019 / 092225, entitled "General Modular Sparse 3D Convolution Design Using Sparse Three-Dimensional (3D) Grouped Convolutions", proposes a general sparse 3D grouped convolution (S3GC) technique. S3GC can linearly reduce the convolution cost to 1 / G by dividing the input features into G disjoint input groups (where G represents the number of groups). Using a grouped convolution approach similar to S3GC, the same acceleration can be achieved for dense 3D convolutions.

[0027] Figure 2 A diagram of a process for performing dense 3D grouped convolution (D3GC) according to an embodiment of the present disclosure is shown. In some embodiments, the process may be performed by a 3D grouped convolution module implemented in software or firmware of a data processing system. Alternatively or additionally, the process may involve the use of dedicated hardware. As described above, although the present example is described with reference to an input feature map (singular form) for simplicity, those skilled in the art will understand that the 3D convolution layer has C input feature maps (channels) and the 3D convolution kernel / filter acts on a 4D tensor (i.e., all channels).

[0028] In the context of this illustration, a 2D slice of a 3D CNN is used for ease of visualization and interpretation. Figure 2As shown, dense 3D group convolution can be provided in the following manner: in the grouping stage 201, the input feature map is divided into G non-intersecting input groups with a linearly increasing sparsity ratio; in the convolution stage 202, G non-intersecting output groups corresponding to the G non-intersecting input groups are generated by performing 3D convolution on the G input groups in sequence; and in the output stage 203, an output feature map containing all voxels previously distributed in the G non-intersecting output groups is generated by merging the output feature values ​​of the voxels of the generated G non-intersecting output groups.

[0029] Figure 3A and Figure 3B is a diagram comparing the efficiency of D3GC and traditional dense 3D convolution. In the context of these diagrams, for ease of visualization and interpretation, 2D slices of 3D CNN are used, and 3×3 convolutions are shown with reference to the 2D slices. In addition, for simplicity, Figure 3B The G in is 2.

[0030] exist Figure 3A In FIG, two consecutive convolution calculations are performed by traditional sparse-dense convolution (where the source voxels are concentrated in the receptive field 3011 and the receptive field 3012, respectively), resulting in 18 operations, including operations on 9 blocks in the receptive field 3011 and operations on 9 blocks in the receptive field 3012.

[0031] In comparison, Figure 3B , 11 operations are generated by performing two consecutive convolution calculations (where the source voxels are respectively concentrated in the receptive field 3021 and the receptive field 3022) through the accelerated dense 3D grouped convolution according to the embodiment, including operations on 5 dark gray blocks in the receptive field 3021 and 6 light gray blocks in the receptive field 3022. Those skilled in the art will understand that, ideally, the computational cost of the dense 3D grouped convolution is approximately 1 / G of the traditional dense 3D convolution, where G represents the number of input groups in which the input feature positions of the input feature map are distributed.

[0032] However, when performing convolution in hardware, a large number of additional operations are added to prepare or select valid data for operation before performing multiplication and accumulation operations. For vector type processors, it will be more difficult to accelerate dense convolution. In order to achieve theoretical acceleration of 3D grouped convolution in vector type hardware, an embodiment of the present disclosure proposes a new method for accelerating dense 3D grouped convolution.

[0033] In addition, it is known that RISC-V vector instructions (RVV) are designed for intensive calculations, and each element in the input data will enter the calculation unit, so for D3GC, there are a lot of unused zero calculations, wasting time and power. The embodiment of the present disclosure proposes a new RVV instruction and process to accelerate D3GC on RVV, so that D3GC achieves a G-fold improvement in time.

[0034] Figure 4 It is a diagram of the overall process for accelerating D3GC according to an embodiment of the present disclosure. In some embodiments, the process can be performed by a 3D grouped convolution module implemented in software or firmware of a data processing system. Alternatively or additionally, the process may involve the use of dedicated hardware. As described above, although the present example is described with reference to an input feature map (singular form) for simplicity, those skilled in the art will understand that the 3D convolution layer has C input feature maps (channels) and the 3D convolution kernel / filter acts on a 4D tensor (i.e., all channels).

[0035] In the context of this illustration, 2D slices of a 3D CNN are used for ease of visualization and interpretation. Figure 4 As shown, in an embodiment of the present disclosure, the acceleration of dense 3D grouped convolution is achieved through the following operations: (401) feature grouping label generation, (402) simultaneous execution of grouped convolutions, and (403) output feature map generation.

[0036] Compared with the traditional 3D grouped convolution process, in the 3D grouped convolution process according to the embodiment of the present disclosure, the data of all groups of the input feature map are Figure 4 All processes shown are stored and processed as a whole.

[0037] In some embodiments, the 3D grouped convolution initially preprocesses the input feature map to increase the sparsity of the input presented to one or more 3D convolution kernels. In an embodiment of the present disclosure, the dense 3D grouped convolution initially divides the input feature map into G disjoint input groups, e.g., Figure 4 , the input feature map includes two groups of feature positions represented by blocks filled with two different colors, respectively. In an embodiment of the present disclosure, the grouping is achieved by generating group labels, rather than actually separating the data of the input feature map in the memory. For example, the data of the input feature map is divided into G non-overlapping input groups by marking the input feature map with mask bits. Although in the context of this example, for simplicity of explanation, G is 2, as further explained below in some embodiments, a larger value of G can be used to achieve improved convolution acceleration, and the G value is up to and including the kernel / filter size (e.g., the size of a 3D convolution kernel K^3).

[0038] In an embodiment of the present disclosure, the feature grouping of the input feature graph may be embodied in a grouping label matrix, wherein the bit length of the label in the grouping label matrix depends on the value of G, that is, depends on the number of input groups in which the input feature positions of the input feature graph are distributed. In an embodiment of the present disclosure, the bit length of the label may be calculated as: L _label = Ceil(log2(G)), where G is the number of input groups in which the input feature positions of the input feature map are distributed. Specifically, if G is 2, the bit length of the label in the group label matrix can be 1 bit; if G is 3, the bit length of the label in the group label matrix can be 2 bits; if G is 5, the bit length of the label in the group label matrix can be 3 bits, and so on.

[0039] In an embodiment of the present disclosure, different values ​​in the group label matrix are used to indicate different input groups, for example, Figure 4 As shown, the value 1 in the 1-bit label matrix represents group 1 and the value 0 represents group 2.

[0040] In an embodiment of the present disclosure, at the input feature map grouping stage, there is no need to physically separate the input data, for example, there is no need to split the data of different input groups into different memories, and therefore, the operation of moving the input data in the memory is omitted.

[0041] In an embodiment of the present disclosure, dividing the input feature map into multiple disjoint input groups may be performed along the spatial dimension of the input feature map and applied to all input channels. In an embodiment of the present disclosure, dividing the input feature map into multiple disjoint input groups may be performed independently for each of the multiple input channel groups.

[0042] In an embodiment of the present disclosure, during the convolution process, convolution operations are performed on G non-overlapping input groups simultaneously based on the generated group label matrix. First, a convolution multiplication operation is performed on the eigenvalues ​​of the input feature map and the convolution kernel parameters. Specifically, the parameters of the convolution kernel of the convolution operation are first element-wise multiplied with the input data block covered by the convolution kernel in the input feature map. Then, the multiplication results of different groups are accumulated respectively according to the group labels. Specifically, the label bits of the features in the corresponding input data blocks in the group label matrix are used as mask bits, and the element multiplication results belonging to the same input group are accumulated to simultaneously obtain the output eigenvalues ​​of the G non-overlapping input groups for the input data block. Finally, the result is naturally placed in the correct position to directly generate the output feature map corresponding to the input feature map.

[0043] In the traditional group convolution operation, the divided G groups need to perform convolution operations independently in sequence and generate corresponding G output groups, and the output feature map corresponding to the input feature map is generated by merging the generated G output groups. Compared with the traditional convolution operation, the convolution process according to the embodiment of the present disclosure saves the operation of merging the group results.

[0044] Figure 5A is a diagram of an example of the convolution process of conventional D3GC. Figure 5B is an illustration of an example of a convolution process of an accelerated D3GC according to an embodiment of the present disclosure. In the context of these illustrations, a 2D slice of a 3DCNN is used for ease of visualization and explanation. And in the context of this example, G is 2 for simplicity of illustration. However, it should be understood that a larger value of G may be used to achieve improved convolution acceleration, which may be up to and including the kernel / filter size (e.g., the size of a 3D convolution kernel K^3).

[0045] like Figure 5A As shown in FIG. 1 , in the convolution process of conventional D3GC, the convolution of 9 voxels in the receptive field 501 (including 5 voxels of input group 1 and 4 voxels of input group 2) includes: 9 multiplication-accumulation operations for input group 1 in the first convolution cycle and 9 multiplication-accumulation operations for input group 2 in the second convolution cycle, that is, the convolution operation on the receptive field 501 requires 18 multiplication-accumulation operations in two convolution cycles. Figure 5B In the accelerated D3GC convolution process shown in FIG. 5 , the convolution of 9 voxels in the same receptive field 501 includes: 5 multiplication-accumulation operations performed on 5 voxels of the first input group in one convolution cycle and 4 multiplication-accumulation operations performed on 4 voxels of the second input group in the same convolution cycle, that is, Figure 5B In the example, the convolution operation on the receptive field 501 only requires 9 multiplication-accumulation operations within one convolution cycle. Compared with the conventional convolution process, the accelerated D3GC according to the embodiment of the present disclosure takes 1 / G of the cycle required by the conventional process (in Figure 5B In the example of , convolution is performed in 1 / 2) cycles, which greatly reduces the convolution cost of D3GC.

[0046] In an embodiment of the present disclosure, a new RVV vector instruction for performing accelerated D3GC is provided to trigger hardware optimized D3GC operations.

[0047] The RVV vector instructions involved in conventional D3GC include: vfmul.vv vd, vs2, vs1, vm # vector floating point multiplication vfredosum.vs vd, vs2, vs1, vm # vector floating point sum reduction The conventional RVV vector instruction completes, for example, two convolutions. The instruction statement that needs to be executed is: vload.vv vs0, rs1, rs2 #load mask operand vm vload.vv vs1, rs3, rs4 #load group 1 into register vs1 vload.vv vs2, rs3, rs5 #load group 2 into register vs2 vload.vv vs3, rs6, rs0 #load the characteristic value into register vs3 vfmul.vv vs5, vs3, vs1, vm # vector floating point multiplication vfredosum.vs vs6, vs5, vs6, vm # vector floating point sum reduction vminv.vv v0, v0, rs0, vm # Perform vector inverse operation on vm value vfmul.vv vs7, vs3, vs2, vm # vector floating point multiplication vfredosum.vs vs8, vs7, vs8, vm #vector floating point sum reduction vfadd.vs vs9, vs7, vs8 #Merge group results It can be seen that conventional RVV vector instructions involve a large number of data preparation operations in the process of performing convolution operations. In order to save operations, the disclosed embodiment provides the following new RVV vector instructions (which are also applicable to integer data types): vfgmul.vv vd, vs2, vs1, vm # vector floating point group multiplication vfgredosum.vs vd, vs2, vs1, vm # vector floating point group sum reduction In an embodiment of the present disclosure, the RVV vector instruction for performing accelerated dense 3D grouped convolution instructs the processor to perform convolution operations for multiple (G) disjoint input groups simultaneously in an operation unit (e.g., a multiply-add operation unit Mac). In an embodiment of the present disclosure, the RVV vector instruction for performing accelerated dense 3D grouped convolution instructs the processor to perform G convolution operations simultaneously in a computing unit.

[0048] In an embodiment of the present disclosure, the hardware implementation of grouped convolution is implemented by grouping operands with the same mask value for both multiplication and accumulation operations, for example, Figure 5B As shown, through this grouping, the same computing unit as the conventional operation can complete two convolutions at the same time.

[0049] In an example, the statement to implement two convolutions using the new RVV vector instruction is: vload.vv vs0, rs1, rs2 #load mask operand vm vload.vv vs1, rs3, rs4 #load group 1 into register vs1 vload.vv vs3, rs6, rs0 #load the characteristic value into register vs3 vfmul.vv vs5, vs3, vs1, vm # vector floating point multiplication vfredosum.vs vs6, vs5, vs6, vm # vector floating point sum reduction It can be seen that compared with conventional RVV instructions, using the newly proposed RVV vector instructions to execute D3GC can save a lot of operations, thereby significantly reducing the convolution cost of D3GC and improving the convolution efficiency of D3GC.

[0050] Table 1 shows the comparison between the number of operations required to implement D3GC using conventional RVV vector instructions and the number of operations required to implement D3GC using improved RVV vector instructions. As shown in Table 1, by using the newly proposed RVV instructions for D3GC, the operation of separating feature data in the memory is saved in the input feature map grouping stage, the operation of grouping convolution of expanded data is saved in the convolution stage, and the operation of output group merging is saved in the result output node. Compared with using traditional RVV vector instructions to execute D3GC, using the newly proposed RVV vector instructions to execute D3GC saves a total of 40% of operations, and the convolution efficiency is significantly improved.

[0051] Table 1

[0052] Figure 6 A flow chart of a method 600 for accelerating 3DGC according to an embodiment of the present disclosure is shown. In one embodiment, the method 600 may be performed by a vector type processor. In one example, the method 600 may be performed by a processor supporting RISC-V. The method 600 may include steps S602, S604, S606, and S608. However, in some embodiments, the method 600 may include more or fewer different steps, and the present disclosure is not limited thereto.

[0053] In step S602, an input feature map is received, for example, by a convolutional layer of a neural network model implemented on a data processing system.

[0054] In step S604, a group label matrix of the received input feature map is generated, where the group label matrix is ​​used to divide the input feature map into a plurality of disjoint input groups.

[0055] In step S606, a convolution operation is simultaneously performed on a plurality of disjoint input groups based on the generated group label matrix.

[0056] At step 608, the convolution results of the plurality of disjoint input groups are used to directly generate an output feature map corresponding to the received input feature map.

[0057] In an embodiment of the present disclosure, in the input feature map grouping stage of the 3D grouped convolution, memory and operations are saved by marking feature groups instead of separating the data in the memory. In the convolution stage of the 3D grouped convolution, a new vector instruction - vector reduction group multiplication (vrgm) instruction is used to perform multiplication-accumulation operations on multiple groups at the same time (such as Figure 4The convolution cycle is reduced by using the convolution operation (as shown). In the output stage, the output feature values ​​of the input group are directly placed in the corresponding positions of the output feature map to save the output data merging operation. Through these three novel stages, the present disclosure linearly reduces the computational cost of the convolution operation of the 3D convolutional neural network, and uses RISC-V to achieve a cost reduction close to the theoretical value in actual hardware.

[0058] Figure 7 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Figure 7 A schematic diagram of hardware resources 700 is shown, which includes one or more processors (or processor cores) 710, one or more memory / storage devices 720, and one or more communication resources 730, wherein each of these processors, memory / storage devices, and communication resources can be communicatively coupled via a bus 740 or other interface circuits. For embodiments utilizing node virtualization (e.g., network function virtualization (NFV)), a hypervisor 702 can be executed to provide an execution environment for one or more network slices / sub-slices to utilize hardware resources 700.

[0059] Processor 710 may include, for example, processor 712 and processor 714. Processor 710 may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP) such as a baseband processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a radio frequency integrated circuit (RFIC), another processor (including those discussed herein), or any suitable combination thereof.

[0060] The memory / storage device 720 may include a main memory, a disk storage device, or any suitable combination thereof. The memory / storage device 720 may include, but is not limited to, any type of volatile, non-volatile, or semi-volatile memory, such as, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory, etc.

[0061] The communication resources 730 may include an interconnect or network interface controller, component, or other suitable device to communicate with one or more peripheral devices 704 or one or more databases 706 or other network elements via the network 708. For example, the communication resources 730 may include a wired communication component (e.g., for coupling via USB, Ethernet, etc.), a cellular communication component, a near field communication (NFC) component, a Bluetooth® (or Bluetooth® low energy) component, a Wi-Fi® component, and other communication components.

[0062] The instructions 750 may include software, a program, an application, an applet, an application, or other executable code for causing at least any one of the processors 710 to perform any one or more of the methods discussed herein. The instructions 750 may reside in whole or in part in at least one of the processors 710 (e.g., in a cache of the processor), the memory / storage device 720, or any suitable combination thereof. In addition, any portion of the instructions 750 may be transmitted to the hardware resources 700 from any combination of the peripheral device 704 or the database 706. Therefore, the memory of the processor 710, the memory / storage device 720, the peripheral device 704, and the database 706 are examples of computer-readable and machine-readable media.

[0063] Some examples may be implemented using or as an article of manufacture or at least one computer-readable medium. The computer-readable medium may include a non-transitory storage medium to store logic. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, and the like. In some examples, logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, APIs, instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols, or any combination of these.

[0064] According to some examples, a computer-readable medium may include a non-transitory storage medium to store or maintain instructions that, when executed by a machine, a computing device, or a system, cause the machine, the computing device, or the system to perform methods and / or operations according to the described examples. The instructions may include any appropriate type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The instructions may be implemented according to a predetermined computer language, manner, or syntax to instruct a machine, a computing device, or a system to perform a specific function. The instructions may be implemented using any appropriate high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0065] One or more aspects of at least one example may be implemented by representative instructions stored on at least one machine-readable medium representing various logic within the processor, which when read by a machine, computing device or system causes the machine, computing device or system to fabricate logic to perform the techniques described herein. Such representations, known as "IP cores", may be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities to load into fabrication machines that actually fabricate the logic or processor.

[0066] The appearance of the phrase "an example" or "an example" does not necessarily refer to the same example or embodiment. Any aspect described herein may be combined with any other aspect or similar aspect described herein, whether or not these aspects are described with respect to the same figure or element. The division, omission or inclusion of block functions depicted in the figures does not infer that the hardware components, circuits, software and / or elements used to implement these functions will necessarily be divided, omitted or included in the embodiments.

[0067] Some examples may be described using the expressions "coupled" and "connected," and their derivatives. These terms are not necessarily intended to be synonymous with each other. For example, descriptions using the terms "connected" and / or "coupled" may indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also refer to two or more elements that are not in direct contact with each other, but still cooperate or interact with each other.

[0068] The terms "first", "second" and the like do not represent any order, quantity or importance in this article, but are used to distinguish one element from another element. The term "one" herein does not represent a limitation on quantity, but rather represents the presence of at least one item mentioned. The term "assertion" used when referring to a signal herein refers to a state of the signal, in which the signal is valid, and the state can be achieved by applying any logic level (whether logic 0 or logic 1) to the signal. The term "subsequently" or "afterwards" may refer to immediately following or following some or some other event. According to alternative embodiments, other step sequences may also be performed. In addition, depending on the specific application, additional steps may be added or removed. Any combination of variations may be used, and those of ordinary skill in the art who benefit from the present disclosure will understand its many variations, modifications and alternative embodiments.

[0069] Unless specifically stated otherwise, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood within the context to be generally used to state that an item, term, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require the presence of each of at least one X, at least one Y, or at least one Z. Furthermore, unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" should also be understood to refer to X, Y, Z, or any combination thereof, including "X, Y, and / or Z."

Claims

1. A method for accelerating dense three-dimensional 3D grouped convolution, comprising: Receiving an input feature map; Generate a group label matrix of the input feature map, where the group label matrix is ​​used to divide the input feature map into multiple disjoint input groups; Perform convolution operations on multiple disjoint input groups simultaneously based on the group label matrix; as well as The convolution results of multiple disjoint input groups are used to directly generate output feature maps corresponding to the input feature maps.

2. The method according to claim 1, wherein: The data of multiple disjoint input groups are stored and processed as a whole during the convolution operation.

3. The method according to claim 1 or 2, wherein: Performing convolution operations on multiple disjoint input groups simultaneously based on the group label matrix includes: Perform convolution multiplication on the parameters of the convolution kernel of the convolution operation and the data of the input data block covered by the convolution kernel in the input feature map; The label of each voxel in the input data block in the group label matrix is ​​used as a mask bit, and the multiplication results belonging to the same input group are accumulated to obtain the output feature values ​​of the input data block for each of the multiple non-overlapping input groups.

4. The method according to claim 3, wherein: Using the convolution results of multiple disjoint input groups to directly generate output feature maps corresponding to the input feature maps includes: The output feature values ​​of each of the multiple non-overlapping input groups for the input data block are directly placed at the corresponding position of the output feature map.

5. The method according to claim 1, further comprising: An RVV vector instruction for performing accelerated dense 3D grouped convolution is provided to trigger hardware optimized dense 3D grouped convolution operations.

6. The method according to claim 5, wherein: The RVV vector instruction for performing accelerated dense 3D grouped convolutions instructs the processor to perform convolution operations for multiple disjoint input groups simultaneously in a compute unit.

7. The method according to claim 5, wherein: The RVV vector instruction for performing accelerated dense 3D grouped convolution instructs the processor to perform multiple convolution operations simultaneously in a computation unit, wherein the number of the multiple convolution operations is equal to the number of the multiple disjoint input groups.

8. A device for accelerating dense three-dimensional (3D) grouped convolution, comprising: processor; as well as a memory coupled to the processor and containing instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform the following operations: Receiving an input feature map; generating a group label matrix of the input feature map to divide the input feature map into a plurality of disjoint input groups; Perform convolution operations on multiple disjoint input groups simultaneously based on the group label matrix; as well as The convolution results of multiple disjoint input groups are used to directly generate output feature maps corresponding to the input feature maps.

9. The device according to claim 8, wherein: The data of multiple disjoint input groups are stored and processed as a whole during the convolution operation.

10. The device according to claim 8 or 9, wherein: The instructions further cause the processor to: Performing a convolution multiplication operation on the parameters of the convolution kernel of the convolution operation and the data of the input data block covered by the convolution kernel in the input feature map; The label of each voxel in the input data block in the group label matrix is ​​used as a mask bit, and the multiplication results belonging to the same input group are accumulated to obtain the output feature values ​​of the input data block for each of the multiple non-overlapping input groups.

11. The device according to claim 10, wherein: The instructions further cause the processor to: The output feature values ​​generated by each of the multiple non-overlapping input groups for the input data block are directly placed at the corresponding position of the output feature map.

12. The device according to claim 8, wherein: The processor is a vector type processor.

13. The apparatus according to claim 8, wherein: The processor is a RISC-V based processor.

14. The device according to claim 13, wherein: The instructions include RVV vector instructions for performing accelerated dense 3D grouped convolution, which are used to trigger hardware optimized dense 3D grouped convolution operations.

15. The device according to claim 14, wherein: The RVV vector instruction for performing accelerated dense 3D grouped convolution instructs the processor to perform convolution operations for multiple disjoint input groups simultaneously in a computational unit.

16. The apparatus according to claim 14, wherein: The RVV vector instruction for performing accelerated dense 3D grouped convolution instructs the processor to simultaneously perform multiple convolution operations in the computational unit, wherein the number of the multiple convolution operations is equal to the number of disjoint input groups into which the input feature map is divided.

17. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to perform the method for accelerating dense 3D grouped convolution according to any one of claims 1-7.

18. A computer program product having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to perform the method for accelerating dense 3D grouped convolution according to any one of claims 1-7.

Images