Intelligent processing unit and operation method of deformable convolution
By combining grid processing circuits and convolution calculation circuits in the intelligent processing unit, grid sampling and convolution operations are performed, the problems of low efficiency and high cost of deformable convolution calculation in the prior art are solved, and an efficient and low-cost calculation solution is realized.
Patent Information
- Application Number
- CN202411662078.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-14
AI Technical Summary
In the prior art, deformable convolutional calculations are performed using a central processing unit (CPU) or a graphics processing unit (GPU), resulting in low computing efficiency and high cost, and are not suitable for low-cost embedded systems.
An intelligent processing unit is designed, including memory, grid processing circuit and convolution calculation circuit. Through grid sampling operations, the deformable convolution input data is converted into general convolution input data, and the convolution operation is performed to generate output data.
Improves the efficiency of deformable convolution operations and reduces costs, making them suitable for low-cost embedded systems.
Smart Images
Figure CN119622171A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of convolution operation, and in particular to an intelligent processing unit and a deformable convolution operation method. Background Art
[0002] Deformable convolution is a type of convolution. Figure 1 It is a schematic diagram of an existing deformable convolution. The deformable convolution operator 100 generates output data Dout after operating the deformable convolution input data DAT (for example, feature map, weights KER (weights, or convolution kernel), offset OST, mask MSK and bias BIS). The offset OST is used to indicate the correspondence between the output data Dout of the deformable convolution and the deformable convolution input data DAT. The operational details of the deformable convolution are well known to technicians in this technical field, so they are not repeated here. It should be noted that in some applications, the mask MSK does not exist.
[0003] The prior art uses a central processing unit (CPU) or a graphic processing unit (GPU) to perform deformable convolution calculations. However, since the CPU and GPU are not circuits specifically used for deformable convolution calculations, the calculation efficiency is not high. Furthermore, since the cost of the CPU and GPU is relatively high, they are not suitable for low-cost embedded systems. Summary of the invention
[0004] One of the purposes of the present application is to provide an intelligent processing unit and a deformable convolution operation method to improve the deficiencies of the prior art.
[0005] One embodiment of the present application provides an intelligent processing unit, comprising: a memory, a grid processing circuit and a convolution calculation circuit. The memory is used to store a portion of a first input data of a deformable convolution operation, a portion of a bias of the deformable convolution operation, a portion of a weight of the deformable convolution operation and a portion of a grid, wherein the grid is converted from an offset of the deformable convolution operation. The grid processing circuit is coupled to the memory and is used to perform a grid sampling operation to generate a second input data based on the first input data and the grid. The convolution calculation circuit is coupled to the memory and is used to perform a convolution operation on the second input data, the weight and the bias to generate an output data. The output data is substantially equal to the result of the deformable convolution operation.
[0006] Another embodiment of the present application provides a deformable convolution operation method, which is executed in an intelligent processing unit, and includes: performing a grid sampling operation to generate a second input data based on a grid and a first input data of a deformable convolution operation, wherein the grid is converted from an offset of the deformable convolution operation; and performing a convolution operation on the second input data, a weight of the deformable convolution operation, and a bias of the deformable convolution operation to generate an output data. The output data is substantially equal to the result of the deformable convolution operation.
[0007] The technical means embodied in the embodiments of the present application can improve at least one of the shortcomings of the prior art, so the present application can improve efficiency and reduce costs compared to the prior art.
[0008] The features, implementation and effects of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 This is a schematic diagram of the existing deformable convolution;
[0010] Figure 2 is a circuit diagram of an embodiment of the electronic device of the present application;
[0011] Figure 3 is a schematic diagram of an embodiment of a deformable convolution operation of the present application;
[0012] Figure 4 is a schematic diagram of another embodiment of the deformable convolution operation of the present application;
[0013] Figure 5 is a flow chart of an embodiment of the present application for converting offset into a grid;
[0014] Figure 6 yes Figure 5 A constant provides a flowchart of an embodiment of step S530;
[0015] Figure 7 Displays a section of program code written in C language to generate constants;
[0016] Figure 8 is a schematic diagram of another embodiment of the deformable convolution operation of the present application;
[0017] Fig. 9 is a flow chart of an embodiment of the grid sampling operation of the present application; and
[0018] Fig.10 It is a schematic diagram of the input and output of the grid sampling operator of this application.
[0019] Reference numerals:
[0020] 100: deformable convolution operator;
[0021] BIS: Bias;
[0022] DAT: deformable convolution input data;
[0023] Dout: output data;
[0024] KER, KER1: weight;
[0025] MSK, MSK1, MSK2: mask;
[0026] OST: offset;
[0027] 200: electronic devices;
[0028] 210: Intelligent processing unit;
[0029] 212: direct memory access circuit;
[0030] 214: Convolution calculation circuit;
[0031] 216: memory;
[0032] 218: grid processing circuit;
[0033] 219: interpolation calculation circuit;
[0034] 220: external memory;
[0035] 300, 400, 800: deformable convolution operation;
[0036] 310, 410, 830: grid sampling operator;
[0037] 320: Multiplication operator;
[0038] 330, 850: convolution operator;
[0039] DAT1: intermediate data;
[0040] DAT2: general convolution input data;
[0041] GRD: Grid;
[0042] 500: conversion process;
[0043] C1, C2, C3, Cf, Cl, Ct, Cx, Cy: constants;
[0044] D1, D2, D3, D4, D10, D11, D12, D13, D5, D6, D7, D8, D9: Data
[0045] S510, S520, S530, S540, S550, S560, S570, S612, S614, S622, S624, S632, S634, S640, S650, S660, S670, S910, S920, S930, S940, S950, S960, S965, S970, S980, S990, S995: steps;
[0046] R1, R2, R3: parameters;
[0047] S_h, S_w: step length;
[0048] 810: transpose operator;
[0049] 820, 840: reshaping operator;
[0050] 900: grid sampling operation;
[0051] Ci, Cg, Co: number of channels;
[0052] GT0, GT1, GT2, GT3: small grid blocks;
[0053] IT0, IT1, IT2, IT3: small input blocks;
[0054] I_h: the height of the deformable convolution input data;
[0055] I_w: width of deformable convolution input data;
[0056] OT0, OT1, OT2, OT3: output small blocks;
[0057] U_h: the height of the general convolution input data;
[0058] U_w: The width of general convolution input data. DETAILED DESCRIPTION
[0059] The technical terms used in the following descriptions are based on the customary terms in the technical field. If some of the terms are explained or defined in this specification, the interpretation of such terms shall be based on the explanation or definition in this specification.
[0060] The disclosure of the present application includes an intelligent processing unit and a method for calculating deformable convolution. Since some of the components included in the intelligent processing unit of the present application may be known components individually, the details of the known components will be omitted in the following description without affecting the full disclosure and feasibility of the device invention. In addition, part or all of the process of the method for calculating deformable convolution of the present application may be in the form of software and / or firmware, and may be executed by the intelligent processing unit of the present application or its equivalent device. Without affecting the full disclosure and feasibility of the method invention, the following description of the method invention will focus on the step content rather than the hardware.
[0061] Figure 2 2 is a circuit diagram of an embodiment of an electronic device of the present application. The electronic device 200 includes an intelligence processing unit (IPU) 210 and an external memory 220 coupled to each other. The intelligence processing unit 210 includes a direct memory access (DMA) circuit 212, a convolution calculation circuit 214, a memory 216, and a grid processing circuit 218 coupled to each other. The grid processing circuit 218 includes an interpolation calculation circuit 219.
[0062] The external memory 220 stores data related to deformable convolution calculation, such as deformable convolution input data DAT, weights KER, offsets OST, masks MSK (if any), and biases BIS.
[0063] The present application performs a grid sampling operation by the grid processing circuit 218 to convert the deformable convolution input data DAT of the deformable convolution into the input data DAT2 of the general convolution (i.e., non-deformable convolution, such as two-dimensional convolution or three-dimensional convolution) (which will be described below in conjunction with Fig. 9 The details of the grid sampling operation are well known to those skilled in the art, and related content can be found at: pytorch.org / docs / stable / generated / torch.nn.functional.grid_sample.html.
[0064] The direct memory access circuit 212 is coupled between the memory 216 and the external memory 220, and is used to read data from the external memory 220 and write the read data to the memory 216, or read data from the memory 216 and write the read data to the external memory 220. Since the capacity of the memory 216 is usually much smaller than the capacity of the external memory 220, the deformable convolution input data DAT, weight KER, offset OST and deviation BIS are often cut into multiple small blocks (tiles) for convolution operations. Cutting data into multiple small blocks is well known to technicians in this technical field, so it is not repeated here. In actual operation, the memory 216 stores at least one small block of the deformable convolution input data DAT, at least one small block of the weight KER, at least one small block of the offset OST, and at least one small block of the deviation BIS.
[0065] The convolution calculation circuit 214 is used to perform a general (ie, non-deformable) convolution operation (eg, a two-dimensional convolution operation or a three-dimensional convolution operation) and store the result of the convolution operation (ie, output data Dout) into the memory 216 .
[0066] The interpolation calculation circuit 219 performs interpolation calculation according to the interpolation coefficients generated by the grid processing circuit 218. In some embodiments (for example only, not to limit the present application), the interpolation calculation circuit 219 operates based on bilinear interpolation or nearest-neighbor interpolation.
[0067] See also Figure 3 , Figure 3 2 is a schematic diagram of an embodiment of a deformable convolution operation of the present application. The deformable convolution operation 300 (i.e., the operation method of the deformable convolution) includes a grid sampling operator 310 (i.e., a grid sampling operation), a multiplication operator 320 (i.e., a multiplication operation) and a convolution operator 330 (i.e., a convolution operation). The convolution operator 330 is a general convolution, not a deformable convolution. The grid sampling operator 310 and the multiplication operator 320 are executed by the grid processing circuit 218. The convolution operator 330 is executed by the convolution calculation circuit 214.
[0068] The input of the grid sampling operator 310 is the deformable convolution input data DAT and the grid GRD. The grid GRD specifies the correspondence between the output data Dout and the deformable convolution input data DAT. More specifically, the grid GRD specifies the correspondence between a certain point (coordinate) of the output data Dout and a certain point (coordinate) of the deformable convolution input data DAT. The offset OST can be converted into a grid GRD based on the attribute parameters of the deformable convolution (which will be described below). Figure 5If the offset OST is a constant, the grid GRD is also a constant. The grid sampling operator 310 converts the deformable convolution input data DAT into the intermediate data DAT1 according to the grid GRD.
[0069] The multiplication operator 320 multiplies the intermediate data DAT1 by the mask MSK to generate the general convolution input data DAT2. The convolution operator 330 performs a general convolution operation (e.g., a two-dimensional convolution operation or a three-dimensional convolution operation) on the general convolution input data DAT2, the weight KER, and the bias BIS to generate the output data Dout. The multiplication operator 320 and the convolution operator 330 are well known to those skilled in the art, so they are not described in detail.
[0070] See also Figure 4 , Figure 4 4 is a schematic diagram of another embodiment of the deformable convolution operation of the present application. The deformable convolution operation 400 includes a grid sampling operator 410 and a convolution operator 330. The grid sampling operator 410 converts the deformable convolution input data DAT into general convolution input data DAT2 according to the grid GRD and the mask MSK. That is, the deformable convolution operation 400 is similar to the deformable convolution operation 300, wherein the operation of the grid sampling operator 410 is equivalent to the combination of the operation of the grid sampling operator 310 and the operation of the multiplication operator 320.
[0071] Figure 3 and Figure 4 The output data Dout is Figure 1 The output data Do ut of the deformable convolution operator 100 is substantially the same.
[0072] See also Figure 5 , Figure 5 It is a flow chart of an embodiment of the present application for converting an offset OST into a grid GRD. Figure 5 The data dimensions shown in are for illustrative purposes only and are not intended to limit this application. Figure 5 In the example, K_h, K_w, O_h, O_w, I_h and I_w respectively represent the height of the weight KER, the width of the weight KER, the height of the output data Dout, the width of the output data Dout, the height of the deformable convolution input data DAT and the width of the deformable convolution input data DAT.
[0073] The conversion process 500 can be performed by the grid processing circuit 218, including the following steps. The reshaping step S510 reshapes the offset OST (dimension: [1, 2*K_h*K_w, O_h, O_w]) into data D1 (dimension: [1, 2, K_h*K_w, O_h, O_w]). The transpose step S520 transposes the data D1 into data D2 (dimension: [1, O_h, O_w, K_h*K_w, 2]). The constant providing step S530 provides the constant C1 (dimension: [1, O_h, O_w, K_h*K_w, 2]). Step S530 will cooperate with the following Figure 6 Detailed description. The addition step S540 adds the data D2 to the constant C1 to generate data D3 (dimension: [1, O_h, O_w, K_h*K_w, 2]). The multiplication step S550 multiplies the data D3 by the constant C2 (dimension: [2 / (I_h-1), 2 / (I_w-1)]) to generate data D4 (dimension: [1, O_h, O_w, K_h*K_w, 2]). The addition step S560 adds the data D4 to the constant C3 (dimension: [-1, -1]), and then the reshaping step S570 reshapes the result of step S560 into a grid GRD (dimension: [1, O_h, O_w*K_h*K_w, 2]).
[0074] In some embodiments, the intelligent processing unit 210 (more specifically, the grid processing circuit 218) performs only when the offset OST is a variable. Figure 5 Conversely, when the offset OST is constant, Figure 5 The conversion process 500 can be completed in advance on a development device (e.g., a general computer). In this case, the grid GRD can be pre-stored in the memory 216, and the intelligent processing unit 210 does not need to execute Figure 5 process.
[0075] See also Figure 6 , Figure 6 yes Figure 5 The constant providing step S530 is a flowchart of an embodiment, including the following steps. The multiplication step S612 multiplies the step size S_h by the constant Cy (dimension: [O_h, 1, 1]) to generate data D5 (dimension: [O_h, 1, 1]). The multiplication step S614 multiplies the step size S_w by the constant Cx (dimension: [1, O_w, 1]) to generate data D6 (dimension: [1, O_w, 1]). The step size S_h and the step size S_w are respectively the step sizes in height and width when the weight KER slides on the deformable convolution input data DAT. The constant Cy and the constant Cx are shown in formulas (1) and (2) respectively.
[0076] Cy={0,1,…,O_h-1} (1);
[0077] Cx={0,1,…,O_w-1} (2);
[0078] The addition step S622 adds the data D5 and the constant Ct to generate data D7 (dimension: [O_h, 1, 1]). The addition step S624 adds the data D6 and the constant Cl to generate data D8 (dimension: [1, O_w, 1]). The constant Ct and the constant Cl are shown in formulas (3) and (4) respectively.
[0079] Ct=(K_h-1) / 2*D_h-Pt (3);
[0080] Cl=(K_w-1) / 2*D_w-Pl (4);
[0081] Among them, D_h and D_w are the expansion coefficients of the weight KER in height and width, respectively, and Pt and Pl are the padding values (padding) of the deformable convolution input data DAT on the top and left, respectively.
[0082] The tile step S632 copies the data D7 according to the parameter R1 (dimension: [1, O_w, 1]) to generate data D9 (dimension: [O_h, O_w, 1]). The tile step S634 copies the data D8 according to the parameter R2 (dimension: [O_h, 1, 1]) to generate data D10 (dimension: [O_h, O_w, 1]). The tile operation includes copying the data expansion tensor in one or more dimensions and is well known to those skilled in the art, so it will not be described in detail.
[0083] The concatenation step S640 concatenates the data D9 and the data D10 to generate the data D11 (dimension: [O_h, O_w, 2]). The reshaping step S650 reshapes the data D11 to generate the data D12 (dimension: [1, O_h, O_w, 1, 2]). The copying step S660 copies the data D12 according to the parameter R3 (dimension: [1, 1, 1, K_h*K_w, 1]) to generate the data D13 (dimension: [1, O_h, O_w, K_h*K_w, 2]). The addition step S670 adds the data D13 to the constant Cf to generate the constant C1. The constant Cf is a matrix, and its content is as follows Figure 7 shown. Figure 7 A section of program code written in C language is shown, which is used to generate the constant Cf. This section of program code is well known to those skilled in the art, so it will not be described in detail.
[0084] In some embodiments, Figure 6The process can be completed in advance on the development device. That is, the constant C1 can be pre-stored in the memory 216, so the intelligent processing unit 210 does not need to execute Figure 6 process.
[0085] See also Figure 8 , Figure 8 FIG. 8 is a schematic diagram of another embodiment of a deformable convolution operation of the present invention. Compared with the deformable convolution operation 400 , the deformable convolution operation 800 shows more details. Figure 8 The data dimensions shown in are for illustrative purposes only and are not intended to limit this application. Figure 8 In the example, Ci and Co represent the number of channels of the deformable convolution input data DAT and output data Dout respectively, and the meanings of the remaining symbols are the same as Figure 5 same.
[0086] The deformable convolution operation 800 includes a transpose operator 810 , a reshape operator 820 , a grid sampling operator 830 , a reshape operator 840 , and a convolution operator 850 .
[0087] The transposition operator 810 transposes the mask MSK (dimension: [1, K_h*K_w, O_h, O_w]) into the mask MSK1 (dimension: [1, O_h, O_w, K_h*K_w]). The reshape operator 820 reshapes the mask MSK1 into the mask MSK2 (dimension: [1, 1, O_h, O_w*K_h*K_w]). In some embodiments, if the mask MSK does not exist, the transposition operator 810 and the reshape operator 820 can be omitted.
[0088] The grid sampling operator 830 converts the deformable convolution input data DAT (dimension: [1, Ci, I_h, I_w]) into general convolution input data DAT2 (dimension: [1, Ci, O_h, O_w*K_h*K_w]) according to the grid GRD (dimension: [1, O_h, O_w*K_h*K_w, 2]) and the mask MSK2 (if any).
[0089] The reshaping operator 840 (executed by the direct memory access circuit 212 ) reshapes the weight KER (dimensions: [Co, Ci, K_h, K_w]) into a weight KER1 (dimensions: [Co, Ci, 1, K_h*K_w]).
[0090] The convolution operator 850 is a general convolution, not a deformable convolution. The convolution operator 850 performs a general convolution operation on the general convolution input data DAT2 (dimension: [1, Ci, O_h, O_w*K_h*K_w]), the weight KER1 (dimension: [Co, Ci, 1, K_h*K_w]) and the bias BIS (dimension: [Co]) to generate the output data Dout (dimension: [1, Co, O_h, O_w]).
[0091] See also Fig. 9 and Fig.10 . Fig. 9 It is a flow chart of an embodiment of the grid sampling operation of the present application. Fig.10 It is a schematic diagram of the input (deformable convolution input data DAT and grid GRD) and output (general convolution input data DAT2) of the grid sampling operator of the present application. The height, width and number of channels of the general convolution input data DAT2 are U_h, U_w and Ci, respectively, while the length, width and number of channels of the grid GRD are U_h, U_w and Cg, respectively. The number of channels Cg represents the number of coordinates of a point. For example, when the number of channels Cg is 2 (or 3), a point corresponds to 2 (or 3) coordinates (i.e., two-dimensional coordinates (or three-dimensional coordinates)). Fig.10 In the embodiment of the invention, the deformable convolution input data DAT includes 4 small blocks (IT0, IT1, IT2, IT3), the grid GRD includes 4 small blocks (GT0, GT1, GT2, GT3), and the general convolution input data DAT2 includes 4 small blocks (OT0, OT1, OT2, OT3). The number of small blocks is only used for example and is not used to limit the present application.
[0092] The grid sampling operation 900 may correspond to the grid sampling operator 310 , the grid sampling operator 410 or the grid sampling operator 830 , and includes the following steps.
[0093] Step S910 : the direct memory access circuit 212 reads a small block of the deformable convolution input data DAT (hereinafter referred to as the input small block) from the external memory 220 , and stores the input small block into the memory 216 .
[0094] Step S920 : the DMA circuit 212 reads a small block of the grid GRD (hereinafter referred to as a small grid block) from the external memory 220 , and stores the small grid block in the memory 216 .
[0095] Step S930: The grid processing circuit 218 searches for multiple reference points of the input small block to be used in the grid small block according to a target point of a small block (hereinafter referred to as the output small block) of the general convolution input data DAT2. More specifically, on the height-width plane, each point of the general convolution input data DAT2 has a one-to-one correspondence with each point of the grid GRD, and a point in the grid GRD points to a point on the height-width plane of the deformable convolution input data DAT. For example, assuming that the target point is the point at the upper left corner of the output small block OT0, the grid processing circuit 218 searches for a coordinate from the corresponding position of the grid GRD (for example, the upper left corner of the grid small block GT0) according to the target point. Next, the grid processing circuit 218 finds an initial reference point corresponding to the coordinate on the height-width plane of the deformable convolution input data DAT according to the coordinate, and then uses all points (a total of Ci) corresponding to the initial reference point on the channel dimension as multiple reference points.
[0096] Step S940: The grid processing circuit 218 calculates the coordinates of the output point (i.e., a part of the general convolution input data DAT2) and the output points in the output small block according to the multiple reference points, and counts the number of the output points. More specifically, in step S940, the grid processing circuit 218 generates interpolation coefficients and passes the interpolation coefficients to the interpolation calculation circuit 219. The interpolation calculation circuit 219 interpolates the multiple reference points based on the multiple interpolation coefficients to calculate the multiple output points.
[0097] Step S950: the grid processing circuit 218 calculates the addresses of the multiple output points in the external memory 220 according to the coordinates of the multiple output points in the output small blocks.
[0098] Step S960: The grid processing circuit 218 determines whether the next output point is continuous. The grid processing circuit 218 determines whether the deformable convolution input data DAT (i.e., multiple reference points) corresponding to the multiple output points have been stored in the memory 216 according to the grid GRD. Because the memory 216 does not store the deformable convolution input data DAT at the same time, but only stores one of the input small blocks (step S910), the multiple reference points may exist in the memory 216 (i.e., the input small blocks to which the multiple reference points belong are stored in the memory 216, hereinafter referred to as situation (1)) or not exist in the memory 216 (i.e., the input small blocks to which the multiple reference points belong are not stored in the memory 216, hereinafter referred to as situation (2)). Therefore, if the reference point corresponding to the next output point is not in the memory 216 (situation (2)), the result of step S960 is no; conversely, if the reference point corresponding to the next output point is in the memory 216 (situation (1)), the result of step S960 is yes. The grid processing circuit 218 continues to execute step S960 and step S965 until the result is no.
[0099] Step S965 : the grid processing circuit 218 stores the output points in the memory 216 , that is, accumulates the output points in the memory 216 .
[0100] Step S970 : the DMA circuit 212 stores the accumulated output points (including the current output point) to the external memory 220 .
[0101] Step S980: The grid processing circuit 218 determines whether all the current output blocks have been written to the external memory 220. If not, step S950 is executed; if yes, step S990 is executed.
[0102] Step S990: The grid processing circuit 218 determines whether all grid blocks (ie, grid blocks GT0-GT3) have been traversed. If not, step S920 is executed; if yes, step S995 is executed.
[0103] Step S995: The grid processing circuit 218 determines whether all input small blocks (ie, input small blocks IT0-IT3) have been traversed. If not, step S910 is executed; if yes, the process ends.
[0104] Steps S950 to S980 are steps for storing output small blocks. By accumulating output points with continuous memory addresses, the direct memory access circuit 212 can continuously write out output data, avoiding fragmented access to the external memory 220. This can improve the efficiency of the direct memory access circuit 212 in writing out data and save memory bandwidth.
[0105] Fig. 9The process includes an outer loop (steps S910 to S995) and an inner loop (steps S920 to S990). The outer loop processes the input small blocks, while the inner loop processes the grid small blocks. In other words, each grid small block will be loaded into the memory 216 multiple times, because each time an input small block is processed, all the grid small blocks will be loaded into the memory 216 in sequence. Because the amount of data in the grid small block is smaller than the input small block, such a process will consume less memory bandwidth (compared to the inner loop processing the input small block and the outer loop processing the grid small block).
[0106] right Figure 3 For the deformable convolution operation 300, the grid processing circuit 218 first performs interpolation calculation (grid sampling operator 310) in step S940, and then multiplies the interpolation result by the mask MSK (multiplication operator 320). Figure 4 Deformable convolution operation 400 and Figure 8 For the deformable convolution operation 800, the grid processing circuit 218 first multiplies the interpolation coefficient and the mask MSK to generate a product in step S940, and then performs interpolation calculation based on the product (grid sampling operator 410 or grid sampling operator 830). Therefore, compared with the deformable convolution operation 300, the deformable convolution operation 400 and the deformable convolution operation 800 can reduce the amount of calculation and the calculation time.
[0107] In summary, by decomposing the deformable convolution operation into a grid sampling operation and a general convolution operation, the execution efficiency of the deformable convolution operation can be improved (including but not limited to reducing the bandwidth requirement for the external memory), and can be executed by a relatively low-cost application-specific integrated circuit (ASIC) (e.g., an intelligent processing unit).
[0108] Although the embodiments of the present application are described above, these embodiments are not intended to limit the present application. Technicians in this technical field may change the technical features of the present application based on the explicit or implicit contents of the present application. All these changes may fall within the scope of patent protection sought by the present application. The scope of patent protection of the present application shall be based on the scope of the patent application of this specification.
Claims
1. An intelligent processing unit, characterized in that: include: a memory for storing a portion of a first input data of a deformable convolution operation, a portion of a bias of the deformable convolution operation, a portion of a weight of the deformable convolution operation, and a portion of a grid, the grid being converted from an offset of the deformable convolution operation; a grid processing circuit, coupled to the memory, for performing a grid sampling operation to generate a second input data based on the first input data and the grid; and a convolution calculation circuit, coupled to the memory, for performing a convolution operation on the second input data, the weight and the bias to generate an output data; Wherein, the output data is substantially equal to the result of the deformable convolution operation.
2. The intelligent processing unit according to claim 1, characterized in that The offset is a variable, and the grid processing circuit further performs a conversion process to convert the offset into the grid.
3. The intelligent processing unit according to claim 2, characterized in that The conversion process includes the following steps: reshaping the offset to generate a first data; Transposing the first data to generate second data; Adding a first constant to the second data to generate a third data; Multiplying the third data by a second constant to generate a fourth data; A third constant is added to the fourth data to generate a result, and the result is reshaped to generate the grid.
4. The intelligent processing unit according to claim 1, characterized in that: The convolution calculation circuit performs a reshaping operation on the weights before performing the convolution operation.
5. The intelligent processing unit according to claim 1, characterized in that: The memory stores an input block of the first input data and a grid block of the grid, and the grid processing circuit performs the following steps to generate the second input data: According to a target point of the second input data, querying a plurality of reference points of the input small block to be used in the small grid block; as well as An output point of the second input data and a coordinate of the output point are calculated according to the multiple reference points.
6. The intelligent processing unit according to claim 5, characterized in that The grid processing circuit includes an interpolation calculation circuit. The interpolation calculation circuit performs an interpolation calculation based on an interpolation method to generate the output point.
7. The intelligent processing unit according to claim 6, characterized in that The grid processing circuit generates an interpolation coefficient, the interpolation calculation circuit multiplies the interpolation coefficient by a mask of the deformable convolution operation to generate a product, and performs the interpolation calculation based on the product.
8. The intelligent processing unit according to claim 5, characterized in that: The intelligent processing unit is coupled to an external memory, and the grid processing circuit further performs the following steps: Calculate an address of the output point in the external memory according to the coordinates; and When a next output point of the output point is not continuous with the output point in the external memory, the output point is stored in the external memory.
9. The intelligent processing unit according to claim 5, characterized in that: The intelligent processing unit is coupled to an external memory, and the grid processing circuit further performs the following steps: Calculate an address of the output point in the external memory according to the coordinates; and When a next output point of the output point is continuous with the output point in the external memory, the output point is stored in the memory.
10. A deformable convolution operation method, characterized in that: Executed in an intelligent processing unit, the calculation method includes: performing a grid sampling operation to generate a second input data based on a grid and a first input data of a deformable convolution operation, wherein the grid is transformed by an offset of the deformable convolution operation; and Performing a convolution operation on the second input data, a weight of the deformable convolution operation, and a bias of the deformable convolution operation to generate an output data; Wherein, the output data is substantially equal to the result of the deformable convolution operation.
11. The calculation method according to claim 10, characterized in that: The offset is a variable, and the calculation method further includes: A conversion process is performed to convert the offset into the grid.
12. The calculation method according to claim 11, characterized in that: The conversion process includes the following steps: reshaping the offset to generate a first data; Transposing the first data to generate second data; Adding a first constant to the second data to generate a third data; Multiplying the third data by a second constant to generate a fourth data; A third constant is added to the fourth data to generate a result, and the result is reshaped to generate the grid.
13. The calculation method according to claim 10, characterized in that: Also includes: A reshaping operation is performed on the weights before performing the convolution operation.
14. The calculation method according to claim 10, characterized in that: The first input data includes an input block, the grid includes a grid block, and the operation of generating the second input data includes the following steps: According to a target point of the second input data, querying a plurality of reference points of the input block to be used in the grid block; and An output point of the second input data and a coordinate of the output point are calculated according to the multiple reference points.
15. The calculation method according to claim 14, characterized in that: Also includes: An interpolation calculation is performed based on an interpolation method to generate the output point.
16. The calculation method according to claim 15, characterized in that: Also includes: multiplying an interpolation coefficient by a mask of the deformable convolution operation to generate a product; as well as The interpolation calculation is performed based on the product.
17. The calculation method according to claim 14, characterized in that: The intelligent processing unit is coupled to an external memory, and the calculation method further includes: Calculate an address of the output point in the external memory according to the coordinates; and When a next output point of the output point is not continuous with the output point in the external memory, the output point is stored in the external memory.
18. The calculation method according to claim 14, characterized in that: The intelligent processing unit includes a memory and is coupled to an external memory. The calculation method further includes: Calculate an address of the output point in the external memory according to the coordinates; and When a next output point of the output point is continuous with the output point in the external memory, the output point is stored in the memory.