Method, computing device, medium and program product for region of interest alignment

Abstract

The invention relates to a method, a computing device, a medium and a program product for region-of-interest alignment. The method comprises: at a host end, traversing each region of interest in an input feature map so as to generate a forward mapping relationship indicating which input sampling points each output point in an output feature map is calculated from; based on the forward mapping relation, a reverse remapping relation between the input feature gradient data and the output feature gradient data is constructed, and the reverse remapping relation indicates which output feature gradient points the input feature gradient data are calculated; and calculating, via the device end, an input feature gradient result based on the acquired output feature gradient data and the reverse remapping relationship. According to the method and the device, even if hardware which does not support efficient atomic operation or a native atomic operation instruction does not exist, a back propagation processing flow of region-of-interest alignment can be effectively realized.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of artificial intelligence, and more particularly to a method, a computing device, a computer-readable storage medium, and a computer program product for aligning a region of interest. Background Art

[0002] Region of Interest Alignment (RoIAlign) is a commonly used feature extraction method in target detection and instance segmentation tasks. In the early days, the feature extraction of target detection and instance segmentation tasks was often done by Region of Interest Pooling (RoIPool). Traditional methods for ROI alignment are, for example, converting a feature map of a region of interest (RoI) of any size into a small feature map of a fixed size. Specifically, the scheme generally includes a forward propagation process and a back propagation process. The back propagation process specifically includes: using atomic operation instructions, calculating the contribution of each sampling point gradient to the surrounding four input feature gradient data (i.e., grad_in) through atomic add instructions to ensure data consistency when there is read-write competition between threads. However, in the above-mentioned traditional ROI alignment method, on the one hand, the back propagation process of ROI alignment implemented by atomic operations is often not very friendly to hardware that does not support efficient atomic operations; on the other hand, there may be no native atomic operation instruction support, and the use of other implementation methods (for example, the implementation method of atomic addition using atomicCAS) is prone to poor performance.

[0003] In summary, the disadvantages of the traditional method for ROI alignment are that it is difficult to effectively implement the reverse processing flow of ROI alignment implemented by atomic operations for hardware that does not support efficient atomic operations or has no native atomic operation instructions. Summary of the invention

[0004] The present invention provides a method, a computing device, a computer-readable storage medium and a computer program product for aligning regions of interest, which can effectively implement the reverse propagation processing flow of region of interest alignment even for hardware that does not support efficient atomic operations or has no native atomic operation instructions.

[0005] According to a first aspect of the present invention, a method for aligning a region of interest is provided. The method comprises: on a host side, traversing each region of interest in an input feature map to generate a forward mapping relationship, wherein the forward mapping relationship indicates which input sampling points each output point in the output feature map is calculated from; based on the forward mapping relationship, constructing a reverse remapping relationship between input feature gradient data and output feature gradient data, wherein the reverse remapping relationship indicates which output feature gradient points the input feature gradient data is calculated from; and, via a device side, calculating an input feature gradient result based on the acquired output feature gradient data and the reverse remapping relationship.

[0006] In some embodiments, calculating the input feature gradient result based on the acquired output feature gradient data and the reverse remapping relationship includes: determining a remapping vector and a remapping index based on the reverse remapping relationship; sending the calculated remapping vector and remapping index to the device side by the host side; and calculating the input feature gradient result based on the output feature gradient data, the remapping vector and the remapping index via the device side.

[0007] In some embodiments, the remapping vector is a linear expansion of the output feature gradient points corresponding to the input feature gradient data whose value is not "0" in the reverse remapping relationship, and the remapping index includes a start index and an end index.

[0008] In some embodiments, determining a remapping vector and a remapping index includes: linearly expanding the output feature gradient points corresponding to the input feature gradient data whose values ​​are not "0" to generate a remapping vector; and generating a remapping index so that the remapping index indicates the position of the output feature gradient point corresponding to the input feature gradient data in the remapping vector.

[0009] In some embodiments, based on the output feature gradient data, the remapping vector and the remapping index, calculating the input feature gradient result includes: loading the output feature gradient data of a predetermined number of channels from the global memory to a cache, wherein the size of the output feature gradient data of the predetermined number of channels is less than or equal to the size of the cache; using all threads in the thread bundle, simultaneously loading the remapping indexes corresponding to the input feature gradient data of the predetermined number of threads from the global memory to the first register; starting traversal based on the corresponding index range indicated by the remapping index loaded in the first register; for each traversal, loading the output feature gradient coordinates and the corresponding weights corresponding to the remapping vector from the global memory to the second register; and for each loading of the output feature gradient coordinates and the corresponding weights corresponding to the remapping vector to the second register, each thread loads the output feature gradient data of the corresponding position from the cache to the third register according to the output feature gradient coordinates, so as to multiply the output feature gradient data loaded to the third register with the weight loaded to the second register, thereby accumulating the multiplication result to the input feature gradient result register of the corresponding channel.

[0010] In some embodiments, starting traversal based on the corresponding index range indicated by the remapping index loaded in the first register includes: initializing the input feature gradient result register corresponding to each thread to "0"; and starting traversal based on the corresponding index range indicated by the remapping index loaded in the first register.

[0011] In some embodiments, calculating the input feature gradient result based on the output feature gradient data, the remapping vector and the remapping index also includes: after the traversal is completed, writing the input feature gradient result from the input feature gradient result register to the corresponding position of the input feature gradient data in the global memory.

[0012] In some embodiments, starting the traversal based on the corresponding index range indicated by the remapping index loaded in the first register further includes: completing the update of input feature gradient data on all channels.

[0013] According to a second aspect of the present invention, a computing device is also provided. The computing device comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the computing device can perform the method of the first aspect of the present invention.

[0014] According to a third aspect of the present invention, a computer-readable storage medium is further provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a machine, the method of the first aspect of the present invention is executed.

[0015] According to a fourth aspect of the present invention, there is further provided a computer program product, comprising a computer program, wherein when the computer program is executed by a machine, the method of the first aspect of the present invention is performed.

[0016] The beneficial effects of the present invention include at least: since the device side can directly obtain the input feature gradient data and all the output feature gradient points required for its calculation based on the constructed reverse remapping relationship, each input feature gradient data can be directly calculated without using traditional atomic operations on the device side. Therefore, the present invention can effectively implement the back propagation processing flow of the alignment of the region of interest even for hardware that does not support efficient atomic operations or has no native atomic operation instructions.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0019] Figure 1 A schematic diagram of a conventional method for aligning a region of interest is shown schematically.

[0020] Figure 2 The figure schematically shows a schematic diagram of a computing device implementing a method for aligning regions of interest according to an embodiment of the present invention.

[0021] Figure 3 A flow chart of a method for aligning regions of interest according to an embodiment of the present invention is shown.

[0022] Figure 4 A schematic diagram showing a forward mapping relationship and a reverse remapping relationship according to some embodiments of the present invention is shown.

[0023] Figure 5 A schematic diagram of a remapping vector and a remapping index according to some embodiments of the present invention is shown.

[0024] Figure 6 A flowchart of a method for determining a remapping vector and a remapping index according to an embodiment of the present invention is shown.

[0025] Figure 7 A schematic diagram of a method for calculating an input feature gradient result according to an embodiment of the present invention is shown.

[0026] Figure 8A flow chart of a method for calculating an input feature gradient result according to an embodiment of the present invention is shown.

[0027] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0029] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0030] As described above, Figure 1 A schematic diagram of a conventional method for aligning regions of interest is schematically shown. A conventional method for aligning regions of interest is, for example, to convert a region of interest (RoI) of any size in an input feature map into an output feature map of a fixed size. Specifically, the scheme generally includes a forward propagation process and a reverse process. Among them, the forward propagation process is mainly divided into the following four steps.

[0031] The first step is to traverse each region of interest (ie, RoI) to read the position information of the region of interest; and based on the position information, calculate the height (ie, roi_height) and width (ie, roi_width) of each region of interest.

[0032] Step 2: Based on the size of the region of interest (roi_height*roi_width), the input feature map corresponding to the region of interest is divided into multiple grids (the number of grids is, for example, PH*PW, where PH is the number of grids in the height direction and PW is the number of grids in the width direction), and the size of each grid is, for example: (roi_height / PH)*(roi_width / PW).

[0033] Step 3: Select a certain number of input sampling points from each grid, and the value of each input sampling point is obtained by bilinear interpolation based on the four input points around it.

[0034] Step 4: Average pooling is performed on all input sampling points in each grid to obtain the value of each grid.

[0035] It should be understood that, according to the forward propagation process, the reverse propagation process also mainly includes four steps. Among them, the first and second steps of the reverse propagation process are similar to the corresponding steps of the forward propagation process described above, and will not be repeated. The third and fourth steps of the reverse propagation process, for example, include:

[0036] Step 3: First perform the average pooling back-propagation process, that is, divide the output feature gradient data (grad_out) corresponding to each grid by the number of sampling points to obtain the sampling point gradient.

[0037] Step 4: Perform the back propagation process of bilinear interpolation again, that is, multiply the sampling point gradient obtained in the previous step by the weight of bilinear interpolation, and contribute it to the four input feature gradient data (grad_in) around the sampling point. For example, Figure 1 As shown, the gradient of the sampling point 110 contributes to the four input feature gradient data around the sampling point 110 , that is, contributes to the four input feature gradient data indicated by marks 120 , 122 , 124 , and 126 .

[0038] As analyzed above, in the back propagation process, in order to perform the back propagation process of bilinear interpolation, the atomic add instruction is used to calculate the contribution of each sampling point gradient to the surrounding four input feature gradient data (i.e., grad_in) to ensure data consistency when there is read-write competition between threads. However, on the one hand, the back propagation process of region of interest alignment implemented by atomic operations is often not very friendly to hardware that does not support efficient atomic operations; on the other hand, there may be no native atomic operation instruction support, and the use of other implementation methods (for example, the implementation of atomic add using atomicCAS) is likely to lead to poor performance.

[0039] In summary, the shortcomings of the traditional method for ROI alignment are that it is difficult to effectively implement the ROI alignment back-propagation processing flow implemented by atomic operations for hardware that does not support efficient atomic operations or has no native atomic operation instructions.

[0040] In order to at least partially solve one or more of the above problems and other potential problems, an example embodiment of the present invention proposes a solution for aligning regions of interest. In this solution, a reverse remapping relationship is constructed on the host side based on the forward mapping relationship to indicate which output feature gradient points the input feature gradient data is calculated from; and the input feature gradient data is calculated based on the acquired output feature gradient data and the reverse remapping relationship via the device side. The result is on the device side. Since the device side can directly obtain the input feature gradient data and all the output feature gradient points required for its calculation based on the constructed reverse remapping relationship, each input feature gradient data can be directly calculated on the device side without using traditional atomic operations. Therefore, the present invention can effectively implement the reverse propagation processing flow of region of interest alignment even for hardware that does not support efficient atomic operations or does not have native atomic operation instructions.

[0041] Figure 2 A schematic diagram of a computing device 200 for implementing a method for aligning regions of interest according to an embodiment of the present invention is shown schematically. Figure 2 As shown, the computing device 200 may have one or more processing units, including special processing units such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a general-purpose graphics processing unit (GPGPU), and a general-purpose processing unit such as a CPU. The computing device 200 also includes at least: a forward mapping relationship generation unit 202, a reverse remapping relationship construction unit 204, and an input feature gradient result calculation unit 206. It should be understood that the forward mapping relationship generation unit 202, the reverse remapping relationship construction unit 204, and the input feature gradient result calculation unit 206 can be software modules, which, for example, run on one or more processing units configured by the computing device 200.

[0042] Regarding the forward mapping relationship generating unit 202, it is used to traverse each region of interest in the input feature map on the host side to generate a forward mapping relationship, wherein the forward mapping relationship indicates which input sampling points each output point in the output feature map is calculated from.

[0043] The reverse remapping relationship construction unit 204 is used to construct a reverse remapping relationship between input feature gradient data and output feature gradient data based on the forward mapping relationship, wherein the reverse remapping relationship indicates which output feature gradient points the input feature gradient data is calculated from.

[0044] Regarding the input feature gradient result calculation unit 206, it is used to calculate the input feature gradient result based on the acquired output feature gradient data and the reverse remapping relationship via the device side.

[0045] The following will be combined Figure 3 , Figure 4 , Figure 5 A method 300 for aligning regions of interest according to an embodiment of the present invention is described. Figure 3 A flow chart of a method 300 for aligning regions of interest according to an embodiment of the present invention is shown. Figure 4 A schematic diagram showing a forward mapping relationship and a reverse remapping relationship according to some embodiments of the present invention is shown. Figure 5 Schematic diagram of remapping vectors and remapping indexes according to some embodiments of the present invention is shown. It should be understood that method 300 can be used, for example, in Figure 2 The method 300 is executed at the described computing device 200. The method 300 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0046] At step 302, the computing device 200 on the host side traverses each region of interest in the input feature map to generate a forward mapping relationship, wherein the forward mapping relationship indicates which input sampling points are used to calculate each output point in the output feature map.

[0047] Regarding the region of interest, it refers to the area to be processed on the processed input feature map in the form of a box, circle, ellipse, or irregular polygon in machine vision and image processing, which is called the region of interest, abbreviated as RoI (region of interest).

[0048] Regarding the forward mapping relationship, it is, for example, but not limited to, a forward mapping table. The forward mapping table indicates, for example: each output point identifier (or coordinate) in the output feature map and the input sampling point identifier (or coordinate) required for the calculation of the output point. It should be understood that it can also be other forms of expression that reflect the association relationship between each output point in the output feature map and the input sampling point required for the calculation of the output point. Assume that each output point is calculated from only one input sampling point, and each input sampling point is calculated from four input points in the input feature map via bilinear interpolation. In fact, the number of input sampling points used to calculate the output point depends on the sampling rate (sampling_ratio), and each output point may be calculated from a different number of input sampling points. The four input points corresponding to each input sampling point may also be repeated.

[0049] Regarding the method for generating a forward mapping relationship, it includes, for example: on the host side, the computing device 200 traverses the parameters of each region of interest, searches for each output point through which input sampling points are bilinearly interpolated and average pooled to calculate; and constructs a forward mapping relationship based on each output point and the input sampling points required for its calculation. Specifically, the computing device 200 divides the width and height of the region of interest into a number of grids; selects a certain number of input sampling points in each grid so that the value of each input sampling point is obtained by bilinear interpolation of the four input points around it; so that each grid represents an output point, and each output point is obtained by average pooling of these input sampling points. Thus, the computing device 200 obtains the corresponding relationship between each output point calculated by a number of input sampling points (the positions of the input sampling points may be repeated) and corresponding weights, thereby generating a forward mapping relationship based on the corresponding relationship. It should be understood that the specific method for generating a forward mapping relationship can also be other ways.

[0050] The following combination Figure 4 Specific example forward mapping relationship, such as Figure 4 As shown. The forward mapping relationship 400, for example, specifically indicates the identifier of each output point and the identifier of the input sampling point required for its calculation. For example, the first output point y0 is calculated by the first input sampling point x0, the second input sampling point x1, the third input sampling point x2, and the fourth input sampling point x3. The second output point y1 is calculated by the second input sampling point x1, the third input sampling point x2, the fourth input sampling point x3, and the fifth input sampling point x4. For another example, the fifth output point y4 is calculated by the first input sampling point x0, the third input sampling point x2, the fifth input sampling point x4, and the sixth input sampling point x5. The sixth output point y5 is calculated by the first input sampling point x0, the second input sampling point x1, the fourth input sampling point x3, and the fifth input sampling point x4.

[0051] Regarding the parameters of the region of interest, they include, for example: input feature map, region of interest position description tensor, output feature map size, spatial scale (spatial_scale), sampling rate (sampling_ratio), alignment flag (aligned), etc.

[0052] The input feature map is, for example, [N, C, H, W], where N represents the batch size, C represents the number of channels, H represents the height, and W represents the width. The batch size refers to the number of samples used each time the model weights are updated.

[0053] Regarding the region of interest position description tensor (rois, or "box position description tensor"), its shape is [K, 5]. K represents K regions of interest, 5 represents the batch index (batch_index), and the coordinates of each region of interest are, for example, (X1, X1, Y2, Y2), where (X1, Y1) and (X2, Y2) represent the coordinates of the upper left corner and lower right corner of the region of interest, respectively.

[0054] Regarding the output feature map size (output_size), it is, for example, [PH, PW], where PH represents the output height of the output feature map after pooling, and PW represents the output width of the output feature map after pooling.

[0055] Regarding the spatial scale (spatial_scale), it represents the scaling of the coordinates of the input feature map compared to the original image.

[0056] Regarding the sampling rate (sampling_ratio), it indicates the number of interpolated samples calculated for each output point after pooling. If the sampling rate is less than or equal to "0", that is, sampling_ratio<=0, an adaptive number of grid points is used, and the calculation formula for the number of samples is formula (1); if the sampling rate is greater than "0", that is, sampling_ratio>0, the calculation formula for the number of samples is formula (2).

[0057] ceil(roi_height / PH)*ceil(roi_width / PW)(1)

[0058] sampling_ratio*sampling_ratio(2)

[0059] In the above formulas (1) and (2), ceil(roi_height / PH) represents the height of the grid. ceil(roi_width / PH) represents the width of the grid. sampling_ratio represents the sampling rate. For example, if the sampling rate is 2, the number of samples for each grid is 4, that is, the number of samples in the horizontal direction is 2 and the number of samples in the vertical direction is 2. PH represents the output height of the output feature map after pooling, and PW represents the output width of the output feature map after pooling.

[0060] Regarding the alignment flag (aligned), it is a hyperparameter indicating whether a -0.5 pixel coordinate offset is added to the coordinates of the region of interest. For example, if the alignment flag indicates "true", a -0.5 pixel coordinate offset is added to the coordinates of the region of interest. If the alignment flag indicates "false", a -0.5 pixel coordinate offset is not added to the coordinates of the region of interest.

[0061] According to a predetermined algorithm (eg, RoIAlign) process, the number of sampling points corresponding to each pooled output point is roi_bin_grid_h*roi_bin_grid_w, where roi_bin_grid_h*roi_bin_grid_w is calculated according to the above formula (1) or (2) respectively, depending on whether the sampling ratio sampling_ratio<=0 or sampling_ratio>0 is satisfied.

[0062] Therefore, under the above parameters, the following formula (3) schematically shows the calculation method of the number of input sampling points corresponding to the output points in all output feature maps.

[0063]

[0064] In the above formula (3), PH represents the output height of the output feature map after pooling. PW represents the output width of the output feature map after pooling. C represents the number of channels. roi_bin_grid_h*roi_bin_grid_w represents the number of sampling points corresponding to each output point after pooling. Among them, roi_bin_grid_h represents the number of sampling points in the h direction. roi_bin_grid_w represents the number of sampling points in the w direction. K represents K regions of interest. At step 304, the computing device 200 constructs a reverse remapping relationship between the input feature gradient data and the output feature gradient data based on the forward mapping relationship, and the reverse remapping relationship indicates which output feature gradient points the input feature gradient data is calculated from.

[0065] Regarding the output feature gradient point, it can also be called "output feature gradient sampling point", which is the sampling point of the output feature gradient corresponding to each grid in the output feature map in the back propagation process. It should be understood that in the back propagation process, the output gradient corresponding to each grid is divided by the number of output feature gradient points (that is, the number of output feature gradient sampling points) to obtain the gradient of the output feature gradient point. The gradient of the output feature gradient point multiplied by the weight of the bilinear interpolation can contribute to the input feature gradient data (grad_in) around the output feature gradient point.

[0066] For example, the computing device 200 is based on Figure 4 The forward mapping relationship 400 in the figure constructs a reverse remapping relationship 410, which, for example, indicates which output feature gradient data (grad_out) sampling points each point of the input feature gradient data (grad_in) is calculated from during the back-propagation processing flow.

[0067] For example, according to Figure 4 From the first column of the forward mapping relationship 400, it can be seen that the first input sampling point x0 is involved in the calculation of the first output point y0, the fourth output point y3, the fifth output point y4, and the sixth output point y5. Therefore, it can be concluded that the first input feature gradient data dx0 is calculated by the first output feature gradient point dy0, the fourth output feature gradient point dy3, the fifth output feature gradient point dy4, and the sixth output feature gradient point dy5. Therefore, the first row in the reverse remapping relationship 410 indicates the reverse remapping relationship between the constructed first input feature gradient data dy0 and the output feature gradient points (dy0, dy3, dy4, dy5) required for its calculation. And so on, the final result is based on Figure 4 The forward mapping relationship 400 in , constructs the reverse remapping relationship 410.

[0068] like Figure 4 As shown, the first input feature gradient data dx0 is calculated from the first output feature gradient point dy0, the fourth output feature gradient point dy3, the fifth output feature gradient point dy4, and the sixth output feature gradient point dy5. The second input feature gradient data dx1 is calculated from the first output feature gradient point dy0, the second output feature gradient point dy1, and the sixth output feature gradient point dy5. The third input feature gradient data dx2 is calculated from the first output feature gradient point dy0, the second output feature gradient point dy1, the third output feature gradient point dy2, the fourth output feature gradient point dy3, and the fifth output feature gradient point dy4.

[0069] At step 306, the input feature gradient result is calculated via the device based on the acquired output feature gradient data and the reverse remapping relationship.

[0070] Regarding the input feature gradient result, it has the same shape as the input feature map, for example, and is used to input to the operator related to the gradient parameters of the model training.

[0071] The method for calculating the output input feature gradient result includes, for example: determining a remapping vector and a remapping index based on a reverse remapping relationship; sending the calculated remapping vector and remapping index to a device end by a host end; and calculating the input feature gradient result based on the output feature gradient data and the remapping vector and the remapping index via the device end. Figure 6 A detailed description is given and no further elaboration is given here.

[0072] Regarding the remapping vector, it is, for example, Figure 5 Indicated by the mark 500 in the figure. In some embodiments, the remapping vector 500 is, for example, a one-dimensional vector, which is a linear expansion of the output feature gradient points corresponding to the input feature gradient data whose value is not "0" in the inverse remapping relationship 410. In some embodiments, each element (e.g., coord_weight) included in the remapping vector 500 indicates the coordinates and weight of the output feature gradient point.

[0073] Regarding the remapping index, it includes, for example, a start index and an end index, and the start index and the end index are used for indication. Figure 5 As shown, the remapping index 510 records the index range of the output feature gradient point corresponding to each input feature gradient data (grad_in) in the remapping vector (remapping_vector). For example, the mark 512 indicates the starting index, and the value "0" of the starting index 512 represents the starting position of all the output feature gradient points (e.g., dy0, dy3, dy 4, dy5) corresponding to the first input feature gradient data dx0 in the remapping vector (remapping_vector), that is, the position of the first output feature gradient point dy0 in the remapping vector 500. The mark 514 indicates the ending index, and the value "3" of the ending index 514 represents the ending position of all the output feature gradient points (e.g., dy0, dy3, dy 4, dy5) corresponding to the first input feature gradient data dx0, that is, the position of the sixth output feature gradient point dy5 in the remapping vector 500.

[0074] In the above scheme, a reverse remapping relationship is constructed on the host side based on the forward mapping relationship to indicate which output feature gradient points the input feature gradient data is calculated from; and the input feature gradient result is calculated on the device side based on the acquired output feature gradient data and the reverse remapping relationship. Since the device side can directly obtain the input feature gradient data and all the output feature gradient points required for its calculation based on the constructed reverse remapping relationship, each input feature gradient data can be directly calculated on the device side without using traditional atomic operations. Therefore, the present invention can effectively implement the reverse propagation processing flow of region of interest alignment even for hardware that does not support efficient atomic operations or does not have native atomic operation instructions.

[0075] The following will be combined Figure 6 A method 600 for determining a remapping vector and a remapping index according to an embodiment of the present invention is described. Figure 6 FIG. 6 is a flowchart of a method 600 for determining a remapping vector and a remapping index according to an embodiment of the present invention. It should be understood that the method 600 may be implemented, for example, in Figure 2 The method 600 is executed at the described computing device 200. The method 600 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0076] It should be understood that in some embodiments, Figure 4 As shown, for each input feature gradient data, the number of output feature gradient points required for calculation is not equal, that is, the data structure of the reverse remapping relationship has row vectors of unequal length. It should be understood that the device may not support data structures with unequal row vector lengths. Therefore, it is necessary to determine the remapping vector and remapping index based on the reverse remapping relationship.

[0077] At step 602, the computing device 200 determines a remapping vector and a remapping index based on the inverse remapping relationship.

[0078] For example, the host side of the computing device 200 linearly flattens the output feature gradient points corresponding to the input feature gradient data whose values ​​are not "0" to generate a remapping vector; and generates a remapping index so that the remapping index indicates the position of the output feature gradient point corresponding to the input feature gradient data in the remapping vector.

[0079] At step 604, the host side of the computing device 200 sends the calculated remapping vector and remapping index to the device side.

[0080] At step 606, the device side of the computing apparatus 200 calculates the input feature gradient result based on the output feature gradient data, the remapping vector, and the remapping index.

[0081] Regarding the method for calculating the input feature gradient result, it includes, for example: the device side loads the output feature gradient data of a predetermined number of channels from the global memory to the cache, and the size of the output feature gradient data of the predetermined number of channels is less than or equal to the size of the cache; using all threads in the thread bundle, the remapping index corresponding to the input feature gradient data of the predetermined number of threads is simultaneously loaded from the global memory to the first register; traversal is started based on the corresponding index range indicated by the remapping index loaded in the first register; for each traversal, the output feature gradient coordinates and the corresponding weights corresponding to the remapping vector are loaded from the global memory to the second register; and for each loading of the output feature gradient coordinates and the corresponding weights corresponding to the remapping vector to the second register, each thread loads the output feature gradient data of the corresponding position from the cache to the third register according to the output feature gradient coordinates, so as to multiply the output feature gradient data loaded to the third register with the weight recorded in the second register, thereby accumulating the multiplication result to the input feature gradient result register of the corresponding channel. The following will be combined with Figure 8 The method for calculating the input feature gradient result is described in detail and will not be repeated here.

[0082] By adopting the above means, the present invention can calculate the input feature gradient result with high performance even if the device side may not support the data structure of unequal length of row vectors of the reverse remapping relationship.

[0083] The following will be combined Figure 7 and Figure 8 A method 800 for calculating an input feature gradient result according to an embodiment of the present invention is described. Figure 7 A schematic diagram of a method for calculating an input feature gradient result according to an embodiment of the present invention is shown. Figure 8 FIG. 8 is a flowchart of a method 800 for calculating an input feature gradient result according to an embodiment of the present invention. It should be understood that the method 800 may be, for example, Figure 2 The method 800 is executed at the described computing device 200. The method 800 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0084] At step 802, the computing device 200 loads the output feature gradient data of the predetermined number of channels from the global memory to the cache, and the size of the output feature gradient data of the predetermined number of channels is less than or equal to the size of the cache. It should be understood that the present invention loads the output feature gradient data in advance through the cache, optimizes the small-grained memory access speed, and further improves the performance.

[0085] The size of the output feature gradient data with respect to the predetermined number of channels does not exceed the size of the cache space.

[0086] It should be understood that Figure 7 As shown, the size of the output feature gradient data 710 in the global memory is, for example, [K, C, PH, PW], which is generally very large. It should be understood that for any region of interest, the position of the shared box for all channels is the same. When bilinear interpolation is used to calculate the value of each sampling point, the weights of the four surrounding points are also shared. However, the space of the cache 730 is limited. Therefore, the output feature gradient data in the global memory 710 needs to be loaded into the cache 730 in batches, and only the output feature gradient data of a portion of the channels can be loaded into the cache 730 each time. Figure 7 As shown, the cache 730 is loaded with output feature gradient data 732 of a predetermined number of channels (eg, 2 channels).

[0087] At step 804 , the computing device 200 utilizes all threads in the warp to simultaneously load remapping indexes corresponding to input feature gradient data of a predetermined number of threads from the global memory to the first register.

[0088] The following is a method for calculating the number of bytes occupied by the remapping index in conjunction with formula (4).

[0089] N*2*H*W*sizeof(int)(4)

[0090] In the above formula (4), 2 represents two indexes, i.e., the start index and the end index, and the index is a 32-bit integer variable. N represents the batch size, H represents the height, and W represents the width. sizeof(int) represents the number of bytes of the 32-bit integer variable, i.e., sizeof(int) = 4 (bytes).

[0091] Regarding the predetermined number of threads, it is, for example, equal to the number of all threads in a warp. In some embodiments, the predetermined number of threads is, for example but not limited to, 32.

[0092] For example, the computing device 200 utilizes all threads in a warp to simultaneously load remapping indexes 720 corresponding to input feature gradient data of a predetermined number of threads from a global memory to a first register.

[0093] At step 806 , the computing device 200 starts traversal based on the corresponding index range indicated by the remapping index loaded in the first register.

[0094] like Figure 7 As shown, the start index 722 in the remapping index 720 indicates, for example, the starting position (as indicated by the dotted arrow 726) of the output feature gradient coordinates (i.e., the coordinates of the output feature gradient point) corresponding to a certain input feature gradient point in the remapping vector 712; the end index 724 indicates, for example, the ending position (as indicated by the dotted arrow 728) of the output feature gradient coordinates corresponding to a certain input feature gradient point in the remapping vector 712. The start index 722 and the end index 724 in the remapping index 720 are used to indicate the index range of the output feature gradient coordinates corresponding to a certain input feature gradient point.

[0095] For example, the computing device 200 initializes the input feature gradient result register corresponding to each thread to “0”; and starts traversal based on the corresponding index range indicated by the remapping index loaded in the register.

[0096] Regarding the input feature gradient result register, it is used to store the input feature gradient result.

[0097] At step 808 , the computing device 200 loads the output feature gradient coordinates and the corresponding weights corresponding to the remapped vector from the global memory to the second register for each traversal.

[0098] Regarding the second register, it and the first register are, for example, different registers in the same thread.

[0099] The following is a method for calculating the number of bytes occupied by the remapping vector in conjunction with formula (5).

[0100]

[0101] In the above formula (5), 4 represents that each sampling point is calculated by bilinear interpolation of 4 input points. coord_weight represents a data type similar to a structure, which records the output feature gradient coordinates and the corresponding weights. sizeof(coord_weight) represents the size of the gradient coordinates and the corresponding weights. The weight is, for example, the bilinear interpolation weight multiplied by 1 / (roi_bin_grid_h*roi_bin_grid_w).

[0102] It should be understood that the remapping vector 712 includes, in addition to the output feature gradient coordinates corresponding to the index range, also a weight 718 corresponding to the output feature gradient coordinates.

[0103] For example, the computing device 200 loads the remapping vector (remapping_vector) and the output feature gradient coordinates corresponding to the index range and the corresponding weight (weight) into the register (register) from the global memory based on the traversed index range.

[0104] like Figure 7 As shown, tag 712 is read from global memory ( Figure 7 The output feature gradient coordinates corresponding to the remapped vector are loaded in (not shown). The loaded, remapped vector corresponding output feature gradient coordinates are Figure 7 The output feature gradient coordinates between the marker 714 and the marker 716 correspond to the index range indicated by the start index 722 and the end index 724 in the remapping index 720. For example, the output feature gradient coordinates between the marker 714 and the marker 716 correspond to the index range indicated by the start index 722 and the end index 724 in the remapping index 720.

[0105] At step 810, the computing device 200 loads the output feature gradient coordinates and the corresponding weights corresponding to the remapped vector into the second register each time, and each thread loads the output feature gradient data of the corresponding position from the cache to the third register according to the output feature gradient coordinates, so as to multiply the output feature gradient data loaded into the third register with the weights loaded into the second register, thereby accumulating the multiplication result into the input feature gradient result register of the corresponding channel.

[0106] Regarding the third register, it is different from the first register and the second register. It should be understood that since each thread obtains the resources required for calculating the input feature gradient result (e.g., all output feature gradient data and weights required for calculation), there is no data competition with other threads.

[0107] For example, each thread (e.g., the third thread 742 in the thread warp 740) loads output feature gradient data of a certain number of channels at a corresponding position from the cache 730 according to the output feature gradient coordinates (k, ph, pw) loaded at step 808 (e.g., according to the output feature gradient coordinates between the markers 714 and 716). Figure 7 The output feature gradient data 732 of the predetermined number of channels shown in the figure (indicated by the four dotted arrows in the output feature gradient data 732) is loaded into the third register. Then, the third thread 742 multiplies the output feature gradient data loaded into the third register (as indicated by the mark 734) with the corresponding weight (as indicated by the mark 736) loaded into the second register at step 808, as indicated by the dotted box 754, thereby generating a multiplication result, and the multiplication result is accumulated to the input feature gradient result register (not shown) of the corresponding channel to generate the input feature gradient result 750.

[0108] In some embodiments, method 800 further includes: after the traversal is completed, writing the input feature gradient result from the register to the corresponding position of the input feature gradient data in the global memory; and completing the update of the input feature gradient data on all channels. Specifically, for example, the computing device 200 repeats steps 808 to 810 until the traversal is completed, and writes the result of the output feature gradient from the register to the corresponding position of the input feature gradient data 750 in the global memory, as indicated by the dotted line 752; then repeats steps 808 to 810 (for example, the dotted box 756 indicates that the third thread 742 multiplies the output feature gradient data of the next channel loaded into the register by the corresponding weight); and repeats the step of "writing the result of the output feature gradient from the register to the corresponding position of the input feature gradient data in the global memory" until the update of the input feature gradient data on all channels is completed.

[0109] In the above scheme, the present invention can optimize the data memory access speed, avoid data competition between threads, and further improve the performance.

[0110] The various processes and processing described above, such as methods 300, 600, and 800, may be performed at a computing device. The computing device includes, for example: at least one processor (at least one graphics processor and at least one central processing unit); and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor. In some embodiments, methods 300, 600, and 800 may be implemented as a computer software program or program product, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on a computing device via a read-only memory (ROM) and / or a communication unit. When the computer program is loaded into a random access memory (RAM) and executed by a GPU and a CPU, one or more actions of methods 300, 600, and 800 described above may be performed.

[0111] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present invention. The computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0112] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each square frame of the flow chart and / or block diagram and the combination of the square frames in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the central processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0114] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made according to design requirements and other factors.

Claims

1. A method for aligning a region of interest, characterized in that: include: On the host side, traverse each region of interest in the input feature map to generate a forward mapping relationship, wherein the forward mapping relationship indicates which input sampling points each output point in the output feature map is calculated from; Based on the forward mapping relationship, constructing a reverse remapping relationship between the input feature gradient data and the output feature gradient data, wherein the reverse remapping relationship indicates which output feature gradient points the input feature gradient data is calculated from; and The input feature gradient result is calculated via the device side based on the acquired output feature gradient data and the reverse remapping relationship.

2. The method according to claim 1, characterized in that: Based on the obtained output feature gradient data and the reverse remapping relationship, the input feature gradient result is calculated including: Based on the inverse remapping relationship, determining a remapping vector and a remapping index; The host sends the calculated remapping vector and remapping index to the device; and The input feature gradient result is calculated on the device side based on the output feature gradient data, the remapping vector and the remapping index.

3. The method according to claim 2, characterized in that The remapping vector is a linear expansion of the output feature gradient points corresponding to the input feature gradient data whose values ​​are not "0" in the reverse remapping relationship, and the remapping index includes a start index and an end index.

4. The method according to claim 2, characterized in that: Determining the remap vector and remap index involves: Linearly expand the output feature gradient points corresponding to the input feature gradient data whose values ​​are not "0" to generate a remap vector; and A remapping index is generated so that the remapping index indicates a position of an output feature gradient point corresponding to the input feature gradient data in the remapping vector.

5. The method according to claim 2, characterized in that: Based on the output feature gradient data, the remapping vector and the remapping index, the input feature gradient result is calculated including: Loading output feature gradient data of a predetermined number of channels from a global memory to a cache, wherein a size of the output feature gradient data of the predetermined number of channels is less than or equal to a size of the cache; Using all threads in the warp, simultaneously load remap indexes corresponding to input feature gradient data of a predetermined number of threads from a global memory to a first register; Starting traversal based on the corresponding index range indicated by the remap index loaded in the first register; For each traversal, load the output feature gradient coordinates corresponding to the remapped vector and the corresponding weight from the global memory to the second register; and For each loading of the output feature gradient coordinates and the corresponding weights corresponding to the remapping vector into the second register, each thread loads the output feature gradient data of the corresponding position from the cache to the third register according to the output feature gradient coordinates, so as to multiply the output feature gradient data loaded into the third register with the weights loaded into the second register, thereby accumulating the multiplication result into the input feature gradient result register of the corresponding channel.

6. The method according to claim 5, characterized in that Starting traversal based on the corresponding index range indicated by the remap index loaded in the first register includes: Initialize the input feature gradient result register corresponding to each thread to "0"; and The traversal is started based on the corresponding index range indicated by the remap index loaded in the first register.

7. The method according to claim 5, characterized in that Based on the output feature gradient data, the remapping vector and the remapping index, the input feature gradient result is calculated and further includes: After the traversal is completed, the input feature gradient result is written from the input feature gradient result register to the corresponding position of the input feature gradient data in the global memory.

8. The method according to claim 7, characterized in that Starting traversal based on the corresponding index range indicated by the remap index loaded in the first register also includes: Complete the update of input feature gradient data on all channels.

9. A computing device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a machine, the method according to any one of claims 1 to 8 is performed.

11. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a machine, performs the method according to any one of claims 1 to 8.

Images