Deformable convolution acceleration method, electronic device, storage medium and program product

Abstract

The application relates to the technical field of artificial intelligence, and provides a deformable convolution acceleration method, an electronic device, a storage medium and a program product, which comprise the following steps: acquiring an input feature map, an offset tensor and an original weight tensor; performing interpolation sampling on the input feature map according to the offset tensor, splicing and unfolding the sampling results belonging to the same spatial position in the channel dimension to generate an intermediate tensor which is stored continuously; performing shape transformation on the original weight tensor to obtain a target weight tensor whose spatial dimension size is one by one and whose channel dimension matches the intermediate tensor; and performing one-by-one convolution operation based on the intermediate tensor and the target weight tensor to obtain an output feature map. Through the collaborative design of data continuous arrangement and weight shape rearrangement, the application converts discrete data calculation into efficient one-by-one convolution operation which supports continuous data input, so that the hardware high-performance computing unit can be fully utilized, and therefore the utilization rate of hardware computing resources and the operator execution performance are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a deformable convolution acceleration method, electronic device, storage medium, and program product. Background Technology

[0002] Convolutional neural networks have been widely used in artificial intelligence fields such as computer vision. Traditional convolution operations use kernels with fixed shapes for local feature extraction, but their adaptability to complex structures such as deformation and rotation of objects in images is limited. To improve the network's adaptability to complex structures such as object deformation, related technologies have introduced deformable convolution, which introduces dynamic offsets to the sampling points of the convolution kernel, enabling the network to adapt to the spatial distribution of the input data.

[0003] Since the offset is typically a decimal, current deformable convolution operations require calculating the approximate pixel value at the offset position using bilinear interpolation before performing the convolution operation. However, because the offset value is dynamic and random, the pixels sampled during interpolation are completely discrete in memory space. This data discreteness prevents data reuse between adjacent sampling points, significantly increasing memory access overhead. Furthermore, the lack of continuity in the input data prevents effective utilization of the high-performance computing units in the underlying AI chip optimized for continuous data, resulting in extremely low overall computational efficiency for the operator. Summary of the Invention

[0004] This invention provides a deformable convolution acceleration method, electronic device, storage medium, and program product to solve the defect in deformable convolution where the discrete sampling data makes it impossible to effectively utilize high-performance hardware computing units, resulting in poor operator execution performance.

[0005] This invention provides a deformable convolution acceleration method, comprising: Obtain the input feature map, offset tensor, and original weight tensor; The input feature map is subjected to interpolation sampling based on the offset tensor, and the sampling results belonging to the same spatial position of the output feature map are spliced ​​and expanded in the channel dimension to generate a continuously stored intermediate tensor. The original weight tensor is subjected to a shape transformation operation to obtain a target weight tensor with a spatial dimension of one by one, and the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor. Perform a one-to-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of deformable convolution.

[0006] According to a deformable convolution acceleration method provided by the present invention, the step of performing interpolation sampling operation on the input feature map according to the offset tensor, and concatenating and unfolding the sampling results belonging to the same spatial position of the output feature map in the channel dimension to generate a continuously stored intermediate tensor includes: Traverse each spatial position of the output feature map and obtain the offset of the sampling point corresponding to each spatial position according to the offset tensor; Based on the offset of each sampling point and the coordinates of the input feature map, the actual pixel position corresponding to each sampling point is determined; Based on the actual pixel position corresponding to each sampling point, interpolation calculation is performed on the input feature map to obtain the interpolated pixel value of each sampling point. The interpolated pixel values ​​of each sampling point corresponding to the same spatial position of the output feature map are continuously arranged into the channel dimension corresponding to the intermediate tensor according to a preset order to obtain the intermediate tensor.

[0007] According to a deformable convolution acceleration method provided by the present invention, determining the actual pixel position corresponding to each sampling point based on the offset of each sampling point and the coordinates of the input feature map includes: The offset of the current sampling point is rounded down to obtain the integer base coordinates; Based on the coordinates of the spatial position currently traversed, the relative coordinates of the current sampling point in the convolution kernel, and the base coordinates, the coordinates of four adjacent pixels are calculated, and the coordinates of the four adjacent pixels are used as the actual pixel position corresponding to the current sampling point.

[0008] According to a deformable convolution acceleration method provided by the present invention, the step of performing interpolation calculation on the input feature map based on the actual pixel position corresponding to each sampling point to obtain the interpolated pixel value of each sampling point includes: Subtract the offset of the current sampling point from the base coordinate to obtain the decimal offset corresponding to the current sampling point; The bilinear interpolation weights corresponding to the coordinates of the four adjacent pixels are calculated using the fractional offset. Four valid pixel values ​​are obtained from the input feature map based on the coordinates of the four adjacent pixels. The interpolated pixel value of the current sampling point is obtained by weighting and summing the four valid pixel values ​​with their corresponding bilinear interpolation weights.

[0009] According to a deformable convolution acceleration method provided by the present invention, the step of performing a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one includes: Obtain the original shape information of the original weight tensor, wherein the original shape information includes the output channel dimension, the input channel dimension, and the original spatial dimension. While keeping the memory arrangement order of the original weight tensor unchanged, the product of the input channel dimension and the original spatial dimension is used as the new input channel dimension of the target weight tensor, and the spatial dimension of the target weight tensor is set to one-to-one to obtain the target weight tensor after shape transformation.

[0010] According to a deformable convolution acceleration method provided by the present invention, the step of concatenating and unfolding sampling results belonging to the same spatial location of the output feature map along the channel dimension to generate a continuously stored intermediate tensor includes: The product of the input channel dimension of the input feature map and the original spatial dimension of the original weight tensor is used as the channel dimension of the intermediate tensor. The spatial dimension of the output feature map is used as the spatial dimension of the intermediate tensor. The sampling results of each sampling point on each channel are written into consecutive memory addresses according to the shape information of the intermediate tensor.

[0011] According to a deformable convolution acceleration method provided by the present invention, before the step of performing a one-to-one convolution operation based on the intermediate tensor and the target weight tensor, the method further includes: The generated intermediate tensor is written to the on-chip tensor buffer; The step of performing a one-to-one convolution operation based on the intermediate tensor and the target weight tensor includes: The intermediate tensor data is continuously read from the on-chip tensor buffer, and the read intermediate tensor data and the target weight tensor are used as continuous input data to be input into the hardware computing unit to perform a one-to-one convolution operation.

[0012] The present invention also provides a deformable convolution acceleration device, comprising: The parameter acquisition unit is used to acquire the input feature map, offset tensor, and original weight tensor. The interpolation expansion unit is used to perform interpolation sampling operation on the input feature map according to the offset tensor, and to splice and expand the sampling results belonging to the same spatial position of the output feature map in the channel dimension to generate a continuously stored intermediate tensor. The weight rearrangement unit is used to perform a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one, wherein the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor. The convolution operation unit is used to perform a one-to-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of deformable convolution.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the deformable convolution acceleration method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deformable convolution acceleration method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the deformable convolution acceleration method as described above.

[0016] The deformable convolution acceleration method, electronic device, storage medium, and program product provided by this invention, after acquiring relevant parameters, addresses the problem of discrete sampling data caused by dynamic offsets. It generates a continuously stored intermediate tensor by splicing and unfolding discrete interpolation sampling results belonging to the same spatial location in the output feature map along the channel dimension. This step transforms the originally fragmented discrete memory access into continuous, regular memory access, thereby eliminating the performance bottleneck caused by data discontinuity in the underlying computation. Simultaneously, a shape transformation operation is performed on the original weight tensor to obtain a target weight tensor with a spatial dimension of one-to-one, ensuring that its rearranged channel dimension matches that of the intermediate tensor. This transforms the originally complex spatial convolution operation into an equivalent one-to-one convolution operation. Through this collaborative design of continuous data arrangement and weight shape rearrangement, this invention enables subsequent one-to-one convolution operations based on the intermediate and target weight tensors to directly call the high-performance computing units optimized for continuous data in the underlying AI chip, thereby improving the utilization of hardware computing resources and the execution efficiency of the entire deformable convolution operator. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating deformable convolution operations in related technologies; Figure 2This is a schematic diagram of the structure of the general-purpose graphics processor provided by the present invention; Figure 3 This is one of the flowcharts of the deformable convolution acceleration method provided by the present invention; Figure 4 This is the second flowchart of the deformable convolution acceleration method provided by the present invention; Figure 5 This is a schematic diagram of the deformable convolution acceleration device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Traditional convolution operations use fixed-shape kernels for local feature extraction, but their adaptability to complex structures such as deformation and rotation of objects in images is limited. To improve this, deformable convolution has been proposed. Deformable convolution is an improved method of traditional convolution operations. Its core idea is to introduce a dynamic offset to each sampling point in the convolution kernel, enabling the network to adapt to the spatial distribution of the input data, thereby improving the network's ability to model complex structures such as deformation and rotation.

[0021] In practical implementations, since the introduced offset is usually a floating-point decimal, the sampling position after adding the offset does not correspond to the actual discrete pixel point on the input feature map. Therefore, deformable convolution in related technologies needs to use methods such as bilinear interpolation to obtain the approximate pixel value of the offset position before performing the regular convolution operation. Specifically, it involves finding the four nearest actual pixels (i.e., the four-neighbor pixels) based on the offset decimal coordinates, calculating weights based on distances, performing a weighted sum to obtain the pixel value of the interpolated point, and then performing the subsequent convolution operation.

[0022] Figure 1 This is a flowchart illustrating deformable convolution operations in related technologies, such as... Figure 1 As shown, deformable convolution operations are typically implemented using a loop accumulation method. First, an accumulator is initialized (e.g., acc=0); then, the size of the current convolution kernel (let's assume it's R) is determined. If the loop (where R represents the number of sampling points in the height direction and S represents the number of sampling points in the width direction) has ended, then proceed with the following steps inside the loop: read the input feature map (denoted as input) and the offset (denoted as offset), perform bilinear interpolation on the input feature map based on the offset to obtain approximate pixel values ​​(denoted as a), load the corresponding filter weights (denoted as w) into the register, and then perform a convolution multiplication-accumulation operation (acc += a). w). When this R After all S iterations are completed, the result in the accumulator is written to the output tensor, thus completing the calculation for the current position.

[0023] However, the aforementioned deformable convolution implementation methods suffer from significant performance bottlenecks in practical computation. For conventional convolution operations, the sampling window slides regularly, resulting in extensive data reuse between adjacent input points. This allows for full utilization of core computational units in modern AI chips (such as matrix multiplication and addition units and tensor computation units) highly optimized for continuous data and regular memory access, thus significantly improving computational performance. However, in deformable convolution, the offset values ​​of each sampling point are dynamic and irregular (theoretically ranging from negative infinity to positive infinity). This leads to the four neighboring pixels accessed during bilinear interpolation being completely discrete and fragmented in memory space.

[0024] The high degree of dispersion of this data leads to two serious drawbacks: first, it prevents efficient reuse of data at the memory level, increasing memory access overhead and bandwidth pressure; second, due to the lack of regularity and continuity in the input data, it cannot utilize the higher-performance core computing units in the underlying hardware (which typically require continuous input data). Therefore, related deformable convolution algorithms often rely on general-purpose computing units to execute with low parallelism, resulting in low overall operator efficiency and failing to meet the urgent need for high-performance computing in modern deep learning models.

[0025] To address this, the present invention provides a deformable convolution acceleration method that splits the originally coupled deformable convolution operation into two independent stages: interpolation sampling and convolution operation. The interpolation sampling result is concatenated and unfolded along the channel dimension and stored in a continuously arranged intermediate tensor. When performing convolution operation, data is loaded from this intermediate tensor, thereby making the input data for the convolution operation continuous. This allows the high-performance computing units in artificial intelligence chips, which are highly optimized for continuous data, to perform convolution operations, thus overcoming the aforementioned shortcomings.

[0026] It should be noted that the execution entity of the deformable convolution acceleration method provided by this invention can be an artificial intelligence chip, such as a GPU (Graphics Processing Unit), GPGPU (General-purpose computing on Graphics Processing Units), TPU (Tensor Processing Unit), NPU (Neural Processing Unit), etc. The structure of the execution entity of this invention will be briefly described below using GPGPU as an example.

[0027] Figure 2 This is a schematic diagram of the structure of the general-purpose graphics processor provided by the present invention, as shown below. Figure 2 As shown, this general-purpose graphics processing unit (GPGPU) includes at least multiple streaming processor clusters (SPCs) and global memory. Each SPC handles one computational task, or multiple SPCs collaboratively process a computational task, with data shared between SPCs via global memory. Each SPC includes multiple compute units (CUs) and intermediate caches (such as L1 caches) for sharing data among the compute units. Each compute unit includes multiple compute cores, which are used to execute specific computational tasks.

[0028] Each computing unit also includes registers, shared caches, or shared memory for hierarchical storage of data related to computing tasks. Shared caches or shared memory within a computing unit are used to share data between the computing cores of that unit. Here, a shared cache (also known as an on-chip cache) is a temporary memory, for example, a Gemm Main Buffer (GMB), which has a smaller capacity than an L1 cache but a faster data exchange speed. In contrast, registers have a smaller capacity than shared caches but a faster data exchange speed. Specifically, registers can be thread-local registers (TLRs).

[0029] GPGPUs may also include an L2 cache, a high-speed cache located between global memory and L1 cache, used to reduce global memory access latency. The L2 cache is typically a shared resource across the entire GPGPU chip, rather than being private to each SPC; multiple SPCs can share data through the L2 cache.

[0030] Furthermore, the computational units in SPC are divided into tensor computation units and vector computation units. Tensor computation units perform tensor computations, such as matrix multiplication and convolution. Within a tensor computation unit, the computational core, also called a tensor core (tcore), is used to perform specific computational tasks. Vector computation units primarily perform arithmetic and logical operations other than tensor computations such as matrix multiplication and convolution, including accumulation, reduction, and regular addition, subtraction, multiplication, and division. Within a vector computation unit, the computational core, also called a vector core (vcore), is used to perform specific computational tasks.

[0031] In addition to the structures described above, the general-purpose graphics processor of this invention may also include other structures, and this invention does not specifically limit them.

[0032] While high-performance hardware units (such as the core computing unit tcore, specifically designed for convolution and matrix multiplication) can significantly improve operator performance, they typically require continuous input data. Traditional deformable convolutions, due to the discreteness of data sampling, often struggle to directly utilize such high-performance computing units. Therefore, this invention achieves efficient matching with the underlying hardware characteristics through algorithm-level reconstruction.

[0033] Furthermore, the deformable convolution acceleration method provided by this invention can be widely applied in image processing, signal processing, and other fields. It is mainly used to implement deformable convolution operations between input data and convolution kernels. The input data can take various forms, depending on the application scenario. For example, when the method provided by this invention is applied to image processing tasks such as image classification, object detection, and image segmentation, the input data can be a single-channel grayscale image or a multi-channel color image. The convolution kernel can extract features such as edges and textures by sliding across the image.

[0034] Figure 3 This is one of the flowcharts illustrating the deformable convolution acceleration method provided by the present invention, such as... Figure 3 As shown, the method includes: Step S10: Obtain the input feature map, offset tensor, and original weight tensor.

[0035] Specifically, when performing feature extraction or feature fusion in a network model, the executing entity first obtains the various input data that participate in the current layer's computation. Among them, the input feature map is the input data of the current layer, and its shape can usually be represented as (N,C,H,W), where N represents the batch, C represents the number of input channels, H represents the height, and W represents the width.

[0036] The original weight tensor is the convolution kernel parameter of the current layer, and its shape is usually (K, C, R, S), where K represents the number of output channels (i.e., the number of convolution kernels, each generating one output channel), and R and S represent the number of sampling points of the convolution kernel in the height and width directions, respectively. For example, for a shape of 3... A convolution kernel of 3, R=3, S=3.

[0037] Meanwhile, since deformable convolution introduces a dynamic offset for each sampling point in the convolution kernel, the execution entity also needs to obtain the corresponding offset tensor (i.e., offset) to characterize this dynamic offset. Its shape can be represented as (N, 2). R S, P, Q). Here, P and Q represent the height and width of the output feature map, 2 R S represents R corresponding to each output position. The offset values ​​of S sampling points in both the height and width directions. It should be understood that in deformable convolution, each sampling point of the convolution kernel has its own independent offset. The feature map is a two-dimensional plane (height H and width W). Therefore, each sampling point needs two independent values ​​to describe its offset: one along the height and the other along the width. Thus, to calculate the result at a certain location on the output feature map, R is needed. There are S sampling points, and each sampling point requires 2 offset values, so a total of 2 are needed. R S offset values.

[0038] Understandably, in deformable convolution algorithms, different channels of the input feature map share offset tensor data. It's generally assumed that different channels of the feature map represent different features (such as edges, colors, textures, etc.) at the same spatial location. Since the deformation of an object is spatially consistent, all feature channels should experience the same geometric deformation at the same spatial location. Sharing offsets can significantly reduce the number of parameters and computational cost.

[0039] Step S20: Perform interpolation sampling operation on the input feature map according to the offset tensor, and splice and expand the sampling results of the same spatial position belonging to the output feature map in the channel dimension to generate a continuously stored intermediate tensor.

[0040] It should be noted that in traditional deformable convolution, the introduced offset does not correspond to actual pixels, resulting in discrete collected data that cannot be reused or utilized with high-performance computing units such as tcore. To address this, this embodiment of the invention decouples and separates interpolation sampling from convolution operations.

[0041] Specifically, firstly, based on the offset position indicated by the offset tensor, interpolation sampling is performed on the input feature map to obtain the corresponding pixel value. Further, R values ​​belonging to the same spatial location in the output feature map are... The sampling results of S sampling points on each channel C (i.e., the corresponding pixel values ​​obtained by sampling) are concatenated and expanded along the channel dimension (i.e., the channel dimension is magnified by R). S times), and the result is stored in an intermediate variable, thus generating a continuously stored intermediate tensor (which can be represented as the tmp tensor). The shape of the tmp tensor is transformed to (N, C). R The originally completely discrete interpolation results are thus continuously written into memory, transforming the irregular discrete data into data blocks with a continuous memory arrangement pattern.

[0042] To more clearly illustrate the effect of converting discrete memory access to continuous memory in this step, a specific example is used for comparison. In traditional deformable convolution operations, assuming the current operation is processing the position at coordinates (p, q) on the output feature map, and using 3... The convolution kernel has a value of 3 (R=3, S=3, for a total of 9 sampling points). Because each sampling point has a fractional offset, the addressing locations of these 9 sampling points in physical memory are completely random and scattered. For example, the first sampling point might need to read four pixels near memory address A for interpolation, while the second sampling point might jump directly to a more distant memory address B for interpolation. When the underlying hardware computing unit performs the convolution operation, it must frequently jump and read these 9 discrete data points throughout the memory space of the entire input feature map, resulting in a low cache hit rate and preventing the hardware's continuous memory block read mechanism from being triggered.

[0043] In this embodiment of the invention, through pre-calculation, the nine sampling results, whose positions are scattered due to offset, are pre-calculated and forced to be written continuously into a specific memory region of the tmp tensor in channel index order. Specifically, for the same output coordinate (p, q), its C R S (i.e., C) 9) The data are tightly laid out and stored in a contiguous memory address space. The originally completely discrete interpolation results are thus written to memory continuously, eliminating the randomness of memory access caused by dynamic offsets and transforming irregular discrete data into data blocks with a continuous memory arrangement pattern.

[0044] This transformation means that when performing convolution and operations, the underlying high-performance tensor computing core (i.e., tcore) no longer needs to address the entire graph. It only needs to initiate an efficient continuous data stream read for the tmp tensor, thereby greatly improving the utilization of the underlying hardware computing unit and memory access bandwidth.

[0045] Step S30: Perform a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one. The channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor.

[0046] Specifically, since the intermediate tensor has amplified the input data channels by R... S times (becomes C) R To ensure valid convolution execution, the shape of the convolution kernel needs to be adjusted synchronously. Therefore, the execution entity reshapes the original weight tensor of shape (K,C,R,S), that is, it enlarges the channels C of the original weight tensor by R. S times, at this point the target weight tensor obtained after the transformation becomes 1. 1. The convolution kernel, whose shape is transformed from (K,C,R,S) to (K,C) R Since the two shapes are arranged in memory in the exact same order, the shape transformation operation only requires a logical change to the shape. Because the input channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor, it is compatible with continuous matrix multiplication logic in existing high-performance operator libraries.

[0047] Step S40: Perform a one-to-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of deformable convolution.

[0048] Specifically, in preparing the intermediate tensor for contiguous storage and 1 After obtaining the target weight tensor of 1, the executing entity will transform the originally complex R... S-deformable convolution is equivalent to transforming continuous data into 1 1. Convolution. Here, 1 1-convolution refers to a convolution kernel size of 1. A convolution operation of 1 is equivalent to matrix multiplication and is often used for channel transformation or feature fusion.

[0049] Specifically, in executing 1 During convolution operations, the execution body directly loads continuous input data from the tmp tensor and performs a regular 1-step operation with the target weight tensor. 1. Convolution (equivalent to matrix multiplication) ultimately yields an output feature map of shape (N,K,P,Q).

[0050] The method provided in this invention, after acquiring relevant parameters, addresses the problem of discrete sampling data caused by dynamic offsets by splicing and unfolding discrete interpolation sampling results belonging to the same spatial location in the output feature map along the channel dimension, thereby generating a continuously stored intermediate tensor. This step transforms the originally fragmented discrete memory access into continuous regular memory access, thus eliminating the performance bottleneck caused by data discontinuity during underlying computation. Simultaneously, a shape transformation operation is performed on the original weight tensor to obtain a target weight tensor with a spatial dimension of one-to-one, ensuring that its rearranged channel dimension matches that of the intermediate tensor. This transforms the originally complex spatial convolution operation into an equivalent one-to-one convolution operation. Through this collaborative design of continuous data arrangement and weight shape rearrangement, this invention enables subsequent one-to-one convolution operations based on the intermediate and target weight tensors to directly call the high-performance computing units optimized for continuous data in the underlying AI chip, thereby improving the utilization of hardware computing resources and the execution efficiency of the entire deformable convolution operator.

[0051] Based on any of the above embodiments, step S20 specifically includes: Step S21: Traverse each spatial position of the output feature map and obtain the offset of the sampling point corresponding to each spatial position according to the offset tensor.

[0052] Specifically, the execution entity needs to traverse all batches (N), all input channels (C), and the height (P) and width (Q) of the output feature map through a multi-layered loop structure. For each spatial location on the output feature map (e.g., the location with coordinates (h, w)), the execution entity further traverses each location of the convolutional kernel (total R). There are S sampling points, and the index of the currently traversed sampling point is denoted as i. During the traversal, the execution body starts from the shape (N, 2). R In the offset tensor of S, P, Q, read the offset data of the two directions (height and width) of the corresponding sampling point i under the current batch and current spatial position, so as to obtain the offset of the sampling point at each spatial position.

[0053] Step S22: Determine the actual pixel position corresponding to each sampling point based on the offset of each sampling point and the coordinates of the input feature map.

[0054] Specifically, since the introduced offset is typically a dynamic value containing a decimal, the execution entity superimposes this offset onto the reference coordinates (i.e., the ideal position combining the current spatial position with the relative coordinates of the sampling point) to determine the actual pixel position corresponding to each sampling point. It should be understood that the position after introducing the offset usually has a decimal part and does not completely correspond to the integer actual physical pixel coordinates on the input feature map (N,C,H,W).

[0055] Step S23: Based on the actual pixel position corresponding to each sampling point, perform interpolation calculation on the input feature map to obtain the interpolated pixel value of each sampling point.

[0056] Specifically, since the actual pixel location may fall on non-integer coordinates, the pixel cannot be directly read. Therefore, an interpolation method (such as bilinear interpolation) is needed to calculate the pixel value at the non-integer coordinate location. For example, the interpolated pixel value (denoted as val) of each sampling point can be obtained by using a weighted average estimate of the surrounding actual pixels, thereby achieving smooth and continuous feature sampling.

[0057] Step S24: Arrange the interpolated pixel values ​​of each sampling point corresponding to the same spatial position of the output feature map into the channel dimension corresponding to the intermediate tensor in a preset order to obtain the intermediate tensor.

[0058] Specifically, after obtaining the interpolated pixel value for each sampling point, instead of immediately performing discrete multiplication and addition operations as in traditional calculations, the result is transferred and stored. Specifically, the execution entity, according to a preset order, transfers R values ​​belonging to the same spatial location... S sampling results are consecutively arranged into the channel dimension corresponding to the intermediate tensor (tmp tensor). For example, this can be achieved by setting the storage index to tmp(n,c). R This is achieved by S+i,h,w)=val, which amplifies the channels of the input tensor (i.e., the input feature map) by R. S times. Here, n represents the batch being traversed, and c represents the input channel being traversed. This means that at the same spatial location (h, w), the data of different channels c and their corresponding sampling points i are flattened and continuously written into the same contiguous memory address, ultimately resulting in an intermediate tensor.

[0059] The method provided in this invention obtains discrete sampling point data through multi-dimensional loop traversal and offset coordinate mapping, combined with interpolation calculation, and utilizes a preset index order (enlarging the channel dimension R). The S-fold convolution stores the interpolated pixel values, which were originally discretely distributed across the feature map, continuously into an intermediate tensor according to their spatial location and sampling point order. This breaks the randomness and discreteness of data access in deformable convolution, and constructs a highly regular continuous data block in memory, enabling subsequent direct calls to high-performance hardware computing units (such as Tcore) for efficient computation.

[0060] Based on any of the above embodiments, step S22 specifically includes: Step S221: Round down the offset of the current sampling point to obtain the integer base coordinates; Step S222: Based on the coordinates of the spatial position currently traversed, the relative coordinates of the current sampling point in the convolution kernel, and the base coordinates, calculate the coordinates of four adjacent pixels, and use the coordinates of the four adjacent pixels as the actual pixel position corresponding to the current sampling point.

[0061] Specifically, after obtaining the floating-point offset of the current sampling point (e.g., vector2 off, where vector2 is a data structure used to represent a two-dimensional vector), in order to locate the point in the discrete pixel grid, the offset of the current sampling point first needs to be rounded down. For example, the floor function can be used, such as h_low=floor(off.x) and w_low=floor(off.y), to obtain the integer base coordinates (h_low, w_low).

[0062] After determining the integer base coordinates, the integer coordinates (_h,_w) of the top-left corner of the target position after offset can be calculated by combining the spatial coordinates (h, w) of the output feature map of the current traversal, the relative coordinates of the current sampling point in the convolution kernel (calculated using index i and kernel size S), and the aforementioned base coordinates (h_low, w_low). The specific calculation formula is as follows: _h = (int)(h + i / S – 1 + h_low); _w=(int)(w+i%S–1+w_low); Where i / S–1 and i%S–1 represent the relative coordinates of the current sampling point i in the convolution kernel.

[0063] Subsequently, the executing entity adds 1 to each of these values ​​to calculate the integer coordinates (_h1, _w1) of the lower right corner of the target position after the offset, i.e., _h1 = _h + 1, _w1 = _w + 1. This yields four coordinate points: the upper left corner (_h, _w), the upper right corner (_h, _w1), the lower left corner (_h1, _w), and the lower right corner (_h1, _w1). These four coordinate points together form the coordinates of four adjacent pixels surrounding the ideal offset position. Using these four adjacent pixel coordinates as the actual pixel position corresponding to the current sampling point i provides accurate physical pixel reference points for subsequent bilinear interpolation.

[0064] The method provided in this invention obtains the base coordinates by rounding down the floating-point offset, and accurately calculates the coordinates of four adjacent integer pixels surrounding the ideal offset position by combining the current spatial position and the relative coordinates of the sampling point. This maps the complex floating-point coordinates into a set of physically addressable pixels in a discrete image grid, providing accurate coordinate basis for the accurate sampling of non-integer coordinate positions and subsequent smooth interpolation in deformable convolution.

[0065] Based on any of the above embodiments, step S23 specifically includes: Step S231: Subtract the offset of the current sampling point from the base coordinate to obtain the decimal offset corresponding to the current sampling point; Step S232: Calculate the bilinear interpolation weights corresponding to the coordinates of the four adjacent pixels using the fractional offset. Step S233: Obtain four valid pixel values ​​from the input feature map based on the coordinates of the four adjacent pixels; Step S234: The four valid pixel values ​​are weighted and summed with their corresponding bilinear interpolation weights to obtain the interpolated pixel value of the current sampling point.

[0066] Specifically, in order to determine the contribution ratio of four adjacent pixels in the interpolation, the offset (off.x, off.y) of the current sampling point is first subtracted from the base coordinate (h_low, w_low) to extract the pure decimal part caused by deformation, and the decimal offset (lh, lw) corresponding to the current sampling point is obtained, that is, lh=off.x–h_low, lw=off.y–w_low.

[0067] In bilinear interpolation, the contribution (i.e., weight) of neighboring pixels to the final estimation result is determined by the spatial distance between the interpolation point and the physical pixel; the principle is that the closer the distance, the greater the weight. To accurately quantify this distance relationship, we can first calculate hh = 1.0f – lh and hw = 1.0f – lw. Here, hh and hw essentially represent complementary decimal offsets in the height and width directions. The constant 1.0f in the above calculation represents the normalized unit distance between adjacent physical pixels, and a single-precision floating-point format is used to avoid implicit type conversion, thus conforming to the single-precision operation characteristics of the underlying hardware computing unit. Because the coordinate spacing between two adjacent physical pixels is normalized to a unit of 1, when subtracting the decimal offsets (i.e., lh and lw) relative to the upper left corner base coordinates using 1, we are actually calculating the relative distance of the interpolation point from the lower and right boundaries.

[0068] Subsequently, using the fractional offset and its calculated complementary fractional offset, cross-multiplication is performed based on the area-weighted equivalence principle in two-dimensional space to calculate the bilinear interpolation weights corresponding to the coordinates of the four adjacent pixels. Specifically, when the interpolation point is very close to the top-left pixel, the values ​​of lh and lw are very small, and their complementary values ​​hh and hw will be close to 1. Therefore, the weight of the top-left pixel is obtained by multiplying the two complementary distances (i.e., top-left weight w1 = hh). hw), thus obtaining the maximum weight; similarly, by using the distance ratio combination in different directions, the upper right weight w2=hh is calculated sequentially. lw, lower left weight w3=lh hw and the lower right weight w4=lh lw. By introducing a complementary distance ratio and performing cross-multiplication, the distance attenuation quantization of the influence of neighboring pixels on the interpolation point in each direction is accurately realized.

[0069] Furthermore, before obtaining the pixel values, a boundary check needs to be performed on the calculated coordinates of the four adjacent pixels (e.g., determining whether they exceed the valid range of the feature map's height H and width W). For coordinates within the valid range, the execution entity reads the actual data of the corresponding channel from the input feature map based on the coordinates of the four adjacent pixels, thereby obtaining four valid pixel values ​​(i.e., v1, v2, v3, v4). If the coordinates are out of bounds, the corresponding pixel value is initialized to 0 by default.

[0070] Finally, the four valid pixel values ​​are weighted and summed with their corresponding bilinear interpolation weights, i.e., val = w1. v1+w2 v2+w3 v3+w4 v4. By integrating information from four neighboring points, the feature values ​​of non-integer coordinate positions are smoothly and continuously estimated, ultimately yielding the interpolated pixel value val for the current sampling point.

[0071] The method provided in this invention extracts fractional offsets to accurately calculate the bilinear interpolation weights of four adjacent pixels, and combines boundary checks to obtain effective neighboring pixel values ​​for weighted summation. This method can smoothly and accurately estimate the feature pixel values ​​at irregular coordinate positions caused by dynamic offsets, thereby ensuring the data continuity and accuracy of deformable convolution when extracting complex spatial geometric deformation features.

[0072] Based on any of the above embodiments, step S30 specifically includes: Step S31: Obtain the original shape information of the original weight tensor, wherein the original shape information includes the output channel dimension, the input channel dimension, and the original spatial dimension. Step S32: While keeping the memory arrangement order of the original weight tensor unchanged, the product of the input channel dimension and the original spatial dimension is used as the new input channel dimension of the target weight tensor, and the spatial dimension of the target weight tensor is set to one-to-one to obtain the target weight tensor after shape transformation.

[0073] Specifically, before performing the convolution operation, the execution entity first obtains the original shape information of the original weight tensor. This original shape can usually be represented as (K, C, R, S), where K is the output channel dimension (also called the number of output channels, i.e., the number of convolution kernels, each generating one output channel), C is the input channel dimension (also called the number of input channels), and R and S represent the number of sampling points of the convolution kernel in the height and width directions, constituting the original spatial dimension (i.e., R). S).

[0074] Because a regular convolution kernel of shape (K,C,R,S) and a kernel of shape (K,C) R The physical data arrangement of the convolution kernels (S, 1, 1) in the underlying memory is completely consistent, so the execution entity does not need to perform actual data copying or moving. The execution entity only needs to logically reshape the product of the input channel dimension and the original spatial dimension (i.e., C and R) while maintaining the original memory arrangement order of the weight tensors. Multiplying S, we get C. R S) is directly used as the new input channel dimension size of the target weight tensor.

[0075] At the same time, the executing entity sets the spatial dimension of the target weight tensor to one by one (i.e., 1). The process involves a 1-convolution, ultimately yielding the target weight tensor after shape transformation. This step essentially modifies the originally complex multi-point convolution kernel logically into an equivalent 1-convolution. 1. Convolution kernel.

[0076] The method provided in this invention, by logically rearranging the dimensions of the original weight tensor without changing the underlying physical data arrangement in memory, transforms the originally complex R... The convolutional kernel in the S-space sampling dimension is transformed into a single point after rearrangement in the channel dimension. A single convolutional kernel not only avoids the additional memory access overhead caused by data rearrangement, but also enables the transformed weight tensor to match the data format of the intermediate tensor, thus making deformable convolution compatible with existing technologies that support 1... 1. A high-performance operator library for continuous matrix multiplication.

[0077] Based on any of the above embodiments, in step S20, the step of concatenating and expanding the sampling results belonging to the same spatial location of the output feature map along the channel dimension to generate a continuously stored intermediate tensor includes: The product of the input channel dimension of the input feature map and the original spatial dimension of the original weight tensor is used as the channel dimension of the intermediate tensor. The spatial dimension of the output feature map is used as the spatial dimension of the intermediate tensor. The sampling results of each sampling point on each channel are written into consecutive memory addresses according to the shape information of the intermediate tensor.

[0078] Specifically, to transform discrete sampled data into continuous input data, the execution entity amplifies the channels of the extracted features when constructing the intermediate tensor. Specifically, this involves multiplying the input channel dimension (C) of the input feature map by the original spatial dimension (R) of the original weight tensor. Calculate S) and take the result (i.e., C) R S) is directly used as the channel dimension of the intermediate tensor. This means that for each feature point on the input tensor, the channel dimension originally in R... The results of sampling from S different spatial locations were flattened and incorporated into the channel dimension.

[0079] Besides channel dimension scaling, the intermediate tensor needs to be spatially aligned with the final output. Therefore, the spatial dimensions of the output feature map (height P and width Q) are directly used as the spatial dimensions of the intermediate tensor. Combined with the batch dimension N, the complete shape of this intermediate tensor (tmp tensor) is determined to be (N, C). R S, P, Q).

[0080] After determining the dimensional features of the intermediate tensor, the execution entity, after obtaining the interpolated pixel value for each non-integer coordinate, writes the sampling results of each sampling point on each channel into consecutive memory addresses according to the shape information of the intermediate tensor. Since the channel dimension has been amplified by R... S times, all discrete sampled data corresponding to the same output feature map spatial location (h,w) are compactly and sequentially laid out in this enlarged channel memory area, thereby transforming the spatial discrete sampling into a data block stored continuously in memory.

[0081] The method provided in this invention uses the product of the channel dimension of the input feature map and the spatial sampling dimension of the convolution kernel as the new channel dimension of the intermediate tensor, and splices the discrete interpolation sampling results of each spatial location according to the shape information and writes them sequentially to consecutive memory addresses. This achieves the reconstruction from discrete spatial sampling to continuous channel flattening at the data organization level, thereby eliminating the problem of scattered distribution of bilinear interpolation results in deformable convolution, and providing a data structure foundation for subsequent high-speed continuous reading by calling the underlying computing unit.

[0082] Based on any of the above embodiments, before step S40, the method further includes: The generated intermediate tensor is written to the on-chip tensor buffer; Accordingly, step S40 specifically includes: The intermediate tensor data is continuously read from the on-chip tensor buffer, and the read intermediate tensor data and the target weight tensor are used as continuous input data to be input into the hardware computing unit to perform a one-to-one convolution operation.

[0083] Specifically, because this invention introduces an additional intermediate tensor (tmp tensor) to store the bilinear interpolation results on top of the original deformable convolution scheme, this tmp tensor exhibits a write-then-read data flow property in the overall computation process. To avoid the severe memory access overhead caused by frequent access to slow global memory, the execution entity directly writes the generated intermediate tensor to the on-chip tensor buffer when generating the tmp tensor. For example, for GPGPU chips, the built-in tensor buffer feature can be fully utilized to temporarily store the channel rearrangement data obtained from interpolation sampling in the on-chip cache area for quick access by subsequent computing units.

[0084] Accordingly, upon entering the actual convolution operation stage, the execution entity no longer retrieves data from global memory, but instead directly reads the data of intermediate tensors continuously from the on-chip tensor buffer. Since the intermediate tensors in the preceding steps have been designed with a highly regular and continuous memory layout, the execution entity can use the read intermediate tensor data and the reshaped target weight tensor as continuous input data. Subsequently, the execution entity directly inputs the continuous data into the hardware computing unit to perform a one-to-one convolution operation. This hardware computing unit (e.g., tcore) is a dedicated acceleration module specifically designed for convolution and matrix multiplication.

[0085] The method provided in this invention utilizes the on-chip tensor buffer unique to artificial intelligence chips to manage intermediate tensors with write-before-read characteristics, and directly sends the input data after channel unrolling and continuous processing and the transformed target weights to the underlying hardware computing unit optimized for matrix operations. This not only reduces the global memory access latency during the intermediate variable access process, but also improves the overall execution performance of deformable convolution algorithms.

[0086] Based on any of the above embodiments Figure 4 This is the second schematic diagram of the deformable convolution acceleration method provided by the present invention, as shown below. Figure 4 As shown, the overall computational process of this deformable convolution acceleration method mainly includes parallel input data processing and weight parameter processing, hardware loading and computation, and result output. The specific process is as follows: After the process begins, the execution entity employs a two-branch parallel processing logic during the data preparation phase. In the feature data processing branch, the input and offset are read first. The input is the feature map to be processed, and the offset is an offset tensor indicating the dynamic offset position of each sampling point. Subsequently, the execution entity performs bilinear interpolation based on the read offset to obtain the tmp tensor. During this process, the pixel values ​​at non-integer coordinate positions are calculated through bilinear interpolation, and multiple discrete sampling results corresponding to the same output spatial position are concatenated and expanded along the channel dimension, thus transforming the scattered data into a continuously stored intermediate tensor (tmp tensor). Next, tcore loads the input data from the tmp tensor; that is, the underlying high-performance core computing unit (tcore) leverages its strength in processing continuous data, efficiently and continuously reading the data of the tmp tensor from the tensor buffer into the register.

[0087] In the kernel parameter processing branch, the execution body first reads the shape (K, C, R, S) of the original weight tensor. This shape represents the sampling point size of the original convolution kernel in the output channel, input channel, and height and width directions. To align with the channel dimension being enlarged by R in the left branch... Align the tmp tensor by S times, and perform a shape transformation operation on the weight, changing its shape from (K,C,R,S) to (K,C). R S,1,1). This step, without changing the underlying physical memory data layout, logically rearranges the originally complex R... The S convolutional kernel is equivalent to the channel rearranged 1. 1. Convolution kernel. Subsequently, tcore loads the convolution kernel, that is, it synchronously loads the transformed target weight tensor into the hardware computing unit.

[0088] After both data streams are prepared and loaded, tcore is triggered to execute step 1. 1. Convolution operation. At this point, tcore, optimized for continuous matrix multiplication, convolves the continuously input feature data with 1... A 1-bit convolutional kernel performs high-speed multiply-accumulate operations, thus transforming deformable convolution, which was originally computationally inefficient due to discrete data, into a standard 1-bit process that hardware excels at. 1. Convolution operation.

[0089] After the computation is complete, the executing entity writes the result to the output tensor. That is, the above 1... The data obtained from the convolution operation are written sequentially into the specified global memory according to the standard shape of the output feature map (usually (N,K,P,Q)), thus completing the feature extraction task of the current layer and ending the process.

[0090] The method provided in this invention separates the coupled interpolation sampling and convolution operation in traditional deformable convolution into two independent steps, and introduces an intermediate tensor to concatenate and expand the discrete sampling results at the same spatial location along the channel dimension. This transforms the originally scattered memory access into a continuous data storage format, while simultaneously supporting the original R... The logical equivalent transformation of the S-convolution kernel is 1. The single convolution kernel not only retains the ability of deformable convolution to model complex deformations, but also solves the bottleneck of high-performance hardware units being unable to be utilized due to the discreteness of sampled data. It realizes the regularization and continuous input of computational data, thereby fully exploring and utilizing the powerful hardware computing power of specific hardware computing units (such as tcore), and improving the overall memory access performance and computational efficiency of deformable convolution algorithms.

[0091] The deformable convolution acceleration device provided by the present invention is described below. The deformable convolution acceleration device described below can be referred to in correspondence with the deformable convolution acceleration method described above.

[0092] Based on any of the above embodiments Figure 5 This is a schematic diagram of the deformable convolution acceleration device provided by the present invention, as shown below. Figure 5 As shown, the device includes: The parameter acquisition unit 510 is used to acquire the input feature map, the offset tensor, and the original weight tensor. The interpolation expansion unit 520 is used to perform interpolation sampling operation on the input feature map according to the offset tensor, and to splice and expand the sampling results belonging to the same spatial position of the output feature map in the channel dimension to generate a continuously stored intermediate tensor. The weight rearrangement unit 530 is used to perform a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one, wherein the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor. The convolution operation unit 540 is used to perform a one-to-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of deformable convolution.

[0093] The apparatus provided in this invention, after acquiring relevant parameters, addresses the problem of discrete sampling data caused by dynamic offsets by splicing and unfolding discrete interpolation sampling results belonging to the same spatial location in the output feature map along the channel dimension, thereby generating a continuously stored intermediate tensor. This step transforms the originally fragmented discrete memory access into continuous regular memory access, thus eliminating the performance bottleneck caused by data discontinuity during underlying computation. Simultaneously, a shape transformation operation is performed on the original weight tensor to obtain a target weight tensor with a spatial dimension of one-to-one, ensuring that the channel dimension of its rearranged form matches that of the intermediate tensor. This transforms the originally complex spatial convolution operation into an equivalent one-to-one convolution operation. Through this collaborative design of continuous data arrangement and weight shape rearrangement, this invention enables subsequent one-to-one convolution operations based on the intermediate and target weight tensors to directly call the high-performance computing units optimized for continuous data in the underlying AI chip, thereby improving the utilization of hardware computing resources and the execution efficiency of the entire deformable convolution operator.

[0094] Based on any of the above embodiments, the interpolation expansion unit includes: The offset acquisition subunit is used to traverse each spatial position of the output feature map and obtain the offset of the sampling point corresponding to each spatial position according to the offset tensor. The actual pixel point determination subunit is used to determine the position of the actual pixel point corresponding to each sampling point based on the offset of each sampling point and the coordinates of the input feature map; The interpolation pixel value calculation subunit is used to perform interpolation calculation on the input feature map based on the actual pixel position corresponding to each sampling point to obtain the interpolated pixel value of each sampling point; The pixel value continuous arrangement sub-unit arranges the interpolated pixel values ​​of each sampling point corresponding to the same spatial position of the output feature map into the channel dimension corresponding to the intermediate tensor in a preset order to obtain the intermediate tensor.

[0095] Based on any of the above embodiments, the actual pixel point determination subunit is specifically used for: The offset of the current sampling point is rounded down to obtain the integer base coordinates; Based on the coordinates of the spatial location currently traversed, the relative coordinates of the current sampling point in the convolution kernel, and the base coordinates, the coordinates of four adjacent pixels are calculated, and the coordinates of the four adjacent pixels are used as the actual pixel position corresponding to the current sampling point.

[0096] Based on any of the above embodiments, the interpolation pixel value calculation subunit is specifically used for: Subtract the offset of the current sampling point from the base coordinate to obtain the decimal offset corresponding to the current sampling point; The bilinear interpolation weights corresponding to the coordinates of the four adjacent pixels are calculated using the fractional offset. Four valid pixel values ​​are obtained from the input feature map based on the coordinates of the four adjacent pixels. The interpolated pixel value of the current sampling point is obtained by weighting and summing the four valid pixel values ​​with their corresponding bilinear interpolation weights.

[0097] Based on any of the above embodiments, the weight rearrangement unit is specifically used for: Obtain the original shape information of the original weight tensor, wherein the original shape information includes the output channel dimension, the input channel dimension, and the original spatial dimension. While keeping the memory arrangement order of the original weight tensor unchanged, the product of the input channel dimension and the original spatial dimension is used as the new input channel dimension of the target weight tensor, and the spatial dimension of the target weight tensor is set to one-to-one to obtain the target weight tensor after shape transformation.

[0098] Based on any of the above embodiments, the interpolation expansion unit is specifically used for: The product of the input channel dimension of the input feature map and the original spatial dimension of the original weight tensor is used as the channel dimension of the intermediate tensor. The spatial dimension of the output feature map is used as the spatial dimension of the intermediate tensor. The sampling results of each sampling point on each channel are written into consecutive memory addresses according to the shape information of the intermediate tensor.

[0099] Based on any of the above embodiments, the device further includes a tensor writing unit, the tensor writing unit being used for: The generated intermediate tensor is written to the on-chip tensor buffer; Accordingly, the convolution operation unit is specifically used for: The intermediate tensor data is continuously read from the on-chip tensor buffer, and the read intermediate tensor data and the target weight tensor are used as continuous input data to be input into the hardware computing unit to perform a one-to-one convolution operation.

[0100] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a deformable convolution acceleration method. This method includes: acquiring an input feature map, an offset tensor, and an original weight tensor; performing interpolation sampling on the input feature map based on the offset tensor, and concatenating and unfolding the sampling results belonging to the same spatial position of the output feature map along the channel dimension to generate a continuously stored intermediate tensor; performing a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one, wherein the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor; and performing a one-by-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of the deformable convolution.

[0101] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deformable convolution acceleration method provided by the above methods. The method includes: acquiring an input feature map, an offset tensor, and an original weight tensor; performing an interpolation sampling operation on the input feature map according to the offset tensor, and splicing and unfolding the sampling results belonging to the same spatial position of the output feature map in the channel dimension to generate a continuously stored intermediate tensor; performing a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one, wherein the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor; and performing a one-by-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of the deformable convolution.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the deformable convolution acceleration method provided by the above methods. The method includes: acquiring an input feature map, an offset tensor, and an original weight tensor; performing an interpolation sampling operation on the input feature map according to the offset tensor, and concatenating and unfolding the sampling results belonging to the same spatial position of the output feature map in the channel dimension to generate a continuously stored intermediate tensor; performing a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one, wherein the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor; and performing a one-by-one convolution operation based on the intermediate tensor and the target weight tensor to obtain the output feature map of the deformable convolution.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deformable convolution acceleration method, characterized in that, include: Obtain the input feature map, offset tensor, and original weight tensor; The input feature map is interpolated and sampled according to the offset tensor, and the sampling results belonging to the same spatial position of the output feature map are spliced ​​and expanded in the channel dimension to generate a continuously stored intermediate tensor. The intermediate tensor is then written into the on-chip tensor buffer. The original weight tensor is subjected to a shape transformation operation to obtain a target weight tensor with a spatial dimension of one by one, and the channel dimension of the target weight tensor matches the channel dimension of the intermediate tensor. The intermediate tensor data is continuously read from the on-chip tensor buffer, and the read intermediate tensor data and the target weight tensor are used as continuous input data to be input into the hardware computing unit to perform a one-to-one convolution operation to obtain the output feature map of deformable convolution.

2. The deformable convolution acceleration method according to claim 1, characterized in that, The step of performing interpolation sampling on the input feature map based on the offset tensor, and concatenating and expanding the sampling results belonging to the same spatial position of the output feature map along the channel dimension to generate a continuously stored intermediate tensor includes: Traverse each spatial position of the output feature map and obtain the offset of the sampling point corresponding to each spatial position according to the offset tensor; Based on the offset of each sampling point and the coordinates of the input feature map, the actual pixel position corresponding to each sampling point is determined; Based on the actual pixel position corresponding to each sampling point, interpolation calculation is performed on the input feature map to obtain the interpolated pixel value of each sampling point. The interpolated pixel values ​​of each sampling point corresponding to the same spatial position of the output feature map are continuously arranged into the channel dimension corresponding to the intermediate tensor according to a preset order to obtain the intermediate tensor.

3. The deformable convolution acceleration method according to claim 2, characterized in that, Determining the actual pixel position corresponding to each sampling point based on the offset of each sampling point and the coordinates of the input feature map includes: The offset of the current sampling point is rounded down to obtain the integer base coordinates; Based on the coordinates of the spatial location currently traversed, the relative coordinates of the current sampling point in the convolution kernel, and the base coordinates, the coordinates of four adjacent pixels are calculated, and the coordinates of the four adjacent pixels are used as the actual pixel position corresponding to the current sampling point.

4. The deformable convolution acceleration method according to claim 3, characterized in that, The step of performing interpolation calculations on the input feature map based on the actual pixel positions corresponding to each sampling point to obtain the interpolated pixel values ​​for each sampling point includes: Subtract the offset of the current sampling point from the base coordinate to obtain the decimal offset corresponding to the current sampling point; The bilinear interpolation weights corresponding to the coordinates of the four adjacent pixels are calculated using the fractional offset. Four valid pixel values ​​are obtained from the input feature map based on the coordinates of the four adjacent pixels. The interpolated pixel value of the current sampling point is obtained by weighting and summing the four valid pixel values ​​with their corresponding bilinear interpolation weights.

5. The deformable convolution acceleration method according to claim 1, characterized in that, The step of performing a shape transformation operation on the original weight tensor to obtain a target weight tensor with a spatial dimension of one by one includes: Obtain the original shape information of the original weight tensor, wherein the original shape information includes the output channel dimension, the input channel dimension, and the original spatial dimension. While keeping the memory arrangement order of the original weight tensor unchanged, the product of the input channel dimension and the original spatial dimension is used as the new input channel dimension of the target weight tensor, and the spatial dimension of the target weight tensor is set to one-to-one to obtain the target weight tensor after shape transformation.

6. The deformable convolution acceleration method according to claim 5, characterized in that, The step of concatenating and expanding the sampling results belonging to the same spatial location in the output feature map along the channel dimension to generate a continuously stored intermediate tensor includes: The product of the input channel dimension of the input feature map and the original spatial dimension of the original weight tensor is used as the channel dimension of the intermediate tensor. The spatial dimension of the output feature map is used as the spatial dimension of the intermediate tensor. The sampling results of each sampling point on each channel are written into consecutive memory addresses according to the shape information of the intermediate tensor.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the deformable convolution acceleration method as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the deformable convolution acceleration method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the deformable convolution acceleration method as described in any one of claims 1 to 6.

Images