Image processing methods, edge chips and storage media

Abstract

This application discloses an image processing method, an edge chip, and a storage medium, belonging to the field of image technology. The method, applied to an edge chip, includes: acquiring the input feature map of the image to be processed, the relative position offset of the deformable convolution kernel, and the position modulation amount of the deformable convolution; dividing these three into P groups according to the number of deformable channel groups, obtaining P group sub-input feature maps, P group sub-convolution kernel relative position offsets, and P group sub-position modulation amounts. Based on the P group sub-convolution kernel relative position offsets and P group sub-position modulation amounts, the P group sub-input feature maps are respectively subjected to grid sampling and then concatenated to obtain a modulated complete sampled feature map; convolution is performed on the modulated complete sampled feature map using the convolution kernel to obtain the deformable convolution result of the image to be processed. By performing an equivalent transformation on the deformable convolution operation, the single operator is decomposed into a series of computational flows convenient for edge chips, rationally utilizing computational resources and improving the efficiency of convolution operations.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to an image processing method, an edge chip, and a storage medium. Background Technology

[0002] In image processing, traditional convolutional neural networks (CNNs) are commonly used for visual tasks such as object recognition, target detection, and semantic segmentation. However, traditional convolutional layers in CNNs sample the input feature map using a fixed, regular grid. This fixed structure is unsuitable for scenarios involving geometric changes in objects within the image, such as scale variations, rotations, and deformations. First-generation deformable convolutional networks (DCNs) build upon CNNs by adding additional learnable spatial offsets to the sampling positions of the convolutional kernels. This allows the sampling process to be dynamically adjusted based on the specific characteristics of the image content. This enhances the ability of CNNs to handle geometric deformations in object recognition, target detection, and semantic segmentation.

[0003] The second-generation deformable convolutional network (DCNv2) incorporates a modulation mechanism compared to DCN. This mechanism not only performs the learned spatial location shift on each sampling point but also modulates the feature value of the pixel (i.e., increases the weight of the feature value corresponding to the spatial location shift), thereby selectively enhancing or weakening the contribution of features. This enhancement allows DCNv2 to control the sampling mode over a wider range of feature levels and in more complex environments, thus providing more accurate feature representations for various visual tasks and significantly improving image processing performance.

[0004] However, the deformable convolution operator in DCNv2 is computationally more complex than the traditional convolution operator in CNNs. In addition to all the operations of traditional convolution, it includes extra steps to calculate the feature value at the new sampling location based on the spatial offset and offset weight on the input feature map, resulting in a longer computation time. Summary of the Invention

[0005] This application provides an image processing method, an edge chip, and a storage medium, which can improve the efficiency of convolution operations. The technical solution is as follows:

[0006] In a first aspect, an image processing method is provided, the method comprising: acquiring an input feature map of an image to be processed, a relative position offset of a deformable convolution kernel, and a position modulation amount of the deformable convolution; the position modulation amount indicating the feature value weight corresponding to the position offset of the convolution kernel; grouping the input feature map, the relative position offset of the convolution kernel, and the position modulation amount into P groups of sub-input feature maps, P groups of sub-convolution kernel relative position offsets, and P groups of sub-position modulation amounts; wherein P represents the number of deformable channel groups, P is a factor of the number of input channels in the input feature map, and P is a positive integer; performing grid sampling on the P groups of sub-input feature maps according to the relative position offset of the P groups of sub-convolution kernels and the P groups of sub-position modulation amounts to obtain P groups of modulated grouped sampled feature maps; concatenating the P groups of modulated grouped sampled feature maps to obtain a modulated complete sampled feature map; and convolving the modulated complete sampled feature map with a convolution kernel to obtain the deformable convolution result of the image to be processed.

[0007] Secondly, an edge chip is provided, comprising: a neural network computing unit, a vector computing unit, a data storage unit, and a data access unit. The data storage unit is an internal storage unit of the edge chip, used to store input feature maps, convolutional kernel relative position offsets, and position modulation amounts. The data access unit is used to group the input feature maps, convolutional kernel relative position offsets, and position modulation amounts into P groups of sub-input feature maps, P groups of sub-convolutional kernel relative position offsets, and P groups of sub-position modulation amounts. P represents the number of deformable channel groups, P is a factor of the number of input channels in the input feature map, and P is a positive integer. The vector computing unit is used to perform grid sampling on the P groups of sub-input feature maps according to the P groups of sub-convolutional kernel relative position offsets and the P groups of sub-position modulation amounts, respectively, to obtain P groups of modulated grouped sampled feature maps. The data access unit is also used to concatenate the P groups of modulated grouped sampled feature maps to obtain a modulated complete sampled feature map. The neural network computing unit is used to convolve the modulated complete sampled feature map according to the convolutional kernel to obtain a deformable convolution result of the image to be processed.

[0008] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0009] Fourthly, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0010] This application provides an image processing method, an edge chip, and a storage medium. According to the scheme provided, the following are obtained: an input feature map of the image to be processed, the relative position offset of the convolution kernel of a deformable convolution, and the position modulation amount of the deformable convolution. The position modulation amount indicates the feature value weight corresponding to the position offset of the convolution kernel. The input feature map, the relative position offset of the convolution kernel, and the position modulation amount are grouped separately, and each of these three is divided into P groups according to the number of deformable channel groups, resulting in P group sub-input feature maps, P group sub-convolution kernel relative position offsets, and P group sub-position modulation amounts. Here, P represents the number of deformable channel groups, that is, how many groups the input channels can be divided into. P is a factor of the number of input channels in the input feature map, and P is a positive integer divisible by the number of input channels. Thus, the dimension of the input feature map, the relative position offset of the convolution kernel, and the position modulation amount related to the input channels is divided into P groups, each group including at least one input channel, and the number of input channels included in each group is the same. Based on the relative position offset of the P-group sub-convolution kernels and the modulation amount of the P-group sub-positions, grid sampling is performed on the P-group sub-input feature maps to obtain P-group modulated grouped sampled feature maps. These P-group modulated grouped sampled feature maps are then concatenated to obtain the modulated complete sampled feature map. Using the relative position offset of the grouped convolution kernels and the modulation amount of the grouped positions, grid sampling is performed on the grouped input feature maps, followed by concatenation. Then, convolution is performed on the modulated complete sampled feature map using the convolution kernels to obtain the deformable convolution result of the image to be processed. This image processing method is applied to an edge chip, which includes vector computing units adept at vector calculations and neural network units adept at matrix multiplication operations in convolution. By performing an equivalent transformation on the deformable convolution operation, the single operator is decomposed into a series of computational flows convenient for the edge chip, which can rationally utilize the edge chip's computational resources and improve the efficiency of convolution operations. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the structure of a deformable convolutional network provided in an embodiment of this application;

[0013] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0014] Figure 3 This is a flowchart of another image processing method provided in the embodiments of this application;

[0015] Figure 4 This is a flowchart of another image processing method provided in the embodiments of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an edge chip provided in an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of an edge device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0020] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, images, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0021] Before providing a detailed explanation of the embodiments of this application, the application scenarios and related technologies of the embodiments of this application will be described first.

[0022] The image processing method provided in this application can be applied to visual tasks such as object detection, object recognition, and semantic segmentation. With technological advancements, the primary carrier of visual tasks is gradually shifting from the cloud to large-scale edge models. These edge models can be deployed in edge chips of edge devices, where the computing power and memory fall between the cloud and the terminal device. Edge chips can be edge artificial intelligence (AI) chips. Compared to traditional central processing units (CPUs) or graphics processing units (GPUs), edge AI chips typically have lower power consumption, smaller size, and higher energy efficiency, reducing bandwidth pressure and better adapting to large-scale edge models on edge devices.

[0023] Deformable convolution operators are typically deployed on GPU platforms, where performance is optimized through well-designed thread organization and memory access patterns. However, edge AI chips are limited in terms of computing power and memory size compared to GPU platforms. Therefore, how to efficiently deploy deformable convolutions on edge chips has become an urgent problem to be solved.

[0024] In related technologies, deformable convolution transforms the fixed-shape convolution process into a variable convolution process that can adapt to the shape of the object. DCN adds a positional offset to each point in the receptive field; the magnitude of this offset is learned. After the offset, the receptive field is no longer a square but matches the actual shape of the object. This ensures that no matter how the object deforms, the convolution region always covers the area surrounding the object's shape.

[0025] For example, such as Figure 1 As shown, Figure 1 This is a schematic diagram of a deformable convolutional network provided in an embodiment of this application. The original receptive field consists of nine connected small squares. After adding an offset to the receptive field, the new receptive field consists of nine dispersed small squares. It can be seen that the process of stacking offsets can simulate various deformations such as target movement, size scaling, and rotation. Deformable convolution in DCN optimizes the sampling position (i.e., adding a learnable spatial position offset to the original fixed position), which makes the convolution operation closer to the actual geometry of the object. This improvement enables DCN to capture more contextual information and adapt to the local shapes of different objects. Due to the powerful modeling capabilities of DCN, deformable convolution has been widely used to improve the performance of complex vision tasks such as object detection, semantic segmentation, and pose estimation.

[0026] The above Figure 1The diagram shows the structure of DCNv2. The nine small squares scattered across the input feature map represent sampling points for deformable convolution. The input feature map with N channels learns the positional offset of each sampling point through a convolutional layer (conv), resulting in a feature map with 2N channels. 2N represents the positional offset (offset) in the x and y directions of each channel, i.e., the relative position offset of the convolutional kernel (2N offset). Since a modulation mechanism is added to the deformable convolution, a feature map with 3N channels can be obtained. Each sampling of the feature map not only involves offsetting but also adjustment using learned weights, enabling more complex geometric transformation learning to complete the deformable convolution and obtain the output feature map.

[0027] While deformable convolution significantly improves the performance of convolution, it includes additional steps beyond all the operations of traditional convolution. These steps involve calculating feature values ​​at newly sampled locations based on the spatial offsets and offset weights on the input feature map. These additional steps mean that, for the same input and output size, deformable convolution requires more computation time than traditional convolution, resulting in lower computational efficiency. Furthermore, practical deployments require more computing resources, typically necessitating deployment on high-performance cloud servers rather than on lower-performance edge chips, thus limiting its application scenarios.

[0028] To address the aforementioned technical problem of low efficiency in deformable convolution operations, this application provides an image processing method, such as... Figure 2 As shown, Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application. The image processing method includes:

[0029] S101. Obtain the input feature map of the image to be processed, the relative position offset of the deformable convolution kernel, and the position modulation amount of the deformable convolution; the position modulation amount indicates the feature value weight corresponding to the position offset of the convolution kernel.

[0030] In this embodiment, the input feature map, the relative position offset of the deformable convolution kernel, and the position modulation amount of the deformable convolution all include four dimensions, which can be represented by axes 0, 1, 2, and 3.

[0031] The structure or shape of the input feature map (IF) can be represented as (N, C) i H i W i), where i represents the input, N represents the number of images in the batch, and C i H represents the number of input channels in the input feature map (also known as the number of input channels). i W represents the height of the input feature map. i This indicates the width of the input feature map.

[0032] The shape of a convolution kernel (CK) can be represented as (C o C i K h K w ), where o represents output, C o K represents the number of output channels of the convolution kernel (also known as the number of output channels). h K represents the height of the convolution kernel. w This indicates the width of the convolution kernel.

[0033] During deformable convolution operations, not only is the offset of the sampling position (i.e., the relative position offset of the convolution kernel) learned, but also the modulation scalar (MS) is learned. MS is used to adjust the degree of influence of the sampling position.

[0034] The shape of the convolution kernel's relative position offset (PO) can be represented as (N, 2D). g K h K w H o W o ), 2 represents the horizontal and vertical coordinate offsets, D g D represents the number of deformable channel groups. g Can be C i Divisible by integers, for example, C i It is 64, D g It's 4, with 16 input channels grouped together. H o W represents the height of the output feature map. o This represents the width of the output feature map. Where 2D... g K h K w This means 2 multiplied by D g Multiply by K h Multiply by K w The following indicates that if there is no sign between two numbers in the shape parameter, it also indicates multiplication, which will not be elaborated further.

[0035] The position modulation amount can also be called the modulation factor, and the shape of the MS is (N, D). g Kh K w H o W o The position modulation amount is a numerical value. A sampling position (including the horizontal and vertical coordinates) corresponds to a numerical value, which represents the feature weight or influence degree at that sampling position. Therefore, compared with the relative position offset of the convolution kernel, there is no 2 in the dimension with axis 1.

[0036] In some embodiments, after acquiring the image to be processed, the image to be processed is preprocessed to obtain an input feature map; the preprocessing is convolution processing or pooling processing.

[0037] In this embodiment, the input feature map can be the original image (i.e., the image to be processed), or it can be obtained by performing convolution or pooling on the original image. When the input feature map is the original image, the above-mentioned H... i W represents the height of the original image. i The input feature map represents the width of the original image, with no restrictions on its height and width, thus expanding the application scenarios for image processing. The input feature map can be a preprocessed feature map of the original image, conforming to the input format of deformable convolutional networks, facilitating subsequent grouping, grid sampling, and other operations, thereby improving the efficiency of convolutional operations.

[0038] S102. The input feature map, the relative position offset of the convolution kernel, and the position modulation amount are grouped to obtain P groups of sub-input feature maps, P groups of sub-convolution kernel relative position offsets, and P groups of sub-position modulation amounts.

[0039] Where P represents the number of deformable channels in the deformable convolution, which is D g P is a factor representing the number of input channels in the input feature map, and P is a positive integer. The number of deformable channel groups refers to the number of groups into which the input channels can be divided, which is learned during the training of the deformable convolutional model. For example, with 32 input channels and 4 deformable channel groups, the input channels can be divided into 4 groups, each containing 8 input channels. The relative position offset and position modulation of the convolutional kernels at the same position in the 8 input channels of the same group are identical. That is, the 8 input channels in the same group share the same set of relative position offsets and position modulations of the convolutional kernels. For example, input channel 1 and input channel 2 belong to the same group. Pixel A in input channel 1 and the pixel at the same position as pixel A in input channel 2 have the same relative position offset and position modulation of the convolutional kernels. Of course, the number of deformable channel groups can also be 1, which also means that the relative position offset and position modulation of the convolutional kernels at the same position in all input channels are identical.

[0040] In this embodiment, IF, PO, and MS are grouped according to the number of deformable channel groups, and the tensors of IF, PO, and MS are divided into D along axis 1. g The k-th group is processed to obtain P-group sub-input feature maps, P-group sub-convolutional kernel relative position offsets, and P-group sub-position modulation amounts. The k-th group is then processed using IF... k PO k MS k PO k The shape is (N, 2K) h K w H o W o MS k The shape is (N, K) h K w H o W o ), IF k The shape is (N, C) i / / D g H i W i ), where k = 0, 1, ..., D g -1, the symbol / / indicates integer division.

[0041] S103. Based on the relative position offset of the P group sub-convolution kernels and the P group sub-position modulation amount, perform grid sampling on the P group sub-input feature maps respectively to obtain the P group modulated group sampled feature maps.

[0042] For the grouped input feature maps, the relative position offset of the grouped convolutional kernels, and the position modulation amount after grouping, grid sampling is performed for each group to obtain the grouped modulated sampled feature maps (i.e., the grouped sampled feature maps after P groups of modulation).

[0043] Since the IF, PO, and MS groups are formed by grouping at least one input channel, there is a correspondence between the groups of IF, PO, and MS. For example, the relative position offset of the sub-convolutional kernel of the k-th group and the modulation amount of the sub-position of the k-th group correspond to the sub-input feature map of the k-th group, where k is a natural number less than P, and k takes values ​​from 0 to P-1. Based on this, when performing network sampling on the input feature map group by group, the sub-input feature map of the k-th group is subjected to grid sampling according to the relative position offset of the sub-convolutional kernel of the k-th group and the modulation amount of the sub-position of the k-th group, resulting in the modulated grouped sampled feature map of the k-th group. Similarly, for other groups, the same grid sampling method as for the k-th group is used to obtain the modulated grouped sampled feature maps of P groups.

[0044] S104. The modulated grouped sampling feature maps of group P are spliced ​​together to obtain the complete modulated sampling feature map.

[0045] The sampled feature maps after group modulation (i.e., the grouped sampled feature maps after P-group modulation) are spliced ​​together to obtain the complete modulated sampled feature map.

[0046] S105. Convolve the modulated complete sampled feature map according to the convolution kernel to obtain the deformable convolution result of the image to be processed.

[0047] The final deformable convolution result is calculated from the convolution kernel and the modulated complete sampled feature map described above. The deformable convolution result can be used in subsequent visual tasks such as object detection (e.g., detecting what objects are), object recognition (e.g., identifying what the object is), and semantic segmentation (e.g., outputting pixel-level label maps), and this embodiment of the application does not limit this application.

[0048] This application relates to the field of machine learning technology and provides an efficient processing method for deformable convolution. According to the scheme provided in this application, the input feature map of the image to be processed, the relative position offset of the convolution kernel of the deformable convolution, and the position modulation amount of the deformable convolution are obtained; the position modulation amount indicates the feature value weight corresponding to the position offset of the convolution kernel; the input feature map, the relative position offset of the convolution kernel, and the position modulation amount are grouped separately, and each of these three is divided into P groups according to the number of deformable channel groups, resulting in P group sub-input feature maps, P group sub-convolution kernel relative position offsets, and P group sub-position modulation amounts. Here, P represents the number of deformable channel groups, that is, how many groups the input channels can be divided into, P is a factor of the number of input channels in the input feature map, P is a positive integer, and P is divisible by the number of input channels. Thus, the dimension of the input feature map, the relative position offset of the convolution kernel, and the position modulation amount related to the input channels is divided into P groups, each group including at least one input channel, and the number of input channels included in each group is the same. Based on the relative position offset of the P-group sub-convolution kernels and the modulation amount of the P-group sub-positions, grid sampling is performed on the P-group sub-input feature maps to obtain P-group modulated grouped sampled feature maps. These P-group modulated grouped sampled feature maps are then concatenated to obtain the modulated complete sampled feature map. Using the relative position offset of the grouped convolution kernels and the modulation amount of the grouped positions, grid sampling is performed on the grouped input feature maps, followed by concatenation. Then, convolution is performed on the modulated complete sampled feature map using the convolution kernels to obtain the deformable convolution result of the image to be processed. This image processing method is applied to an edge chip, which includes vector computing units adept at vector calculations and neural network units adept at matrix multiplication operations in convolution. By performing an equivalent transformation on the deformable convolution operation, the single operator is decomposed into a series of computational flows convenient for the edge chip, which can rationally utilize the edge chip's computational resources and improve the efficiency of convolution operations.

[0049] In some embodiments, the above Figure 2 S103 may also include S1031 and S1032. For example... Figure 3 As shown, Figure 3 This is a flowchart of another image processing method provided in the embodiments of this application.

[0050] S1031. Based on the relative position offset of the kth sub-convolution kernel in the relative position offset of the P group sub-convolution kernels, perform grid sampling on the kth sub-input feature map of the P group sub-input feature map to obtain the kth sampling result; k is a natural number less than P.

[0051] In this embodiment, the relative position offset of the P-group sub-convolutional kernels, the modulation amount of the P-group sub-positions, and the P-group sub-input feature maps have a corresponding relationship. During network sampling, the P-group sub-input feature maps are processed based on the relative position offset of the P-group sub-convolutional kernels. Figure 1 One-to-one sampling is performed to obtain the sampling results for each group.

[0052] For example, based on the preset reference position and the relative position offset of each group of sub-convolutional kernels in the P-group sub-convolutional kernel relative position offset, the absolute position offset unit of each group of convolutional kernels is determined; the preset reference position is related to the sliding rule of the convolutional kernels. Using the absolute position offset unit of the k-th group of convolutional kernels, grid sampling is performed on the k-th group of sub-input feature maps in the P-group sub-input feature maps to obtain the k-th sampling result.

[0053] S1032. Based on the sampling results of the kth group and the sub-position modulation amount of the kth group in the P group, determine the group sampling feature map after modulation of the kth group, thereby obtaining the group sampling feature map after modulation of the P group.

[0054] In this embodiment, the relative position offset of the P-group sub-convolutional kernels, the P-group sub-position modulation amount, and the P-group sub-input feature map have a corresponding relationship. During position modulation, the P-group sub-position modulation amount modulates the P-group sampling results one-to-one, obtaining the modulated grouped sampling feature map for each group. For example, the k-th sub-position modulation amount in the P-group sub-position modulation amount is used to modulate the k-th group sampling result to obtain the modulated grouped sampling feature map for the k-th group. Similarly, for other groups, the same modulation method as the k-th group is used to obtain the modulated grouped sampling feature map for the P-group.

[0055] In this embodiment, sampling results for each group are obtained through one-to-one sampling, and then modulated one-to-one for each group to obtain modulated grouped sampling feature maps. By decomposing the single operator into vector calculations that edge chips excel at, the computational resources of the edge chips are rationally utilized, thus improving computational efficiency.

[0056] In some embodiments, when performing network sampling in S1031 above, the relative position offset of each group of sub-convolutional kernels in the P-group sub-convolutional kernel relative position offset can be divided first, and then the absolute position offset unit of the convolutional kernel can be obtained before performing network sampling.

[0057] For example, the relative position offset of each group of sub-convolutional kernels in the P groups of sub-convolutional kernel relative position offsets is divided to obtain P×M convolutional kernel relative position offset units; M represents the size of the convolutional kernel, and M is a positive integer. Based on the divided convolutional kernel relative position offset units, the above S1031 can also be implemented in the following way. According to the preset reference reference position and the k-th and j-th convolutional kernel relative position offset units in the P×M convolutional kernel relative position offset units, the k-th and j-th convolutional kernel absolute position offset units are determined; the preset reference reference position is related to the sliding rule of the convolutional kernel, k is a natural number less than P, and j is a natural number less than M; using the k-th and j-th convolutional kernel absolute position offset units, the k-th sub-input feature map in the P groups of sub-input feature maps is subjected to grid sampling to obtain the k-th and j-th sampling result units, thereby obtaining the k-th sampling result; the k-th sampling result includes M sampling result units.

[0058] In this embodiment, M represents the size of the convolution kernel, and K... h K represents the height of the convolution kernel. w M represents the width of the convolution kernel, where M equals K. h K w The relative position offsets of the grouped convolutional kernels (i.e., the relative position offsets of the sub-convolutional kernels) are divided according to the kernel size to obtain the convolutional kernel relative position offset cells required for grid cell computation. For example, for the k-th group of PO... k The tensor is partitioned along axis 1, resulting in M ​​groups. The relative position offset of the k-th group and the j-th product kernel is represented by PO. k,j This means that j = 0, 1, ..., K h K w -1, PO k,j The shape is (N, 2, H) o W o ).

[0059] Since the relative position offset unit of the convolution kernel represents the relative position of the pixel, it needs to be converted into the absolute position of the pixel. That is, each relative position offset unit of the convolution kernel needs to be converted into an absolute position offset unit of the convolution kernel. Let's take the relative position offset unit PO of the j-th convolution kernel in the k-th group as an example. k,j For example, the absolute position offset unit AP of the j-th convolutional kernel in the k-th group. k,j =BP k,j +PO k,j BP k,j This represents the j-th reference position in the k-th group. Each reference position in each group is pre-set and related to the sliding rule of the convolution kernel.

[0060] Using the absolute position offset unit of each convolutional kernel in each group, grid sampling is performed on the sub-input feature map of the corresponding group to obtain the C of the sub-input feature map of the corresponding group. i / / D g Each sampling result unit. Taking the k-th group as an example, the absolute position offset unit of the j-th convolutional kernel in the k-th group is used to perform grid sampling on the feature map of the k-th sub-input, resulting in the sampling result unit GS of the k-th group and the j-th sampled result unit. k,j =grid_sample(IF k AP k,j (mode = bilinear).

[0061] Here, `grid_sample` is a commonly used grid sampling operator, `mode` represents the interpolation method used to implement grid sampling, and `bilinear` represents bilinear interpolation. The shape of the absolute position offset unit for each convolutional kernel in each group is (N, 2, H). o W o The shape of each sub-input feature map is (N, C). i / / D g H i W i Based on the absolute position offset unit of each convolutional kernel in each group, the pixel value of each pixel point on the sub-input feature map (which corresponds to the pixel point on the absolute position offset unit of the convolutional kernel) is obtained. For each group of sub-input feature maps, the shape of the result after grid sampling is (N, 1, H). o W o Since the sub-input feature maps have a total of C... i / / D g Group, for C i / / D g After mapping, GS is obtained. k,j GS k,j The shape is (N, 1, C) i / / D g H o W o ), here GS k,j There is one more dimension, making a total of five dimensions.

[0062] It should be noted that the target interpolation method can also be other interpolation methods. The above is only an example of bilinear interpolation and does not mean that the embodiments of this application are limited to this.

[0063] In this embodiment, P×M absolute position offset units for convolutional kernels are calculated based on a preset reference position and P×M relative position offset units for convolutional kernels. Each of these P×M absolute position offset units is used to perform one-to-one grid sampling on P groups of sub-input feature maps, resulting in P×M sampling result units. By dividing the relative position offsets of each group of sub-convolutional kernels and then performing one-to-one grid sampling on the sub-input feature maps, the single operator is decomposed into vector calculations that edge chips excel at, thus making reasonable use of the edge chip's computing resources and improving computational efficiency.

[0064] In some embodiments, grid sampling is achieved in image processing using a target interpolation method. Target interpolation methods include, but are not limited to, bilinear interpolation, linear interpolation, bicubic interpolation, and nearest neighbor interpolation. Based on this, S1031 above can be implemented as follows: Using the target interpolation method, based on the relative position offset of the k-th sub-convolutional kernel in the relative position offset of the P-group sub-convolutional kernels, grid sampling is performed on the k-th sub-input feature map of the P-group sub-input feature map to obtain the k-th sampling result.

[0065] Similarly, when calculating each sampling result unit, the target interpolation method can be used. Taking the k-th group and j-th sampling result unit as an example, the target interpolation method is used to perform grid sampling on the k-th group sub-input feature map of the P-group sub-input feature map based on the absolute position offset unit of the k-th group and j-th convolution kernel, so as to obtain the k-th group and j-th sampling result unit.

[0066] Nearest neighbor interpolation directly assigns the grayscale value of the pixel to be calculated to its nearest neighbor. This method has low computational cost, reduces resource consumption, and improves computation speed. Linear interpolation, a first-order interpolation method, estimates the value of an unknown point by performing linear interpolation between two known points. This method is smoother than nearest neighbor interpolation, improving image quality. Bilinear interpolation is an extension of linear interpolation in two directions, performing linear interpolation not only in the x-direction but also in the y-direction. This method is smoother than linear interpolation, is applicable to two-dimensional image interpolation, and improves image quality. Bicubic interpolation is a more advanced interpolation method that uses a weighted average of the surrounding 16 pixels to calculate the value. This method can provide even smoother image results.

[0067] In this embodiment, different interpolation methods can be used to implement grid sampling and obtain sampling results, which increases the diversity and richness of the grid sampling implementation process.

[0068] In some embodiments, when performing position modulation, the sub-position modulation amounts in the P groups can be divided into sub-position modulation amounts before position modulation is performed.

[0069] For example, each sub-position modulation quantity in the P groups of sub-position modulation quantities is divided into P×M position modulation quantity units; M represents the size of the convolution kernel, M is a positive integer, and the k-th sampling result includes M sampling result units. Based on this, the above S1032 can also be implemented in the following way. According to the k-th group, the j-th sampling result unit and the k-th group and the j-th position modulation quantity units in the P×M position modulation quantity units, the k-th group and the j-th modulated group sampling feature map unit are determined, thereby obtaining the P-group modulated group sampling feature map; the k-th group modulated group sampling feature map includes M modulated group sampling feature map units, j is a natural number less than M.

[0070] In this embodiment, the grouped position modulation amounts (i.e., sub-position modulation amounts) are divided according to the convolution kernel size to obtain the position modulation amount units required for grid cell calculation. For example, for the k-th group of MS... k The tensor can be partitioned along axis 1 into M groups, where the modulation unit at position j in the k-th group is represented by MS. k,j This means that j = 0, 1, ..., K h K w -1, MS k,j The shape is (N, 1, H) o W o ).

[0071] Using the j-th position modulation unit MS in the k-th group k,j For the above k-th group and j-th sampling result unit GS k,j Perform position modulation, that is, modulate the j-th position modulation unit MS of the k-th group. k,j With the j-th sampling result unit GS of the k-th group k,j The Hadamard product is used as the modulated group sampling feature map unit of the k-th group and the j-th group.

[0072] The Hadamard product is a matrix operation defined as the element-wise multiplication of two matrices at the same positions, requiring that the two matrices have identical dimensions. The definition of the Hadamard product is as follows: Given two matrices A and B, whose elements are a... ij and b ij The Hadamard product of A and B is denoted as A*B, and its elements are (A*B). ij =a ij ×b ijThe Hadamard product is also called the Schur product or element-wise product. The k-th modulated grouped sampled feature map unit MO... k,j =GS k,j *MS k,j , where the symbol * represents element-wise multiplication, and pixel-to-pixel multiplication means multiplying pixels at corresponding positions in the matrix.

[0073] In this embodiment, based on the aforementioned P×M sampling result units and P×M position modulation units, P×M modulated grouped sampling feature map units are determined one-to-one. The P×M modulated grouped sampling feature map units are then concatenated to obtain P groups of modulated grouped sampling feature maps; each group of modulated grouped sampling feature maps includes M modulated grouped sampling feature map units. By decomposing the single operator into vector calculations that the edge chip excels at, the computational resources of the edge chip are rationally utilized, thus improving computational efficiency.

[0074] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0075] like Figure 4 As shown, Figure 4 This is a flowchart of another image processing method provided in the embodiments of this application, which includes S201-S203.

[0076] S201. The input data of deformable convolution, the relative position offset of the convolution kernel of deformable convolution, and the position modulation amount of deformable convolution are grouped according to the number of deformable channel groups.

[0077] After grouping the input feature map, convolutional kernel relative position offset, and position modulation amount according to the number of deformable channel groups to obtain P groups of sub-input feature maps, P groups of sub-convolutional kernel relative position offsets, and P groups of sub-position modulation amounts, each sub-convolutional kernel relative position offset in the P groups of sub-convolutional kernel relative position offsets and each sub-position modulation amount in the P groups of sub-position modulation amounts are further divided according to the size of the convolutional kernel, resulting in P×M convolutional kernel relative position offset units and P×M position modulation amount units, respectively.

[0078] For example, for the PO of the kth group k MS k The tensor is partitioned along axis 1, resulting in M ​​groups. The relative position offset units of the k-th group and the j-th convolutional kernel, and the position modulation units of the k-th group and the j-th group are represented by PO. k,j and MS k,j This means that j = 0, 1, ..., K h K w -1, their shapes are (N, 2, H) o W o) and (N, 1, H o W o ).

[0079] It should be noted that the execution steps of dividing the relative position offset of each group of sub-convolutional kernels and the modulation amount of each group of sub-positions according to the size of the convolutional kernels are not sequential and can be executed simultaneously; alternatively, the relative position offset of the sub-convolutional kernels can be divided first, followed by the modulation amount of the sub-positions; or the modulation amount of the sub-positions can be divided first, followed by the relative position offset of the sub-convolutional kernels. This application does not impose any limitations on these steps.

[0080] S202. Perform grid sampling on the grouped input feature map, the relative position offset of the grouped convolutional kernel, and the position modulation amount of the grouped input feature map one by one to obtain the modulated grouped sampled feature map. Then, stitch the modulated grouped sampled feature maps together to obtain the modulated complete sampled feature map.

[0081] Based on the preset reference position and the relative position offset units of the k-th and j-th convolutional kernels in the P×M convolutional kernel relative position offset units, the absolute position offset units of the k-th and j-th convolutional kernels are determined, i.e., AP k,j =BP k,j +PO k,j And so on, we obtain the k-th group of convolution kernel absolute position offset units, which includes M convolution kernel absolute position offset units.

[0082] Each kernel absolute position offset unit in the k-th sub-convolutional kernel absolute position offset is used to perform grid sampling on the k-th sub-input feature map, while simultaneously performing position modulation processing. That is, the k-th group and j-th kernel absolute position offset unit are used to perform grid sampling on the k-th sub-input feature map in the P-th sub-input feature map, resulting in the k-th group and j-th sampling result unit, i.e., GS. k,j =grid_sample(IF k AP k,j ,mode=bilinear),GS k,j The shape is (N, 1, C) i / / D g H o W o Similarly, using the absolute position offset units of each convolutional kernel in the k-th group of convolutional kernel absolute position offset units, grid sampling is performed on the k-th group of sub-input feature maps in the P groups of sub-input feature maps to obtain the k-th group of sampling results, which includes M sampling result units. Based on the k-th group and the j-th sampling result units and the k-th group and the j-th position modulation units in the P×M position modulation units, the k-th group and the j-th modulated grouped sampling feature map units are determined, that is, MO k,j =GSk,j *MS k,j MS k,j The shape is (N, 1, H) o W o MO k,j shape and GS k,j The shapes are consistent, being (N, 1, C). i / / D g H o W o Similarly, we can obtain M modulated grouped sampling feature map units in the k-th group, where M equals K. h K w For the obtained K h K w MO k,j The unit tensor results (i.e., the modulated grouped sampled feature map units) are concatenated to obtain the k-th modulated grouped sampled feature map MO. k MO k The shape is (N, K) h K w C i / / D g H o W o For the other groups, the same grid modulation method as the k-th group is used to obtain the grouped sampling feature map after P-group modulation.

[0083] By concatenating the P modulated grouped sampling feature maps, a complete modulated sampling feature map is obtained, where P equals D. g For D g MO of each group k The tensors are concatenated along axis 2 to obtain the modulated complete sampled feature map (MO), with the shape of MO being (N, K). h K w C i H o W o ).

[0084] S203. Perform convolution operation based on the convolution kernel and the modulated complete sampled feature map to obtain the final deformable convolution result.

[0085] In some embodiments, the above Figure 2S105 can be implemented in the following way: According to the structure of the output feature map, the modulated complete sampled feature map is transformed to obtain the first matrix; according to the structure of the output feature map, the convolution kernel is transposed and then transformed to obtain the second matrix; matrix multiplication is performed on the first matrix and the second matrix to obtain the convolution result; according to the structure of the output feature map, the convolution result is transformed to obtain the deformable convolution result of the image to be processed.

[0086] In this embodiment, the convolution process can be performed using efficient matrix multiplication. The shape of the modulated complete sampled feature map (MO) is (N, K) h K w C i H o W o The shape of the convolution kernel (CK) can be represented as (C o C i K h K w The shape of the output feature map is (N, C). o H o W o Based on this, before matrix multiplication, the shape of MO is transformed into the first matrix (MO'), and the shape of MO' is (N, K). h K w C i H o W o Transposing CK yields (C) o K h K w C i The shape is transformed into a second matrix (CK'), and the shape of CK' is (C o K h K w C i Then, matrix multiplication is performed on MO' and CK' to obtain the convolution result (MM), that is, MM = matmul(CK', MO'), where matmul represents the function for performing matrix multiplication. The shape of MM is (N, C). o H o W o The deformable convolution result is obtained by transforming the shape of the modulo-matrix (MM). This deformable convolution result is the output feature map, and its shape is (N, C). o H o W o ).

[0087] In this embodiment of the application, by performing shape transformation on the modulated complete sampled feature map according to the structure of the output feature map, and by transposing the convolution kernel and then performing shape transformation, the two transformed parts can be multiplied by matrix, and the output feature map can be obtained after matrix multiplication, which reduces the complexity of the operation and improves the efficiency of the operation.

[0088] Based on the image processing method provided in the above embodiments, Figure 5 This is a schematic diagram of an edge chip structure provided in an embodiment of this application. The edge chip can be implemented as part or all of an edge device by software, hardware, or a combination of both. See also... Figure 5 The edge chip 50 includes a neural network computing unit 501, a vector computing unit 502, a data access unit 503, and a data storage unit 504. The data storage unit 504 is an internal storage unit of the edge chip, used to store input feature maps, convolutional kernel relative position offsets, and position modulation amounts. The data access unit 503 is used to group the input feature maps, convolutional kernel relative position offsets, and position modulation amounts into P groups of sub-input feature maps, P groups of sub-convolutional kernel relative position offsets, and P groups of sub-position modulation amounts; P represents the number of deformable channel groups. P is a factor of the number of input channels in the input feature map, and P is a positive integer; the vector calculation unit 502 is used to perform grid sampling on the P group sub-input feature maps according to the relative position offset of the P group sub-convolution kernels and the modulation amount of the P group sub-positions, to obtain the P group modulated group sampled feature maps; the data access unit 503 is also used to stitch the P group modulated group sampled feature maps to obtain the modulated complete sampled feature map; the neural network calculation unit 501 is used to perform convolution on the modulated complete sampled feature map according to the convolution kernel to obtain the deformable convolution result of the image to be processed.

[0089] In this embodiment, the computing units included in the edge chip (e.g., an edge AI chip) can be divided into a neural network computing unit 501 (NU), a vector computing unit 502 (VU), a data access unit 503, and a data storage unit 504 (DM). The NU can be used to accelerate neural network calculations such as convolution and matrix multiplication, the VU can be used to handle some relatively flexible vector calculations, and the DM is on-chip storage. Storing the input feature map, the relative position offset of the convolution kernel, and the position modulation amount on-chip can reduce data access time and improve data processing efficiency. The data access unit 503 can be used to move and transform (e.g., group or concatenate) the data executed by the NU and VU. Based on the start and end positions of the data, the DM moves and temporarily stores the data that needs to be grouped or concatenated. The NU performs corresponding matrix multiplication operations on the moved data, and the VU performs corresponding vector operations on the moved data. After the operation is completed, the data access unit moves the data to the DM. Edge chips typically have relatively limited computing power and memory size, thus limiting their ability to perform complex calculations. For deformable convolution, traditional deployment methods struggle to meet real-time requirements. This application addresses this by performing an equivalent transformation on deformable convolution operations, decomposing the single operator into a series of computational flows convenient for edge chips. Specifically, the matrix multiplication step between the convolution kernel and the modulated complete sampled feature map can be efficiently executed on the numeric core (NU). Grid sampling, highly flexible due to the dynamic adjustment of sampling position offsets, allows for efficient execution on the virtual unit (VU) of the input feature maps of the P sub-groups, based on the relative position offsets of the P sub-convolution kernels and the modulation amounts of the P sub-group positions. Through grouping and unit partitioning, computation can be performed separately for each group of data or each unit within each group, reducing the centralized on-chip storage requirements and improving data processing efficiency.

[0090] Edge chips can also employ scheduling optimization strategies to optimize the overall process. For example, they can group data in parallel, perform grid sampling on each group in parallel, and perform position modulation on each group in parallel. This improves convolution operation efficiency and resource utilization. This image processing method can make reasonable use of the edge chip's computing resources, achieving more efficient performance and improving convolution operation efficiency.

[0091] The edge chip and image processing method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0092] In some embodiments, the data access unit 503 is used to control the data transfer between the neural network computing unit 501, the vector computing unit 502 and the data storage unit 504, as well as the data transfer between the external storage unit and the data storage unit 504 for the edge chip, and to temporarily store the transferred data.

[0093] In this embodiment, the data access unit 503 is used for data transfer and temporary storage within the edge chip. This embodiment does not limit the specific structure of the data access unit 503, as long as it has the function of transferring data and controlling the execution of data processing logic. Taking direct memory access (DMA) as an example, DMA has two transfer processes: one is transferring data between the DM and the NU or VU; the other is transferring off-chip data between the DM and the DM's on-chip data. Through the data access unit, convolution kernel matrix multiplication operations on the NU and vector calculations on the VU can be implemented using on-chip storage, reducing data access time and improving data processing efficiency.

[0094] It should be noted that the edge chip provided in the above embodiments is only illustrated by the division of the above functional modules when processing images. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the edge chip can be divided into different functional modules to complete all or part of the functions described above.

[0095] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0096] Based on the image processing method provided in the above embodiments, Figure 6 This is a schematic diagram of the structure of an edge device provided in an embodiment of this application, such as... Figure 6 As shown, the edge device 60 includes: an edge chip 50, a memory 601, and a computer program 602 stored in the memory 601 and executable on the edge chip 50. When the edge chip 50 executes the computer program 602, it implements the steps in the image processing method in the above embodiments.

[0097] Edge device 60 includes, but is not limited to, routers, routing switches, integrated access devices (IADs), multiplexers, and various metropolitan area network (MAN) and wide area network (WAN) access devices. This application embodiment does not limit the type of edge device 60. Those skilled in the art will understand that… Figure 6 This is merely an example of edge device 60 and does not constitute a limitation on edge device 60. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0098] The memory 601 can be an external storage device of the edge device 60, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the edge device 60. The memory 601 is used to store the operating system, applications, boot loader, data, and other programs. The memory 601 can also be used to temporarily store data that has been output or will be output.

[0099] This application also provides an edge device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0100] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0101] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.

[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0103] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0104] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The input feature map of the image to be processed, the relative position offset of the deformable convolution kernel, and the position modulation amount of the deformable convolution are obtained; the position modulation amount indicates the feature value weight corresponding to the position offset of the convolution kernel. The input feature map, the relative position offset of the convolution kernel, and the position modulation amount are grouped to obtain P groups of sub-input feature maps, P groups of sub-convolution kernel relative position offsets, and P groups of sub-position modulation amounts; where P represents the number of deformable channel groups, P is a factor of the number of input channels in the input feature map, and P is a positive integer; Based on the relative position offset of the P group sub-convolutional kernels and the modulation amount of the P group sub-positions, grid sampling is performed on the P group sub-input feature maps to obtain the P group modulated grouped sampled feature maps. The P groups of modulated grouped sampling feature maps are stitched together to obtain the complete modulated sampling feature map; The modulated complete sampled feature map is convolved using a convolution kernel to obtain the deformable convolution result of the image to be processed.

2. The method as described in claim 1, characterized in that, The step of performing grid sampling on the P-group sub-input feature maps according to the relative position offset of the P-group sub-convolutional kernels and the modulation amount of the P-group sub-positions to obtain the P-group modulated grouped sampled feature maps includes: Based on the relative position offset of the kth sub-convolutional kernel in the P-group sub-convolutional kernel relative position offset, grid sampling is performed on the kth sub-input feature map in the P-group sub-input feature map to obtain the kth sampling result; where k is a natural number less than P; Based on the sampling results of the kth group and the sub-position modulation amount of the kth group in the P group, the group sampling feature map after modulation of the kth group is determined, thereby obtaining the group sampling feature map after modulation of the P group.

3. The method as described in claim 2, characterized in that, The method further includes: The relative position offset of each group of sub-convolutional kernels in the P groups of sub-convolutional kernel relative position offsets is divided to obtain P×M convolutional kernel relative position offset units; where M represents the size of the convolutional kernel and M is a positive integer. The step of performing grid sampling on the k-th sub-input feature map in the P-group sub-input feature map based on the relative position offset of the k-th sub-convolutional kernel in the P-group sub-convolutional kernel relative position offset, to obtain the k-th sampling result, includes: Based on the preset reference reference position and the relative position offset units of the kth and jth convolutional kernels in the P×M convolutional kernel relative position offset units, the absolute position offset units of the kth and jth convolutional kernels are determined; the preset reference reference position is related to the sliding rule of the convolutional kernels, where k is a natural number less than P and j is a natural number less than M. Using the absolute position offset unit of the kth and jth convolutional kernels, grid sampling is performed on the kth sub-input feature map of the P-group sub-input feature map to obtain the kth and jth sampling result units, thereby obtaining the kth sampling result; the kth sampling result includes M sampling result units.

4. The method as described in claim 2, characterized in that, The method further includes: Each sub-position modulation quantity in the P groups of sub-position modulation quantities is divided into P×M position modulation quantity units; M represents the size of the convolution kernel, M is a positive integer, and the kth group of sampling results includes M sampling result units; The step of determining the grouped sampling feature map after modulation of the kth group based on the kth group sampling results and the kth group sub-position modulation amount in the P group sub-position modulation amount includes: Based on the kth and jth sampling result units in the kth sampling result group and the kth and jth position modulation units in the P×M position modulation units, the kth and jth modulated group sampling feature map units are determined; the kth modulated group sampling feature map includes M modulated group sampling feature map units, where j is a natural number less than M.

5. The method as described in claim 2, characterized in that, The step of performing grid sampling on the k-th sub-input feature map in the P-group sub-input feature map based on the relative position offset of the k-th sub-convolutional kernel in the P-group sub-convolutional kernel relative position offset, to obtain the k-th sampling result, includes: The target interpolation method is adopted. Based on the relative position offset of the kth sub-convolution kernel in the relative position offset of the P group sub-convolution kernels, the kth sub-input feature map of the P group sub-input feature map is subjected to grid sampling to obtain the kth sampling result. The target interpolation method is bilinear interpolation, linear interpolation, bicubic interpolation or nearest neighbor interpolation.

6. The method according to any one of claims 1-5, characterized in that, The step of convolving the modulated complete sampled feature map with a convolution kernel to obtain the deformable convolution result of the image to be processed includes: Based on the structure of the output feature map, the modulated complete sampled feature map is subjected to shape transformation to obtain the first matrix; Based on the structure of the output feature map, the convolution kernel is transposed and then its shape is transformed to obtain the second matrix. Perform matrix multiplication on the first matrix and the second matrix to obtain the convolution result; Based on the structure of the output feature map, the shape transformation of the convolution result is performed to obtain the deformable convolution result of the image to be processed.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: The image to be processed is preprocessed to obtain the input feature map; the preprocessing is convolution or pooling.

8. An edge chip, characterized in that, The edge chip includes a neural network computing unit, a vector computing unit, a data storage unit, and a data access unit, wherein the data storage unit is the internal storage unit of the edge chip; The data storage unit is used to store the input feature map, the relative position offset of the convolution kernel, and the position modulation amount; The data access unit is used to group the input feature map, the relative position offset of the convolution kernel, and the position modulation amount to obtain P groups of sub-input feature maps, P groups of sub-convolution kernel relative position offsets, and P groups of sub-position modulation amounts; where P represents the number of deformable channel groups, P is a factor of the number of input channels in the input feature map, and P is a positive integer; The vector calculation unit is used to perform grid sampling on the P-group sub-input feature map according to the relative position offset of the P-group sub-convolution kernel and the modulation amount of the P-group sub-position, so as to obtain the P-group modulated grouped sampled feature map. The data access unit is also used to stitch together the P-group modulated grouped sampling feature maps to obtain a complete modulated sampling feature map; The neural network computing unit is used to convolve the modulated complete sampled feature map according to the convolution kernel to obtain the deformable convolution result of the image to be processed.

9. The edge chip as described in claim 8, characterized in that, The data access unit is used to control data transfer between the neural network computing unit, the vector computing unit, and the data storage unit; and to transfer data between the external storage unit of the edge chip and the data storage unit.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images