Convolution calculation method, data processing method and device, equipment and readable medium

Abstract

This disclosure provides a convolution calculation method, the method comprising: generating dynamic convolution kernels based on input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolution kernel; exchanging channel information between the feature channels of the input features to obtain feature information of each feature channel having exchanged channel information; and performing weighted processing on the feature information of the corresponding feature channels based on the dynamic convolution kernels corresponding to each feature channel to obtain the output value of each feature channel. This disclosure also provides a data processing method, apparatus, device, and readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of processor technology, and in particular to a convolution calculation method, data processing method and apparatus, device and readable medium. Background Technology

[0002] Convolutional Neural Networks (CNNs), as one of the representative network structures in deep learning technology, have made breakthrough progress and been widely used in many fields such as image processing, speech recognition, and natural language processing.

[0003] Convolutional layers are one of the basic building blocks of CNNs. CNNs use convolutional layers to perform convolution operations on input data to extract features. Compared to static convolution (where each convolutional layer has a single convolutional kernel, and the entire data array is convolved by a single kernel), dynamic convolution can generate a set of convolutional kernels for different data values ​​at different positions in the entire data array (referred to as data windows or sliding windows) based on the features of the input data. This allows for dynamic adjustment of the weights of each convolutional kernel based on the input features, achieving adaptive dynamic convolution.

[0004] Therefore, dynamic convolution is significantly superior to static convolution in improving the model's expressive power and performance. However, dynamic convolution will exacerbate the increase in the computational cost of the network model, leading to a greater overall computational burden. Summary of the Invention

[0005] This disclosure provides a kernel sharing method and apparatus, processing kernel, electronic device, and computer-readable medium based on a many-core system.

[0006] In a first aspect, this disclosure provides a convolution calculation method, which includes: generating dynamic convolution kernels based on input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolution kernel; exchanging channel information between the feature channels of the input features to obtain feature information of each feature channel that has exchanged channel information; and performing weighted processing on the feature information of the corresponding feature channels based on the dynamic convolution kernels corresponding to each feature channel to obtain the output value of each feature channel.

[0007] Secondly, this disclosure provides a data processing method, which includes: receiving predetermined data; processing the predetermined data using a neural network model to obtain a processing result of the predetermined data; wherein the neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the convolution calculation method described in the first aspect on the predetermined data.

[0008] Thirdly, this disclosure provides a convolution calculation device, which includes: a convolution kernel generation module for generating dynamic convolution kernels based on input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolution kernel; a channel information exchange module for exchanging channel information between feature channels of the input features to obtain feature information of each feature channel that has exchanged channel information; and a weighted processing module for weighting the feature information of the corresponding feature channels based on the dynamic convolution kernel corresponding to each feature channel to obtain the output value of each feature channel.

[0009] Fourthly, this disclosure provides a data processing apparatus, which includes: a receiving module for receiving predetermined data; and a processing module for processing the predetermined data using a neural network model to obtain a processing result of the predetermined data; wherein the neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the above-mentioned convolution calculation method on the input features of the predetermined data.

[0010] Fifthly, this disclosure provides a processing kernel for loading a neural network model to complete deep learning processing, wherein the neural network model includes at least one convolutional layer, and each convolutional layer is used to perform the above-described convolution calculation method according to the input features of predetermined data.

[0011] In a sixth aspect, this disclosure provides an electronic device comprising: a plurality of processing cores; and an on-chip network configured to interact with data between the plurality of processing cores and external data; wherein one or more processing cores store one or more instructions, and the one or more instructions are executed by the one or more processing cores to enable the one or more processing cores to perform the convolution calculation method or the data processing method described above.

[0012] In a seventh aspect, this disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processing core, implements the above-described convolution calculation method or the above-described data processing method.

[0013] The convolution calculation method, data processing method, apparatus, device, and readable medium provided in this disclosure can generate each dynamic convolution kernel corresponding to only one feature channel, thereby significantly reducing the computational load of convolution processing. After generating the dynamic convolution kernel, the channel information exchange between feature channels can avoid the loss of channel information. Based on the dynamic convolution kernel corresponding to each feature channel, the feature information of the corresponding feature channel is convolved to obtain the output value of each feature channel. The entire dynamic convolution processing can reduce the computational load while avoiding the loss of channel information exchange, improving model processing efficiency and enhancing network representation capabilities.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A schematic diagram of the processing flow for involution operations provided in this embodiment of the disclosure;

[0017] Figure 2 A flowchart of the convolution calculation method provided in the embodiments of this disclosure;

[0018] Figure 3 A flowchart illustrating the specific processing steps for channel information exchange provided in this embodiment of the disclosure;

[0019] Figure 4 A flowchart illustrating the specific process of weighted processing of feature information using dynamic convolutional kernels provided in this embodiment of the disclosure;

[0020] Figure 5 A flowchart illustrating the specific process of calculating feature information using dynamic convolutional kernels provided in this embodiment of the disclosure;

[0021] Figure 6 A schematic diagram illustrating the specific process of a convolution calculation method that first weights the dynamic convolution kernel, as provided in an embodiment of this disclosure;

[0022] Figure 7 A schematic diagram illustrating the specific process of a convolution calculation method that first convolves the input features, as provided in an embodiment of this disclosure;

[0023] Figure 8 This is a flowchart illustrating the specific process of generating dynamic convolutional kernels in this embodiment of the disclosure;

[0024] Figure 9 This is a flowchart illustrating the specific process of generating dynamic convolutional kernels in this embodiment of the disclosure;

[0025] Figure 10 A schematic diagram illustrating the specific process of the convolution calculation method with adjusted channel number provided in this embodiment of the disclosure;

[0026] Figure 11 A schematic diagram illustrating the specific process of a convolution calculation method that performs convolution calculation first, as provided in an embodiment of this disclosure;

[0027] Figure 12 A flowchart of a data processing method provided in an embodiment of this disclosure;

[0028] Figure 13 A block diagram of the convolution computing apparatus provided in the embodiments of this disclosure;

[0029] Figure 14 This is a block diagram of the data processing apparatus provided in the embodiments of this disclosure;

[0030] Figure 15 A block diagram of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0033] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0035] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0036] In some application scenarios, to solve the problem of large computational complexity of dynamic convolution, convolution operations can be performed through Involution operations. The parameter values of the convolution kernel corresponding to Involution at a spatial position are related to (or only determined by) the input feature vector corresponding to this spatial position. Therefore, different weights can be adaptively assigned to different spatial positions. To better understand Involution, the following description compares Involution with standard convolution.

[0037] In the embodiments of the present disclosure, the standard convolution (i.e., conventional convolution) operation has two basic characteristics: spatial agnosticism and channel specificity. The former ensures weight sharing between different spatial regions and realizes translational invariance; the latter is responsible for encoding feature information into different channels to enhance feature expression.

[0038] Contrary to the above characteristics of standard convolution, the two basic characteristics of Involution include channel agnosticism and spatial specificity. Since Involution can divide the input features into groups and divide each convolution kernel into corresponding groups for convolution within the corresponding groups, its channel agnosticism is reflected in that the features within each group in the channel dimension share the parameters of one convolution kernel; the spatial specificity is manifested as different convolution kernels are used at different spatial positions within the same group.

[0039] In the embodiments of the present disclosure, the number of parameters required for the Involution Kernel is equal to H*W*k*k*g. Here, H and W represent the height and width of the input feature, such as the Feature Map, k represents the size of the Involution Kernel, g represents the number of groups (Group) of the Involution operation, and g is much smaller than the number of feature channels C of the input feature (g << C), where all feature channels share g convolution kernels.

[0040] In the following description of the embodiments, the same symbols, such as H, W, k, and g, represent the same meanings and will not be elaborated in the embodiments of the present disclosure.

[0041] In Involution, a fixed weight matrix is not used as a learnable parameter. Instead, the corresponding Involution Kernel can be generated based on the input features, so as to ensure that the Kernel Size and the size of the input features can be automatically aligned in the spatial dimension.

[0042] Figure 1 This is a schematic diagram of the processing flow for the involute operation provided in an embodiment of this disclosure.

[0043] exist Figure 1 In this context, the computational process of involution operations can be divided into two parts: generating the involution kernel and calculating the involution. The specific processing flow of these two parts is described below.

[0044] First, generate the involution kernel.

[0045] In step S01, as Figure 1 In As shown, for an input feature X (size: height H * width W * number of feature channels C) at a spatial location, the feature vector v... i，j ∈R C×1×1 (1≤i≤H, 1≤j≤W), through processing unit The eigenvector is processed sequentially using FC-BN-RELU-FC to obtain ω∈R 1×1×K×K , where k represents the size of the involution kernel.

[0046] In this step, FC stands for Fully Connected (FC), BN stands for Batch Normalization (BN), and ReLU stands for Rectified Linear Unit (RLU), which is typically implemented using the ReLU activation function. The first FC reduces dimensionality, preventing overfitting due to excessive neuron parameters. BN-RELU uses batch normalization to accelerate convergence and uses the ReLU activation function to prevent gradient problems during backpropagation (such as vanishing and exploding gradients). The last FC acts as a classifier.

[0047] In step S02, as Figure 1 As shown in the "reshape" diagram, for ω∈R 1×1×K×K Perform a reshape operation to obtain the feature vector v. i，j ∈R C×1×1 The corresponding convolution kernel H∈R K×K×1 .

[0048] In this step, the reshape operation can transform data of one channel dimension (w∈R) into a single channel dimension. 1×1×K×K This involves rearranging the data in a spatial dimension (channel-to-space) to achieve data dimensionality transformation. For example, in this embodiment of the disclosure, ω∈R 1×1×K×KThe reshape operation unfolds the input feature into the shape of a convolution kernel, thus obtaining the involute kernel corresponding to the feature vector at a spatial location of the input feature.

[0049] Next, calculate the involution.

[0050] This step is used to perform a multiplication-add operation (Multiply-Add) on the generated convolutional kernel and the feature vectors in the neighborhood of this coordinate point on the input feature map (e.g., ...). Figure 1 In This indicates element-wise multiplication. (This represents element-wise addition) to obtain the output feature Y.

[0051] In step S03, as Figure 1 middle As shown, H and v i，j ∈R C×1×1 Multiplying the vectors in the neighborhood yields the first eigenvector (R). k×k×C This step can be understood as stretching the involution kernel H into a k*k*C vector.

[0052] In step S04, as Figure 1 middle As shown, the first feature vector is added dimension by dimension to obtain the final output feature Y. This step can be understood as adding k*k vectors of size 1*1*C to replace the original feature vector, thus completing the involution operation.

[0053] Based on the processing flow of the involution operation described above, the involution convolution operation can perform two 1*1 convolution transformations on the input features to obtain new features with the number of channels co*(ci / g)*k*k. Then, the feature information at each spatial location is resized into a convolution kernel of size (co, ci / g, k, k), which is then broadcast to (co, ci, k, k). Finally, it can be convolved with each sliding window of the input features to obtain the output features. Here, ci is the number of input feature channels, co is the number of output feature channels, and the meanings of k and g are the same as those described above, and will not be repeated here.

[0054] As can be seen from the above processing flow, the basic principle of involution is: to generate standard convolution kernels for each spatial location of the input features (which can be shared g times in the input channels), and then use these convolution kernels to perform convolution operations on the input features.

[0055] pass Figure 2It can be seen that the computational cost of involution is divided into two parts: generating the involution kernel and calculating the involution. The computational cost of these two parts is lower than that of standard convolution. However, considering that the value of the number of channels co*(ci / g)*k*k is not small, if this value is reduced by increasing the number of groups g, it will significantly affect the performance of the network model. Therefore, if the value of this number of channels can be reduced, the computational cost of the network model can be significantly reduced.

[0056] In this embodiment, the convolution processing flow can be improved based on the working principles of depthwise convolution (DWC) and pointwise convolutional (PWC). DWC can be understood as a grouped convolution where the number of groups is equal to the number of input feature channels, with one convolution kernel corresponding to one feature channel, and each feature channel being convolved by only one kernel. However, DWC performs convolution operations independently on each channel of the input layer, failing to effectively utilize the feature information of different channels at the same spatial location. Therefore, PWC is needed to combine the feature information of different channels at the same spatial location to generate new feature information.

[0057] Based on this, the present disclosure provides a convolution calculation method that can reduce the computational load of the network model while ensuring the exchange of channel information.

[0058] Figure 2 A flowchart illustrating a convolution calculation method provided in this embodiment of the disclosure. (Refer to...) Figure 2 This disclosure provides a convolution calculation method, which includes the following steps.

[0059] S210, Generate dynamic convolution kernels based on the input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolution kernel.

[0060] S220, channel information is exchanged between the feature channels of the input features to obtain the feature information of each feature channel that has exchanged channel information.

[0061] S230, based on the dynamic convolution kernel corresponding to each feature channel, performs weighted processing on the feature information of the corresponding feature channel to obtain the output value of each feature channel.

[0062] According to the convolution calculation method of this disclosure, each dynamic convolution kernel generated based on the input features corresponds to only one feature channel (referring to the principle in the DWC operation where one convolution kernel corresponds to one feature channel, and one feature channel is convolved by only one convolution kernel), thereby significantly reducing the computational load of subsequent convolution processing. After generating the dynamic convolution kernel, channel information exchange between feature channels (referring to the PWC operation where feature information of different channels at the same spatial location is combined) can avoid the loss of channel information exchange. Based on the dynamic convolution kernel corresponding to each feature channel, the feature information of the corresponding feature channel is convolved to obtain the output value of each feature channel. The entire dynamic convolution processing can reduce the computational load while avoiding the loss of channel information exchange, improving model processing efficiency and enhancing network representation capabilities.

[0063] In step S210 above, the step of generating dynamic convolution kernels may include: performing two 1*1 convolution transformations on the input features to obtain new features with the number of channels co*(ci / g)*k*k, and then resizing each feature point into a convolution kernel of size (1, ci / g, k, k), where ci is the number of input feature channels, co is the number of output feature channels, and k and g have the same meaning as k and g above, and will not be repeated here.

[0064] In this embodiment, the steps for generating dynamic convolution kernels are basically the same as those for generating involute convolution kernels. The difference is that the size of the involute convolution kernel is (co, ci / g, k, k), while the size of the dynamic convolution kernel in this embodiment is (1, ci / g, k, k). Each feature channel of the input feature corresponds to a dynamic convolution kernel. Compared with involute convolution, the computational cost of subsequent convolution operations is significantly reduced.

[0065] Figure 3 A flowchart illustrating the specific processing steps for channel information exchange provided in this embodiment of the disclosure.

[0066] Reference Figure 3 In some embodiments, step S220 may specifically include the following steps.

[0067] S31, perform average pooling on the feature information of each feature channel of the input feature to obtain the average pooled feature information of each feature channel.

[0068] In this step, average pooling is used to divide the input features into several sub-regions in space, and outputs the average value of all elements for each sub-region. Average pooling can enhance feature extraction within a channel and retain more feature information of the predetermined data (such as background information of an image).

[0069] Specifically, when performing average pooling on the feature information of each feature channel, the mean of the feature vectors within a specific range can be calculated for each spatial location of the input feature, based on a preset neighborhood window, to obtain the average pooled feature information of each feature channel. Average pooling does not affect the information interaction between feature channels; it is an operation performed within each channel. Average pooling can enhance feature extraction within a channel. To enable the neural network model to perform subsequent processing based on the multi-channel fused feature information, it is also necessary to fuse the channel information between feature channels.

[0070] S32, based on the feature information after average pooling, perform channel information exchange between feature channels to obtain the feature information of each feature channel that has exchanged channel information.

[0071] In step S32, channel information refers to the different feature information corresponding to different feature channels. For example, a linear transformation is performed between different feature channels using a convolution kernel of a predetermined size, thereby achieving the exchange of channel information between feature channels. As a specific example, the predetermined size can be, for example, 1*1, 3*3, 5*5, ..., n*n, where n is an integer greater than or equal to 1, for example, an odd number greater than or equal to 1. This disclosure does not limit the specific value of n.

[0072] Taking a 1*1 convolution kernel as an example, the convolution calculation is performed on the feature information after average pooling using a 1*1 convolution kernel, thereby achieving the fusion of different feature information between channels. This allows the neural network to perform subsequent processing based on the feature information fused from multiple channels.

[0073] In this step, since the dynamic convolution kernel generated in step S110 corresponds to only one feature channel, and one feature channel is convolved by only one convolution kernel, the convolution operation performs convolution calculation independently on each feature channel of the input feature, without effectively utilizing the feature information of different channels at the same spatial location. Therefore, steps S21 and S22 can be used to perform average pooling and channel information exchange processing on the feature information of each feature channel of the input feature, so as to exchange the channel information of the average pooled feature information between the feature channels, forming the feature information of each feature channel that has exchanged channel information. This solves the problem of missing channel information exchange caused by the characteristic that each dynamic convolution kernel generated in this embodiment corresponds to only one feature channel, and completes the channel information exchange between feature channels.

[0074] Figure 4 A flowchart illustrating the specific process of weighting feature information using dynamic convolutional kernels provided in this embodiment of the disclosure. (Refer to...) Figure 4 In some embodiments, step S230 may specifically include the following steps.

[0075] S41, multiply the dynamic convolution kernel corresponding to each feature channel with the feature information of the corresponding feature channel to obtain the weighted convolution kernel corresponding to each feature channel.

[0076] S42 uses the weighted convolution kernel corresponding to each feature channel to perform convolution calculation on the input features and obtain the output value of each feature channel.

[0077] Through steps S41 and S42 above, the dynamic convolution kernels corresponding to each feature channel are first weighted, and then the input features are partially convolved using the weighted convolution kernels to obtain the output value of each feature channel. In some application scenarios, weighting the dynamic convolution kernels themselves first, and then using the weighted convolution kernels for convolution calculation, can accelerate the extraction of some important features from the input features.

[0078] Figure 5 A flowchart illustrating the specific process of calculating feature information using a dynamic convolutional kernel provided in this embodiment of the disclosure. (Refer to...) Figure 5 In some embodiments, step S230 may specifically include the following steps.

[0079] S51 uses the dynamic convolution kernel corresponding to each feature channel to perform convolution calculation on the input features to obtain the output features of each feature channel.

[0080] S52, multiply the output feature of each feature channel with the feature information of the corresponding feature channel to obtain the output value of each feature channel.

[0081] Through steps S51 and S52, the input features are first convolved using the dynamic convolution kernel corresponding to each feature channel. After convolution, the input features are multiplied point-by-point with the feature information of each feature channel that has had its channel information swapped, to obtain the output value of each feature channel. In some application scenarios, compared to the processing of steps S41 and S42, steps S51 and S52 have a lower memory access cost.

[0082] In this embodiment, the weighted processing method for the feature information of the corresponding feature channels based on dynamic convolution kernels, as described in steps S41 and S42 above, is the same as the corresponding processing method in steps S51 and S52 above, and the output features obtained by both are the same output features. In practical application scenarios, either processing method can be selected according to actual needs.

[0083] The following is through Figure 6 and Figure 7 This section describes the specific flow of the convolution calculation method according to embodiments of the present disclosure. Figure 6A schematic diagram illustrating the specific process of a convolution calculation method that first weights the dynamic convolution kernel, as provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram illustrating the specific process of a convolution calculation method that first convolves the input features, as provided in an embodiment of this disclosure.

[0084] exist Figure 6 and Figure 7 In the input feature x p The output feature is x q The number of feature channels for the input feature is c. p The number of feature channels in the output feature is c. q c p =c q =c num c num The specific value of the number of feature channels for the input feature, and c num It is an integer greater than or equal to 1.

[0085] like Figure 6 As shown, in some embodiments, the convolution calculation method includes the following steps.

[0086] As shown by the two "c1*1" in S61, for the received input feature x p Two 1x1 convolution transformations are performed to obtain new features.

[0087] As shown in S62, the new features are rearranged to obtain the input features x. p The dynamic convolution kernel corresponding to each feature channel is denoted as w1. p .

[0088] As shown in "Average Pooling" in S63, for input feature x i The feature information of each feature channel is averaged and pooled to obtain the averaged feature information of each feature channel.

[0089] As shown in "1*1" in S64, channel information is exchanged for the average pooled feature information of each feature channel to obtain the feature information of each feature channel after the channel information exchange, denoted as s. p .

[0090] Steps S61 and S62 described above can be performed in parallel with steps S63 and S64 described above to improve data processing efficiency.

[0091] As shown in "pointwise multiplication" in S65, the feature information of each feature channel that has had its channel information exchanged is broadcast and then compared with the input feature x. pThe dynamic convolution kernels corresponding to each feature channel are multiplied together to obtain a new dynamic convolution kernel for each feature channel, denoted as w2. p Among them, w2 p With w1 p They have the same shape.

[0092] As shown in "Dynamic Convolution" in S66, the new dynamic convolution kernel corresponding to each feature channel is denoted as w2. p With input features x p Perform dynamic convolution calculations to obtain the output value of each feature channel, denoted as x. q .

[0093] In steps S61-S66 above, the processing performed in steps S62 and S63 can also be called quasi-squeeze and excite (quasi-SE) processing, and the network performing the corresponding processing can be called quasi-SENet. Specifically, the squeezing excite transformation is used to explicitly model the interdependencies between feature channels in order to adaptively recalibrate the feature responses between channels. This allows the network to automatically acquire the importance of each feature channel through self-learning, and then, based on this importance, it enhances features useful for the current task and suppresses features that are not very useful for the current task.

[0094] The convolution calculation method of this invention can refer to the basic principle of squeeze excitation transformation, first perform average pooling on the input features, and then perform channel information exchange on the average pooled feature information. This allows the neural network to learn to use the fused feature information between multiple channels for subsequent calculations, avoids the loss of channel information exchange, and enhances the network's representation ability. In addition, the convolution calculation method of this invention can also generate dynamic convolution kernels corresponding to each feature channel based on the input features of predetermined data, so as to significantly reduce the amount of computation in subsequent convolution processing.

[0095] In some embodiments, Figure 7 The processing flow of the convolution calculation method shown is the same as Figure 6 The processing flow for convolution calculation shown is basically the same, that is, steps S71-S74 and steps S61-S64 perform the same processing flow, which will not be repeated in this embodiment. The difference lies in that... Figure 7 The input feature x is obtained from the process. p The dynamic convolution kernel corresponding to each feature channel (denoted as w1) p Following this, the steps S75 and S76 are as follows.

[0096] As shown in "Dynamic Convolution" in S75, each feature channel utilizes a dynamic convolution kernel w1.p For input feature x p Perform convolution calculations to obtain the output features of each feature channel, denoted as x. t .

[0097] As shown in S76, the "pointwise multiplication" multiplies the output feature x of each feature channel. t The feature information s of each feature channel that has exchanged channel information p Perform point-by-point multiplication to obtain the output value of each feature channel, denoted as x. q .

[0098] In steps S71-S76 above, the processing performed in S71-S72 and S75 can also be called quasi-involution processing, which draws on... Figure 1 The difference in the generation process of dynamic convolutional kernels in the described involution operation flow is that each generated convolutional kernel corresponds to a feature channel of the input feature, that is, each feature channel of the input feature corresponds to a dynamic convolutional kernel, relative to... Figure 1 The described involution operation process significantly reduces the computational load of subsequent convolution processing.

[0099] In this embodiment of the disclosure, Figure 6 and Figure 7 The processing results of convolution calculations are the same, but the former processing flow can accelerate the extraction of some important features from the input features, while the latter processing flow has a lower memory access cost compared to the former.

[0100] In practical applications, any processing method can be selected according to actual needs.

[0101] Figure 8 This is a flowchart illustrating the specific process of generating dynamic convolutional kernels in an embodiment of this disclosure. Figure 8 As shown, in some embodiments, step S210 may specifically include the following sub-steps.

[0102] S81, If ​​the number of feature channels of the input feature is different from the number of predetermined output feature channels, the size of the input feature is adjusted to the size of the predetermined output feature channels; S82, A dynamic convolution kernel is generated based on the size-adjusted input feature.

[0103] Through the above steps S81 and S82, when the number of feature channels of the input feature is different from the predetermined number of output feature channels, the size of the input feature is first adjusted so that the size of the adjusted input feature is the same as the size of the output feature channel. Then, a dynamic convolution kernel is generated based on the size-adjusted input feature so that it corresponds to the size of the output feature.

[0104] Figure 9 This is a flowchart illustrating the specific process of generating dynamic convolutional kernels in an embodiment of this disclosure. Figure 9 As shown, in some embodiments, step S230 may specifically include the following sub-steps.

[0105] S91, if the number of feature channels of the input feature is less than the predetermined number of output feature channels, then determine the existing feature channels and the added feature channels corresponding to the feature channels of the input feature according to the predetermined number of output feature channels.

[0106] S92, based on the dynamic convolution kernel corresponding to each feature channel in the existing feature channels, performs weighted processing on the feature information of the corresponding feature channels to obtain the first output value of each feature channel.

[0107] S93, merge the first output value of each feature channel with the feature information of the added feature channel to obtain the second output value of each feature channel.

[0108] Through the above steps S91-S93, when the number of feature channels of the input feature is different from the predetermined number of output feature channels, the convolution calculation operation of the unchanging channel's partial feature information is first performed on the feature vector in the first output result of each feature channel to obtain the feature vector in the first output result of each feature channel; then, the channel dimension transformation is performed on the feature vector corresponding to the currently missing feature channel of the input feature; the feature vector in the first output result of each feature channel and the feature vector obtained by the channel dimension transformation are merged to obtain the second output result, so as to fill in the number of the currently missing feature channels of the input feature, and the second output result is used as the final output result corresponding to the increase in the number of input feature channels.

[0109] In this embodiment of the disclosure, for the use of Figure 8 The description details the specific process for generating dynamic convolutional kernels, and how... Figure 9 The description details the specific process of weighted processing of feature information using dynamic convolution kernels. The results of the convolution calculations for both methods are the same. Since the computational cost of padding missing feature channels in the input features is relatively low, the combination of... Figure 9 The described convolution calculation process involves relatively little computation. In practical applications, the appropriate method can be selected based on actual needs. Figure 7 and Figure 8 Any of the processing methods described.

[0110] The following is through Figure 10 and Figure 11 This section describes the specific flow of the convolution calculation method according to embodiments of the present disclosure. Figure 10 A schematic diagram illustrating the specific process of the convolution calculation method with adjusted channel number provided in this embodiment of the disclosure; Figure 11This is a schematic diagram illustrating the specific process of a convolution calculation method that performs convolution calculations first, as provided in an embodiment of this disclosure.

[0111] like Figure 10 As shown, the convolution calculation method may include the following steps.

[0112] As shown in “c1*1” in S1001, the size of the input feature is adjusted to the size of the predetermined output channel.

[0113] As shown by the two "c1*1" in S1002, two 1*1 convolution transformations are performed based on the size-adjusted input features to obtain new features.

[0114] As shown in S1003, the new features are rearranged to obtain the input features x. p The dynamic convolution kernel corresponding to each feature channel is denoted as w1. p .

[0115] As shown in "Average Pooling" in S1004, for input feature x p The feature information of each feature channel is averaged and pooled to obtain the averaged feature information of each feature channel.

[0116] As shown in “c1*1” in S1005, channel information exchange is performed on the feature information after average pooling of each feature channel to obtain the feature information of each feature channel after the channel information exchange, denoted as s. p .

[0117] As shown in "Dynamic Convolution" in S1006, dynamic convolution kernel w1 is used for each feature channel. p For input feature x p Perform convolution calculations to obtain the output features of each feature channel, denoted as x. t .

[0118] As shown in "pointwise multiplication" in S1007, the output feature x of each feature channel is multiplied... t The feature information s of each feature channel that has exchanged channel information p Perform point-by-point multiplication to obtain the output value of each feature channel, denoted as x. q .

[0119] exist Figure 10 Steps S1003-S1007 in the above embodiment are combined with Figure 7 Steps S72-S76 described herein perform the same processing flow; in other embodiments, steps S1003-S1007 may also be combined with those in the above embodiments. Figure 6Steps S62-S66 described herein perform the same processing flow, and will not be repeated in this embodiment.

[0120] like Figure 11 As shown, the convolution calculation method may include the following steps.

[0121] As shown by the two "c1*1" in S1101, for the received input feature x p Two 1x1 convolution transformations are performed to obtain new features.

[0122] As shown in S1102, the new features are rearranged to obtain the input features x. p The dynamic convolution kernel corresponding to each feature channel is denoted as w1. p .

[0123] As shown in "Average Pooling" in S1103, for input feature x p The feature information of each feature channel is averaged and pooled to obtain the averaged feature information of each feature channel.

[0124] As shown in “c1*1” in S1104, channel information exchange is performed on the feature information after average pooling of each feature channel to obtain the feature information of each feature channel after the channel information exchange, denoted as s. p .

[0125] As shown in "Convolution Calculation" in S1105, the dynamic convolution kernel w1 corresponding to each feature channel is used. p For input feature x p Perform convolution calculations to obtain the output features of each feature channel, denoted as x. t .

[0126] As shown in S1106, "pointwise multiplication" multiplies the output features x of each feature channel. t The feature information s of each feature channel that has exchanged channel information p Perform point-by-point multiplication to obtain the first output value of each feature channel, denoted as x. 1q .

[0127] As shown in “c1*1” in S1107, the dimension of the input feature is adjusted by a 1*1 convolution kernel to supplement the input feature with additional channels.

[0128] As shown in S1108, the first output value of each feature channel is merged with the feature information of the added feature channel to obtain the second output value of each feature channel, denoted as x. 2q .

[0129] exist Figure 11 Steps S1101-S1106 in the above embodiment are combined with Figure 7 Steps S71-S76 described herein perform the same processing flow; in other embodiments, steps S1101-S1106 may also be combined with those in the above embodiments. Figure 6 Steps S61-S66 described herein perform the same processing flow, and will not be repeated in this embodiment.

[0130] In this embodiment of the disclosure, DWC can be used for bonding. Figure 1 The described involutional convolution is improved, corresponding to the above. Figure 7 , Figure 10 and Figure 11 The quasi-involution processing flow is described. In this embodiment, the quasi-involution processing flow yields a convolution kernel of (1, ci / g, k, k) instead of a convolution kernel of (co, ci / g, k, k). Therefore, the computational cost of weight generation is 1 / co of the computational cost of weight generation in the original involution operation; furthermore, the computational cost of depthwise convolution is less than that of standard convolution. The computational cost of the convolution operation after generating the dynamic convolution kernel in this embodiment is the same as that of the corresponding convolution operation in the original involution (where c is the kernel size of the standard convolution).

[0131] In this embodiment of the disclosure, channel information exchange processing can be performed in conjunction with PWC, corresponding to the above-described combination. Figure 7 , Figure 10 and Figure 11 The quasi-SENet processing flow is described. Quasi-SENet is executed in parallel with quasi-Involution, and channel information is exchanged through pointwise multiplication (1*1 convolution). Compared to the sequential execution of DWC and PWC (DWC first, then PWC), this embodiment executes quasi-Involution (corresponding to DWC) and quasi-SENet (corresponding to PWC) in parallel. Then, the feature information of each feature channel is weighted using the dynamic convolution kernel corresponding to each feature channel obtained from quasi-Involution.

[0132] When the number of feature channels in the input features is equal to the amount of data in the output feature channels, i.e., c i =c o =c num In this case, the computational cost of the quasi-Involution process is h×w×c. num It's from DWC. The computational cost of the quasi-SENet processing flow is h×w×c.num ×c num The computational load itself is not large.

[0133] Therefore, when the number of feature channels in the input features is equal to the amount of data in the output feature channels, the total computational cost of the convolution processing method in this embodiment is:

[0134] And combined Figure 1 The total computational cost of the described involution is If g takes the value 8, c num With a value of 512 and k = 3, the total computational cost of the convolution processing method in this embodiment of the present disclosure is... Figure 1 The total computational cost of the described involution is 1 / 57.6.

[0135] The total computational cost of the convolution processing method in this embodiment is greater than the total computational cost of static lightweight convolution DWC and PWC: (h×w×c) num ×k×k+h×w×c num ×c num However, the convolution processing method of this embodiment is dynamic convolution, and its performance far exceeds that of static convolution.

[0136] Figure 12 This is a flowchart of a data processing method provided in an embodiment of the present disclosure.

[0137] like Figure 12 As shown, in some embodiments, the data processing method may include the following steps.

[0138] S1201, Receive pre-ordered data.

[0139] S1202, using a neural network model to process predetermined data to obtain the processing result of the predetermined data; wherein, the neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the convolution calculation method described in the above embodiments on the predetermined data.

[0140] In some embodiments, the predetermined data may be any of the following data items: image data, audio data, video data, and text data. The neural network model processes the predetermined data to achieve applications such as target recognition, semantic segmentation, and reinforcement learning in the corresponding technical fields.

[0141] According to the data processing method of the present disclosure, each convolutional layer of the neural network model is used to perform the convolution calculation method described in the above embodiments on predetermined data. This method can significantly reduce the amount of computation in convolution processing, avoid the loss of channel information exchange, improve model processing efficiency, and enhance network representation capabilities.

[0142] Figure 13 This is a block diagram of a convolution calculation device provided in an embodiment of the present disclosure.

[0143] Reference Figure 13 This disclosure provides a convolution calculation device 1300, which includes the following modules.

[0144] The kernel generation module 1310 is used to generate dynamic convolution kernels based on the input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolution kernel.

[0145] The channel information exchange module 1320 is used to exchange channel information between the feature channels of the input feature to obtain the feature information of each feature channel that has exchanged channel information.

[0146] The weighted processing module 1330 is used to perform weighted processing on the corresponding feature information based on the dynamic convolution kernel corresponding to each feature channel to obtain the output value of each feature channel.

[0147] In some embodiments, the channel information exchange module 1320 includes: an average pooling unit, configured to perform average pooling on the feature information of each feature channel of the input feature to obtain the average pooled feature information of each feature channel; and a channel information exchange unit, configured to perform channel information exchange between feature channels based on the average pooled feature information to obtain the feature information of each feature channel that has exchanged channel information.

[0148] In some embodiments, the weighted processing module 1330 specifically includes: a first processing unit, configured to multiply the dynamic convolution kernel corresponding to each feature channel with the feature information of the corresponding feature channel to obtain the weighted convolution kernel corresponding to each feature channel; and a second processing unit, configured to perform convolution calculation on the input features using the weighted convolution kernel corresponding to each feature channel to obtain the output value of each feature channel.

[0149] In some embodiments, the weighted processing module 1330 specifically includes: a third processing unit, used to perform convolution calculation on the input features using the dynamic convolution kernel corresponding to each feature channel to obtain the output features of each feature channel; and a fourth processing unit, used to perform multiplication calculation on the output features of each feature channel and the feature information of the corresponding feature channel to obtain the output value of each feature channel.

[0150] In some embodiments, the kernel generation module 1310 is specifically used to: if the number of feature channels of the input feature is different from the number of predetermined output feature channels, adjust the size of the input feature to the size of the predetermined output feature channels; and generate a dynamic kernel based on the size-adjusted input feature.

[0151] In some embodiments, the weighted processing module 1330 specifically includes: a fifth processing unit, configured to determine, based on the predetermined number of output feature channels, existing feature channels and added feature channels corresponding to the feature channels of the input feature if the number of feature channels of the input feature is less than the number of predetermined output feature channels; a sixth processing unit, configured to perform weighted processing on the feature information of the corresponding feature channels based on the dynamic convolution kernel corresponding to each feature channel in the existing feature channels, to obtain a first output value of each feature channel; and a seventh processing unit, configured to merge the first output value of each feature channel with the feature information of the added feature channels to obtain a second output value of each feature channel.

[0152] According to the convolutional computing apparatus of this disclosure, each dynamic convolutional kernel generated based on the input features corresponds to only one feature channel, thereby significantly reducing the computational load of subsequent convolutional processing. After generating the dynamic convolutional kernel, channel information exchange between feature channels is used to avoid the loss of channel information exchange. Based on the dynamic convolutional kernel corresponding to each feature channel, the feature information of the corresponding feature channel is convolved to obtain the output value of each feature channel. The entire dynamic convolutional processing process can reduce the computational load while avoiding the loss of channel information exchange, thereby improving the model processing efficiency and enhancing the network representation capability.

[0153] Figure 14 This is a block diagram of a data processing apparatus provided in an embodiment of the present disclosure.

[0154] Reference Figure 14 This disclosure provides a data processing device 1400, which includes the following modules.

[0155] Receiver module 1410 is used to receive predetermined data;

[0156] The processing module 1420 is used to process predetermined data using a neural network model to obtain the processing result of the predetermined data; wherein the neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the convolution calculation method described in the above embodiments on the predetermined data.

[0157] According to the data processing apparatus of the present disclosure, each convolutional layer of the neural network model is used to perform the convolution calculation method described in the above embodiments on the predetermined data. This method can significantly reduce the computational load of convolution processing, avoid the loss of channel information exchange, improve model processing efficiency, and enhance network representation capabilities.

[0158] It should be clarified that this disclosure is not limited to the specific configurations and processes described in the foregoing embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0159] This disclosure also provides a processing core, which includes the above-described convolution calculation device or data processing device.

[0160] This disclosure also provides a processing core for loading a neural network model to complete deep learning processing. The neural network model includes at least one convolutional layer, and each convolutional layer is used to execute the convolution calculation method described in the above embodiments based on the input features of predetermined data.

[0161] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0162] Figure 15 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0163] Reference Figure 15 This disclosure provides an electronic device that includes multiple processing cores 1501 and an on-chip network 1502. The multiple processing cores 1501 are all connected to the on-chip network 1502, and the on-chip network 1502 is used to exchange data between the multiple processing cores and external data.

[0164] One or more processing cores 1501 store one or more instructions, and the one or more instructions are executed by one or more processing cores 1501 to enable one or more processing cores 1501 to perform the above-mentioned convolution calculation method and data processing method.

[0165] Furthermore, this disclosure also provides a computer-readable medium storing a computer program thereon, wherein the computer program, when executed by a processing kernel, implements the above-described convolution calculation method and data processing method.

[0166] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0167] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A convolution calculation method, comprising: Dynamic convolutional kernels are generated based on input features from predetermined data, wherein each feature channel of the input features corresponds to a dynamic convolutional kernel; the predetermined data includes at least one of image data, audio data, video data, and text data; Channel information is exchanged between the feature channels of the input feature to obtain feature information of each feature channel that has exchanged channel information. Based on the dynamic convolution kernel corresponding to each feature channel, the feature information of the corresponding feature channel is weighted to obtain the output value of each feature channel. The step of exchanging channel information between the feature channels of the input feature to obtain feature information of each feature channel that has had its channel information exchanged includes: The feature information of each feature channel of the input feature is averaged by pooling to obtain the averaged pooled feature information of each feature channel. Based on the feature information after average pooling, channel information is exchanged between feature channels to obtain the feature information of each feature channel that has exchanged channel information.

2. The method of claim 1, wherein, The step involves weighting the feature information of each feature channel based on the dynamic convolution kernel corresponding to each feature channel to obtain the output value of each feature channel, including: The dynamic convolution kernel corresponding to each feature channel is multiplied with the feature information of the corresponding feature channel to obtain the weighted convolution kernel corresponding to each feature channel. The input features are convolved using the weighted convolution kernels corresponding to each feature channel to obtain the output value of each feature channel.

3. The method of claim 1, wherein, The step involves weighting the feature information of each feature channel based on the dynamic convolution kernel corresponding to each feature channel to obtain the output value of each feature channel, including: The input features are convolved using the dynamic convolution kernels corresponding to each feature channel to obtain the output features of each feature channel. The output feature of each feature channel is multiplied by the feature information of the corresponding feature channel to obtain the output value of each feature channel.

4. The method of claim 1, wherein, The generation of dynamic convolution kernels based on input features from predetermined data includes: If the number of feature channels of the input feature is different from the number of predetermined output feature channels, then the size of the input feature is adjusted to the size of the predetermined output feature channels; Dynamic convolutional kernels are generated based on the size-adjusted input features.

5. The method of claim 4, wherein, The step involves weighting the feature information of each feature channel based on the dynamic convolution kernel corresponding to each feature channel to obtain the output value of each feature channel, including: If the number of feature channels of the input feature is less than the number of predetermined output feature channels, then based on the number of predetermined output feature channels, the existing feature channels and the added feature channels corresponding to the feature channels of the input feature are determined; Based on the dynamic convolution kernel corresponding to each feature channel in the existing feature channels, the feature information of the corresponding feature channels is weighted to obtain the first output value of each feature channel. The first output value of each feature channel is merged with the feature information of the added feature channel to obtain the second output value of each feature channel.

6. A data processing method, comprising: Receive pre-ordered data; The predetermined data is processed using a neural network model to obtain the processing result of the predetermined data; The neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the convolution calculation method according to any one of claims 1-5 on the predetermined data.

7. A convolution calculation apparatus for implementing the method as described in any one of claims 1-5, comprising: A kernel generation module is used to generate dynamic kernels based on input features of predetermined data, wherein each feature channel of the input features corresponds to a dynamic kernel; The channel information exchange module is used to exchange channel information between the feature channels of the input feature to obtain the feature information of each feature channel that has exchanged channel information. The weighted processing module is used to perform weighted processing on the feature information of the corresponding feature channels based on the dynamic convolution kernel corresponding to each feature channel, so as to obtain the output value of each feature channel.

8. A data processing apparatus, comprising: The receiving module is used to receive pre-defined data; A processing module is used to process predetermined data using a neural network model to obtain the processing result of the predetermined data; wherein the neural network model includes at least one convolutional layer, and during the processing of the neural network model, each convolutional layer is used to perform the convolution calculation method according to any one of claims 1-5 on the input features of the predetermined data.

9. A processing kernel for loading a neural network model to complete deep learning processing, wherein, The neural network model includes at least one convolutional layer, each of which is used to perform the convolution calculation method according to any one of claims 1-5 based on the input features of predetermined data.

10. An electronic device, comprising: Multiple processing cores; as well as The on-chip network is configured to interact with data between the multiple processing cores and external data; One or more processing cores store one or more instructions, and the one or more instructions are executed by one or more processing cores to enable one or more processing cores to perform the convolution calculation method of any one of claims 1-5, or the data processing method of claim 6.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processing kernel, it implements the convolution calculation method as described in any one of claims 1-5, or the data processing method as described in claim 6.

Images