Data processing method, device, electronic device and computer readable storage medium

Abstract

A data processing method, device, electronic device and computer-readable storage medium. The data processing method includes: obtaining initial data for convolution operation, wherein the initial data includes an input matrix and a convolution kernel to be performed on the convolution operation, and initial convolution parameters set for the convolution operation; obtaining at least one constraint condition for constraining the convolution operation; judging whether the initial data satisfies at least one constraint condition, and if any constraint condition is not satisfied, performing a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies at least one constraint condition, wherein the transformation operation includes: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each transformed convolution kernel from the input matrix; performing a convolution operation based on the transformed data to obtain a calculation result.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] In the deep learning framework, operators (also called operations or functions) are the basic units for performing computing tasks. Writing operators means converting the computing tasks in the deep learning model into codes that can be executed in parallel on the GPU (graphics processing unit). Operator writing is a key technology in the field of deep learning framework and GPU computing.

[0003] Convolution operators originated from the field of signal processing and can be used to filter and extract features from signals in the time domain or spatial domain. Convolution operators can implement convolution operations between input matrices and convolution kernels. After the rise of deep learning, the emergence of convolutional neural networks (CNNs) has enabled convolution operators to achieve remarkable results in tasks such as image recognition, target detection, and semantic segmentation. Convolution operators themselves are also constantly innovating and optimizing to adapt to different tasks and data characteristics. Researchers have also explored a variety of variant convolution operators, such as deformable convolutions and dilated convolutions, to better capture the contextual information and spatial structure in images, and achieve diversity in convolution types and parameters. Summary of the invention

[0004] At least one embodiment of the present disclosure provides a data processing method, comprising: obtaining initial data for a convolution operation, wherein the initial data includes an input matrix and a convolution kernel for a convolution operation to be performed, and initial convolution parameters set for the convolution operation; obtaining at least one constraint condition for constraining the convolution operation; judging whether the initial data satisfies the at least one constraint condition, and if any of the constraints is not satisfied, performing a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies the at least one constraint condition, wherein the transformation operation comprises: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each transformed convolution kernel from the input matrix; and performing a convolution operation based on the transformed data to obtain a calculation result.

[0005] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, the at least one constraint condition includes at least one of a group number condition, a step size condition, a spacing condition and a size condition; the group number condition includes: the number of groups of the input matrix and the convolution kernel is not greater than a group number threshold; the step size condition includes: the step size of the convolution operation is not greater than a step size threshold; the spacing condition includes: in the case of a dilated convolution during the convolution operation, the spacing of the convolution kernel points in the dilated convolution is not greater than a spacing threshold; the size condition includes: the size of the convolution kernel is not greater than a size threshold, wherein the size of the convolution kernel includes the height and width of the convolution kernel.

[0006] For example, in the data processing method provided by at least one example of the above embodiments of the present disclosure, when the group number condition is not met, the convolution kernel includes M groups of convolution kernel data, and the input matrix includes M groups of matrix data, wherein the number of input channels of each group of the convolution kernel data and the number of input channels of each group of the matrix data are both 1 / M of the target input channel number, and M is an integer greater than 1;

[0007] Among them, the transforming of the convolution kernel to obtain at least one transform convolution kernel, and determining the transform matrix corresponding to each transform convolution kernel from the input matrix, includes: performing data expansion on each group of the convolution kernel data so that the number of input channels of each group of expanded convolution kernel data is consistent with the target input channel number, and combining multiple groups of expanded convolution kernel data in the output channel direction, and the combined convolution kernel is used as a transform convolution kernel; superimposing the M groups of matrix data along the input channel direction to obtain a transform matrix.

[0008] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, when any one of the step size condition, the spacing condition and the size condition is not met, the convolution kernel is transformed to obtain at least one transformed convolution kernel, and a transformation matrix corresponding to each transformed convolution kernel is determined from the input matrix, including: splitting the convolution kernel in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and splitting a sub-input matrix corresponding to each sub-convolution kernel from the input matrix, wherein each sub-convolution kernel serves as a transformed convolution kernel, and each sub-input matrix serves as a transformed matrix.

[0009] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, when the step size condition is not met, at least one of the width direction step size and the height direction step size in the initial convolution parameters is greater than the step size threshold; wherein the convolution kernel is split in the width direction and the height direction to obtain a plurality of sub-convolution kernels, and a sub-input matrix corresponding to each of the sub-convolution kernels is split from the input matrix, including: determining a first sub-width of each of the sub-convolution kernels based on the width direction step size and the width of the convolution kernel, and determining a first sub-height of each of the sub-convolution kernels based on the height direction step size and the height of the convolution kernel. height; based on the first sub-width and the first sub-height, determine the sub-convolution kernel corresponding to each element in the convolution kernel and the position in the corresponding sub-convolution kernel; based on the width direction step size, the width of the convolution kernel, the width of the input matrix and the padding width of the input matrix, determine the second sub-width of each of the sub-input matrices, and based on the height direction step size, the height of the convolution kernel, the height of the input matrix and the padding height of the input matrix, determine the second sub-height of each of the sub-input matrices; based on the second sub-width and the second sub-height, determine the sub-input matrix corresponding to each element in the input matrix and the position in the corresponding sub-input matrix.

[0010] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, when the size condition is not met, the convolution kernel is split in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix, including: dividing the convolution kernel in the width and height directions of the convolution kernel so that the width and height of each sub-convolution kernel obtained by the block meet the size condition, and the position of each element of the sub-convolution kernel in the convolution kernel is continuous; for each sub-convolution kernel, determining the traversal range of the sub-convolution kernel in the input matrix, and dividing the elements within the traversal range into sub-input matrices corresponding to the sub-convolution kernel.

[0011] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, when the spacing condition is not met, the convolution kernel is split in the width direction and the height direction of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix, including: taking each element in the convolution kernel as a sub-convolution kernel; for each sub-convolution kernel, determining the traversal range of the sub-convolution kernel in the input matrix, and dividing the elements in the traversal range into sub-input matrices corresponding to the sub-convolution kernel.

[0012] For example, in the data processing method provided by at least one example of the above embodiments of the present disclosure, the transformed data includes multiple groups of data, each group of data includes a sub-convolution kernel and a sub-input matrix corresponding to the sub-convolution kernel, and the convolution parameters of the convolution operation between each sub-convolution kernel and the corresponding sub-input matrix satisfy the at least one constraint condition;

[0013] Among them, a convolution operation is performed based on the multiple sub-convolution kernels and the sub-input matrix corresponding to each of the sub-convolution kernels to obtain a calculation result, including: for each group of data, using a sliding window to select elements from the upper left corner of the sub-input matrix and sliding through the sub-input matrix, calculating the multiplication and addition value of the sub-convolution kernel and the elements selected by the sliding window each time; accumulating the multiplication and addition values ​​at the same position of the multiple groups of data as the elements at the corresponding position of the output matrix.

[0014] For example, in the data processing method provided by at least one example of the above-mentioned embodiments of the present disclosure, determining whether the initial data satisfies the at least one constraint condition includes: determining whether the group number condition is satisfied; if the group number condition is not satisfied, performing a first transformation operation on the input matrix and the convolution kernel; if the group number condition is satisfied, determining whether the step size condition is satisfied; if the step size condition is not satisfied, performing a second transformation operation on the input matrix and the convolution kernel; if the step size condition is satisfied, determining whether the spacing condition is satisfied; if the spacing condition is not satisfied, performing a third transformation operation on the input matrix and the convolution kernel; if the spacing condition is satisfied, determining whether the size condition is satisfied; if the size condition is not satisfied, performing a fourth transformation operation on the input matrix and the convolution kernel; if the size condition is satisfied, obtaining the transformed data.

[0015] At least one embodiment of the present disclosure provides a data processing device, including: a data acquisition unit, a condition acquisition unit, a processing unit, and a computing unit.

[0016] A data acquisition unit, configured to acquire initial data for a convolution operation, wherein the initial data includes an input matrix and a convolution kernel for a to-be-performed convolution operation, and initial convolution parameters set for the convolution operation;

[0017] A condition acquisition unit, configured to acquire at least one constraint condition for constraining the convolution operation;

[0018] A processing unit, configured to determine whether the initial data satisfies the at least one constraint condition, and if any of the constraints are not satisfied, perform a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies the at least one constraint condition, wherein the transformation operation includes: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each of the transformed convolution kernels from the input matrix;

[0019] The operation unit is configured to perform a convolution operation based on the transformed data to obtain a calculation result.

[0020] At least one embodiment of the present disclosure provides an electronic device, comprising a processor; a memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the processor to implement the data processing method provided by any embodiment of the present disclosure.

[0021] At least one embodiment of the present disclosure provides a computer-readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the data processing method provided by any embodiment of the present disclosure can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0023] Figure 1 A flowchart of a data processing method provided by at least one embodiment of the present disclosure is shown;

[0024] Figure 2 A flowchart showing a processing procedure provided by at least one embodiment of the present disclosure is shown;

[0025] Figure 3 A schematic diagram of Group Convolution is shown;

[0026] Figure 4 A schematic diagram showing a transformation method provided by at least one embodiment of the present disclosure;

[0027] Figure 5 and 6 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure;

[0028] Figure 7 and 8 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure;

[0029] Fig. 9 and 10 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure;

[0030] Fig.11 A schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure is shown;

[0031] Fig.12 A schematic block diagram of a first electronic device provided for some embodiments of the present disclosure;

[0032] Fig.13 A schematic block diagram of a second electronic device provided for some embodiments of the present disclosure; and

[0033] Fig.14 A schematic diagram of a storage medium provided for some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0036] Convolution operations can be performed in the processor core. Before the convolution operation, it is necessary to obtain data such as the input matrix, convolution kernel, and convolution parameters. The physical memory can be set inside the processor core. During calculation, the hardware uses instructions to read the input matrix (activation) and weight (weight, also called convolution kernel) into different spaces of the physical memory in batches, and then obtains local results through instruction calculation, and saves the local results in batches to the video memory or registers.

[0037] On the one hand, the hardware has computing limitations, physical memory size limitations, instruction length and rule limitations. On the other hand, in order to improve memory utilization, some convolution operators have restrictions on computing data and convolution parameters. The computing data can be the input matrix and the convolution kernel. The convolution parameters can include the stride, the spacing of the convolution kernel points in the dilated convolution, the number of additional elements filled on the border of the input matrix, etc. The restrictions on computing data can include restrictions on the size of the input matrix and / or the size of the convolution kernel (kernel size), the number of groups, etc., and the restrictions on convolution parameters can include restrictions on stride, spacing (dilation), etc. For example, some convolution operators only support kernel size <= 7, stride = 1, dilation = 1.

[0038] In addition, the input data in different computing scenarios may have different shapes, that is, the size is not uniform. For each combination of input parameters, a specific operator needs to be written. The operator functions are repeated and complicated, and there is no general process to handle multiple situations.

[0039] At least one embodiment of the present disclosure provides a data processing method, a data processing device, an electronic device, and a computer-readable storage medium. The data processing method includes: obtaining initial data for a convolution operation, wherein the initial data includes an input matrix and a convolution kernel to be performed on the convolution operation, and an initial convolution parameter set for the convolution operation; obtaining at least one constraint condition for constraining the convolution operation; determining whether the initial data satisfies the at least one constraint condition, and if any of the constraints is not satisfied, performing a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies the at least one constraint condition, wherein the transformation operation includes: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each transformed convolution kernel from the input matrix; performing a convolution operation based on the transformed data to obtain a calculation result.

[0040] This data processing method can determine whether the initial data meets all the constraints, and transform the input matrix and convolution kernel if it does not meet the constraints, so that the transformed input matrix and convolution kernel can meet all the constraints. Based on this scheme, hardware limitations can be overcome, and the originally unrealizable convolution operator input conditions can be converted into a hardware acceptable range through corresponding transformation processing. The coverage of convolution input conditions is relatively comprehensive, and almost all conventional deep learning networks can be applied, realizing the general process of convolution operator calculation, without the need to implement operators for specific input combinations.

[0041] Figure 1 A flow chart of a data processing method provided by at least one embodiment of the present disclosure is shown.

[0042] like Figure 1 As shown, the method may include steps S110 to S140.

[0043] Step S110: Acquire initial data for convolution operation, wherein the initial data includes an input matrix and convolution kernel for the convolution operation to be performed, and initial convolution parameters set for the convolution operation.

[0044] Step S120: Obtain at least one constraint condition for constraining the convolution operation.

[0045] Step S130: Determine whether the initial data satisfies at least one constraint condition, and if any constraint condition is not satisfied, perform a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies at least one constraint condition. The transformation operation may include: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each transformed convolution kernel from the input matrix.

[0046] Step S140: Perform a convolution operation based on the transformed data to obtain a calculation result.

[0047] For example, in step S110, the input matrix and the convolution kernel, as well as the initial convolution parameters set for the convolution operation of the input matrix and the convolution kernel, can be obtained. The input matrix is ​​an input multidimensional array to be convolutionally calculated, the convolution kernel is used for sliding window feature extraction in the convolution calculation, the convolution kernel can be a multidimensional or one-dimensional array, and in some of the following embodiments, the convolution kernel is also referred to as a weight. The initial convolution parameters include, for example, the initial stride, the initial spacing (dilation) of the convolution kernel points in the dilated convolution, the number of additional elements (padding) filled on the boundary of the input matrix, the number of groups (group), etc. Among them, the stride includes the stride in the width direction (W direction) (abbreviated as SW) and the stride in the height direction (H direction) (abbreviated as SH), the spacing (dilation) includes the spacing in the width direction (abbreviated as DW) and the spacing in the height direction (abbreviated as DH), and the number of additional elements filled (padding) includes the number of padding in the width direction (abbreviated as PW) and the number of padding in the height direction (abbreviated as PH).

[0048] For example, in step S120, the at least one constraint condition is used to constrain the size of the input matrix and / or the size of the convolution kernel (kernel size), and can also be used to constrain the convolution parameters of the convolution operation.

[0049] For example, the at least one constraint condition includes at least one of a group number condition, a step length condition, a spacing condition, and a size condition. The group number condition includes: the group number of the input matrix and the convolution kernel is not greater than the group number threshold. The step length condition includes: the step length of the convolution operation is not greater than the step length threshold, wherein the step length may include a width direction step length and a height direction step length, and in the step length condition, both the width direction step length and the height direction step length are not greater than the step length threshold. The spacing condition includes: in the case of a dilated convolution during the convolution operation, the spacing of the convolution kernel points in the dilated convolution is not greater than the spacing threshold. The size condition includes: the size of the convolution kernel is not greater than the size threshold, wherein the size of the convolution kernel includes the height and width of the convolution kernel, and in the size condition, both the height and width of the convolution kernel are not greater than the size threshold. The height of the convolution kernel can be represented by the number of rows of multiple rows of elements arranged in the height direction of the convolution kernel, and the width of the convolution kernel can be represented by the number of columns of multiple columns of elements arranged in the width direction of the convolution kernel, and the same is true for the height and width of the input matrix.

[0050] For example, the at least one constraint condition includes: group=1, kernel size<=7, stride=1, dilation=1. In this example, the group number threshold is 1, the size threshold is 7, the stride threshold is 1, and the spacing threshold is 1. The present disclosure is not limited thereto, and in other embodiments, each threshold may adopt other values, which may be determined according to actual conditions.

[0051] For example, in step S130, when the constraint conditions include multiple items, it can be determined in sequence whether the multiple constraints are satisfied. For example, when a constraint condition is not satisfied, the first transformation operation is performed to obtain the convolution kernel after the first transformation and the input matrix after the first transformation, and it is determined whether the convolution operation between the convolution kernel after the first transformation and the input matrix after the first transformation satisfies the next constraint condition. If it is satisfied, the subsequent judgment is continued. If it is not satisfied, the second transformation operation is performed to obtain the convolution kernel after the second transformation and the input matrix after the second transformation, and it is determined whether the convolution operation between the convolution kernel after the second transformation and the input matrix after the second transformation satisfies the next constraint condition, and so on, until multiple constraints are satisfied.

[0052] For example, step S130 may include: determining whether the group number condition is met; if the group number condition is not met, performing a first transformation operation on the input matrix and the convolution kernel; if the group number condition is met, determining whether the step size condition is met; if the step size condition is not met, performing a second transformation operation on the input matrix and the convolution kernel; if the step size condition is met, determining whether the spacing condition is met; if the spacing condition is not met, performing a third transformation operation on the input matrix and the convolution kernel; if the spacing condition is met, determining whether the size condition is met; if the size condition is not met, performing a fourth transformation operation on the input matrix and the convolution kernel; if the size condition is met, obtaining the transformed data.

[0053] Figure 2 A flowchart of a processing procedure provided by at least one embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, get the tensor to be performed on the convolution operation: input matrix (activation) and weight (weight), and get the parameters of the convolution operation: kernel size ( Figure 2denoted by kernel in the text), stride, padding, dilation, group and other input parameters. You can first determine whether the group parameter is greater than 1 (group>1?). If so, perform group decomposition on the input matrix and weight to obtain a new input matrix (act1) and weight (wei1), and group becomes 1, where act1, wei1=splitG(activation, weight) means that act1 and wei1 are obtained by decomposing activation and weight; otherwise, the original input matrix and weight are used directly, that is, act1=activation, wei1=weight, indicating that act1 is the original input matrix activation, and wei1 is the original weight weight. Then, determine whether the stride parameter is greater than 1 (stride>1?). If so, perform stride split on the input matrix and weight to obtain a new input matrix (act2) and weight (wei2), with stride becoming 1 and padding becoming 0, where act2,wei2=splitC(act1, wei1) means that act2 and wei2 are obtained by decomposing act1 and wei1; otherwise, directly use the input matrix and weight obtained in the previous step, that is, act2=act1, wei2=wei1, indicating that act2 is the above act1 and wei2 is the above wei1.Then, determine whether the dilation parameter is greater than 1 (dilation>1?). If so, perform dilation splitting on the input matrix and weights to obtain new input matrices (act3) and weights (wei3), kernel size becomes 1, and padding becomes 0, where act3, wei3=splitD(act2, wei2) means act3 and wei3 are obtained by decomposing act2 and wei2, and ker1=1 means kernel size=1 after decomposition; otherwise, determine whether the kernel size parameter is greater than 7 (kernel>7?). If so, perform kernel splitting on the input matrix and weights to obtain new matrices, weights, and kernel size, and padding becomes 0, where act3, wei3, ker1=splitK(act2, wei2, kernel) means act3 and wei3 are obtained by decomposing act2 and wei2, and ker1 is the decomposition of the original kernel. size is obtained; otherwise, the input matrix and weights obtained in the previous step are directly used, where act3=act2, wei3=wei2, and ker1=kernel means that act3 is the above-mentioned act2, wei3 is the above-mentioned wei2, and ker1 is the above-mentioned kernel size. Among them, padding can be used in the subsequent splitting process of the input matrix. For example, the padding number PW of the input matrix in the width direction and the padding number PH in the height direction are used in the splitting process, which are not shown in the figure. After completing the transformation operation, the final transformed input matrix (act3) and weight (wei3) and the changed input parameters (ker1, 1, 0, 1, 1) are passed into the original convolution operator for calculation to obtain the final result. Various splitting methods will be described in detail in subsequent embodiments.

[0055] For example, in other embodiments, a judgment order different from that of the above embodiments may be adopted. For example, it may be determined whether the number of groups is satisfied first, then whether the size condition is satisfied, then whether the step condition is satisfied, and finally whether the spacing condition is satisfied.

[0056] According to the data processing method of the embodiment of the present disclosure, it is determined whether the initial data meets all the constraints, and if not, the input matrix and convolution kernel are transformed so that the transformed input matrix and convolution kernel can meet all the constraints. Based on this scheme, hardware limitations can be overcome, and the originally unrealizable convolution operator input conditions can be converted into a hardware acceptable range through corresponding transformation processing, and the coverage of convolution input conditions is relatively comprehensive, and almost all conventional deep learning networks can be applied, realizing the general process of convolution operator calculation, without the need to implement operators separately for specific input combinations.

[0057] For example, when the number of groups condition is not met, the convolution kernel includes M groups of convolution kernel data, and the input matrix includes M groups of matrix data, wherein the number of input channels of each group of convolution kernel data and the number of input channels of each group of matrix data are both 1 / M of the target input channel number, and M is an integer greater than 1. In step S130, the convolution kernel is transformed to obtain at least one transformation convolution kernel, and the transformation matrix corresponding to each transformation convolution kernel is determined from the input matrix, which may include: performing data expansion on each group of convolution kernel data so that the number of input channels of each group of expanded convolution kernel data is consistent with the target input channel number, and combining multiple groups of expanded convolution kernel data in the output channel direction, and the combined convolution kernel is used as a transformation convolution kernel; superimposing M groups of matrix data along the input channel direction to obtain a transformation matrix. For example, the expanded element value is 0.

[0058] Figure 3 A schematic diagram of Group Convolution is shown.

[0059] like Figure 3As shown, assume that the shape of the input matrix (activation) is [N, IC, H, W], and the shape of the weight (convolution kernel) is [OC, IC / G, KH, KW]. In the input matrix, N represents the number of images in a group, that is, the batch size, IC is the number of input channels, H is the height of the input matrix (such as the number of rows), and W is the width of the input matrix (such as the number of columns). In the convolution kernel, OC represents the number of output channels, IC is the same as above, KH is the height of the convolution kernel (such as the number of rows), and KW is the width of the convolution kernel (such as the number of columns). When performing grouped convolution, the input matrix is ​​cut into G parts on average in the IC direction (G is an integer greater than 1), and the size of each part in the IC direction is cin / G. The weight is cut into G parts on average in the OC direction, and the size of each part in the OC direction is cout / G. At this time, there are G matrices of shape [N, IC / G, H, W] and weights of shape [OC / G, IC / G, KH, KW] that are convolved to obtain G results of shape [N, OC / G, H, W]. Finally, they are concatenated in the OC / G dimension to obtain the final result (output), and the shape of the output matrix is ​​[N, OC, H, W]. For example, the shape of the input matrix is ​​[1,4,4,4], the number of input channels is 4, the data size of each channel is 4×4, and the shape of the weight is [4,1,1,1]. In order to implement the convolution operation, the input matrix is ​​split into 4 input matrices with a shape of [1,1,4,4], and the weight is split into 4 weight matrices with a shape of [1,1,1,1]. In this way, four groups of data are formed, each group of data includes an input matrix with a shape of [1,1,4,4] and a weight matrix with a shape of [1,1,1,1]. Each group of data performs a convolution operation to obtain a group of output matrices with a shape of [1,1,4,4]. The 4 groups of output matrices are spliced ​​in the output channel direction to obtain an output matrix with a shape of [1,4,4,4]. However, for the calculation method of group convolution, ordinary processor cores read data at a fixed size, and may read dirty data of other groups, resulting in calculation errors. To this end, the present disclosure provides the following embodiments.

[0060] Figure 4 A schematic diagram showing a transformation method provided by at least one embodiment of the present disclosure is shown.

[0061] like Figure 4As shown, in some embodiments of the present disclosure, when the input matrix (activation) and weight (weight) include multiple groups, the weight can be expanded, and the multiple groups of data of the original weight can be copied to the expanded weight, and the remaining data can be filled with 0. For example, the shape of each group of data of the original weight is [OC / G, IC / G, KH, KW], and the weight can be expanded to the shape [OC, IC, KH, KW], and the expanded element value is 0. The original position of the i-th group of data in the original weight is [OC / G·i : OC / G·(i+1), …], and the copied position is [OC / G·i : OC / G·(i+1), IC / G·i : IC / G·(i+1), …]. It can also be understood that for each set of data of the original weight, it is expanded in the IC direction (the up and down direction in the figure), and each set of data is expanded to [OC / G, IC, KH, KW], and then multiple sets of data are combined in the OC direction (the left and right direction in the figure) to expand to a new weight with a shape of [OC, IC, KH, KW]. For the input matrix, the shape of each set of data of the original input matrix is ​​[N, IC / G, H, W]. Combining multiple sets of data of the original input matrix in the IC direction can obtain an input matrix with a shape of [N, IC, H, W]. For example, the original weight includes 4 weight matrices with a shape of [1, 1, 1, 1]. Each set is expanded to [1, 4, 1, 1] in the input channel direction, and then the expanded data is combined in the output channel direction to obtain a transformation input matrix with a shape of [4, 4, 1, 1]. The original input matrix includes 4 input matrices of shape [1,1,4,4]. Multiple data are combined in the input channel direction to obtain a transformation weight matrix (i.e., transformation convolution kernel) of shape [1,4,4,4]. In this way, the number of groups of input matrix and expanded weight can be 1. Ordinary convolution is performed on the input matrix and expanded weight. Because the newly added data in the expanded weight is all 0, the value remains unchanged after the product accumulation, so it can be achieved. Figure 3 The algorithms shown are equivalent and dirty data will not be read by the hardware.

[0062] For example, when any one of the step size condition, spacing condition and size condition is not met, in step S130, the convolution kernel is transformed to obtain at least one transformed convolution kernel, and the transformation matrix corresponding to each transformed convolution kernel is determined from the input matrix, which may include: splitting the convolution kernel in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and splitting the sub-input matrix corresponding to each sub-convolution kernel from the input matrix, wherein each sub-convolution kernel serves as a transformed convolution kernel and each sub-input matrix serves as a transformed matrix.

[0063] For example, when the step size condition is not met, at least one of the width direction step size and the height direction step size in the initial convolution parameters is greater than the step size threshold. Wherein, the convolution kernel is split in the width direction and the height direction of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix, which may include: determining the first sub-width of each sub-convolution kernel based on the width direction step size and the width of the convolution kernel, and determining the first sub-height of each sub-convolution kernel based on the height direction step size and the height of the convolution kernel; determining the sub-convolution kernel corresponding to each element in the convolution kernel and the position in the corresponding sub-convolution kernel based on the first sub-width and the first sub-height; determining the second sub-width of each sub-input matrix based on the width direction step size, the width of the convolution kernel, the width of the input matrix and the padding width of the input matrix, and determining the second sub-height of each sub-input matrix based on the height direction step size, the height of the convolution kernel, the height of the input matrix and the padding height of the input matrix; determining the sub-input matrix corresponding to each element in the input matrix and the position in the corresponding sub-input matrix based on the second sub-width and the second sub-height.

[0064] Figure 5 and 6 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure is shown.

[0065] like Figure 5 As shown in the figure, the input matrix can be padded, and the padded element p can be 0. The weight (shape is [KH, KW]) slides on the padded matrix (shape is [H+2·PH, W+2·PW]) with stride as the step size, and the numbers in the overlapping window range are multiplied and summed. Among them, PH is the number of additional elements padded in the H direction of the input matrix (such as the number of padded rows), and PW is the number of additional elements padded in the W direction (such as the number of padded columns). Figure 5In the example shown, the width step is 3 and the height step is 2, that is, the convolution kernel (weight) first overlaps with the upper left corner of the input matrix (activation), and then moves in the width direction with three columns as the moving unit until the convolution kernel (weight) traverses the 1st to 3rd rows of the input matrix. The convolution kernel (weight) moves down two rows, that is, moves to the 3rd to 5th rows of the input matrix. In the 3rd to 5th rows, it traverses from the left to the right, and the moving unit is still three columns. And so on, until the input matrix is ​​traversed. It can be observed that during the movement of the convolution kernel, some elements in the convolution kernel appear repeatedly at the same position. For example, the positions where elements a, d, i and l appear overlap, the positions where elements b and j appear overlap, the positions where elements c and k appear overlap, the positions where elements e and r appear overlap, and the positions where elements f and g do not overlap with other elements. In this way, the convolution kernels can be grouped, and several elements that appear at overlapping positions can be grouped as a sub-convolution kernel. In order to distinguish, the overlapping elements are represented by the same grayscale, and different grayscales are used to represent different groups. Correspondingly, the input matrix can be grouped to obtain multiple sub-input matrices. The grouped situation is as follows Figure 6 As shown. In this way, when the corresponding sub-input matrix and sub-convolution kernel perform convolution operations, the step length is 1. The implementation process of splitting the convolution kernel and the input matrix can be shown in the following steps A1 to A4.

[0066] Step A1: Apply for the transformed matrix and weight memory, let TKH=(KH+SH-1) / SH, TKW=(KW+SW-1) / SW, the shape of the sub-convolution kernel is [OC, IC·SH·SW, TKH, TKW], let OH=(H+2·PH-KH) / SH+1, OW=(W+2·PW-KW) / SW+1, then the shape of the sub-input matrix is ​​[N, IC·SH·SW, OH+TKH-1, OW+TKW-1]. Among them, KH is the height of the original convolution kernel, KW is the width of the original convolution kernel, TKH is the height of the sub-convolution kernel, TKW is the width of the sub-convolution kernel, SH is the step size in the H direction, and SW is the step size in the W direction. PH is the padding of the input matrix in the H direction, and PW is the padding of the input matrix in the W direction.

[0067] Step A2: Copy each pixel point [n, c, h, w] in the original input matrix to the corresponding position of the sub-input matrix [n, c·SW·SH+(h+PH)%SH·SW+(w+PW)%SW, (h+PH) / SH, (w+PW) / SW], where the second element c·SW·SH+(h+PH)%SH·SW represents the c·SW·SH+(h+PH)%SH·SW sub-input matrix, and the third and fourth elements represent the coordinates in the sub-input matrix. Figure 3 The description of [N,IC,H,W] is mentioned. To distinguish it from [n,c,h,w], the following explanation is given: [N,IC,H,W] represents the shape of the input matrix, and [n,c,h,w] represents the coordinates of the pixels in the input matrix. For example, the input matrix may include a set of images, each image includes multiple input channels, and each channel includes a two-dimensional matrix. In [N,IC,H,W], N represents the number of images in a set, IC represents the number of input channels included in each image, and H and W represent the height and width of the two-dimensional matrix included in each channel, respectively. In [n,c,h,w], n represents the image where the pixel is located, c represents the channel where the pixel is located, h represents the number of rows where the pixel is located, and w represents the number of columns where the pixel is located. The pixel here refers to the pixel in the input matrix. The following [n,c,h,w] is the same.

[0068] Step A3: Copy each pixel point [oc, ic, kh, kw] in the original weight to the corresponding position [oc, ic·SW·SH+kh%SH·SW+kw%SW, kh / SH, kw / SW] of the sub-weight (i.e., sub-convolution kernel), where the second element ic·SW·SH+kh%SH·SW+kw%SW represents the ic·SW·SH+kh%SH·SW+kw%SW sub-weight, and the third and fourth elements represent the coordinates in the sub-weight. Figure 3 The description of [OC, IC / G, KH, KW] is mentioned. To distinguish it from [oc, ic, kh, kw], the explanation is as follows: [OC, IC / G, KH, KW] represents the shape of the weight, and [n, c, h, w] represents the coordinates of the pixel points in the weight. For example, in [OC, IC / G, KH, KW], IC represents the number of input channels, OC represents the number of output channels, and KH and KW represent the height and width of the weight, respectively. In [oc, ic, kh, kw], oc represents the output channel where the pixel point is located, ic represents the input channel where the pixel point is located, kh represents the number of rows where the pixel point is located, and kw represents the number of columns where the pixel point is located. The pixel point here refers to the pixel point in the weight. The same is true for [oc, ic, kh, kw] that appear below.

[0069] Step A4: Perform ordinary convolution with stride=1 and padding=0 on the transformed sub-input matrix and sub-weight to obtain the final result.

[0070] For example, the transformed data can be divided into multiple groups of data, each group of data includes a sub-convolution kernel and a sub-input matrix corresponding to the sub-convolution kernel, and the convolution parameters of the convolution operation of each sub-convolution kernel and the corresponding sub-input matrix meet the at least one constraint condition. Wherein, in step S140, the convolution operation is performed based on multiple sub-convolution kernels and the sub-input matrix corresponding to each sub-convolution kernel to obtain the calculation result, which may include: for each group of data, using a sliding window to select elements from the upper left corner of the sub-input matrix and sliding through the sub-input matrix, calculating the multiplication and addition value of the sub-convolution kernel and the elements selected by the sliding window each time; accumulating the multiplication and addition values ​​at the same position of multiple groups of data as the elements at the corresponding position of the output matrix.

[0071] For example, Figure 6 Taking the example shown as an example, the sub-input matrix and sub-weight of the same grayscale can be used as a group. In each group, the sub-weight traverses the sub-input matrix with stride=1 (the width step and the height step are both 1). The results of the corresponding steps in each group are accumulated, and the accumulated result can be used as an element value of the input matrix. For example, in the first group, the first step is to multiply and add the elements a, d, i, l in the sub-weight with the four upper left elements p, p, p, 9 in the corresponding sub-input matrix, and obtain the result N1 of the first step of the first group; in the second group, the first step is to multiply and add the elements b, j in the sub-weight with the two upper left elements p, p in the corresponding sub-input matrix, and obtain the result N2 of the first step of the second group; in the third group, the first step is to multiply and add the elements c, k in the sub-weight with the two upper left elements p, 8 in the corresponding sub-input matrix, and obtain the result N3 of the first step of the third group; and so on, the results N4, N5, and N6 of the first step of the fourth, fifth, and sixth groups can be obtained. Then, add the results of the first group of each group N1+ N2+ N3+ N4+N5+N6, and the result of the addition is the same as Figure 5 The first step of the convolution operation shown has the same result, which can be used as the value of the first element of the output matrix.

[0072] For example, based on the above embodiments, a convolution operation with a step size greater than 1 can be converted into a convolution operation with a step size of 1, which can overcome the hardware limitation on the step size.

[0073] For example, when the size condition is not met, the convolution kernel is split in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix. This may include: dividing the convolution kernel into blocks in the width and height directions of the convolution kernel so that the width and height of each sub-convolution kernel obtained by the blocks meet the size condition, and the positions of the elements of each sub-convolution kernel in the convolution kernel are continuous; for each sub-convolution kernel, determining the traversal range of the sub-convolution kernel in the input matrix, and dividing the elements within the traversal range into sub-input matrices corresponding to the sub-convolution kernels.

[0074] Figure 7 and 8 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure is shown.

[0075] like Figure 7 As shown in the figure, in this example, the width KW of the convolution kernel is 7, the height KH is 8, and the size constraint kernel size <= 7 is not satisfied. Therefore, the convolution kernel can be divided into blocks. For example, the elements in the first four rows and the first four columns can be divided into one block as a sub-convolution kernel; the elements in the first four rows and the last three columns can be divided into one block as another sub-convolution kernel; the elements in the last four rows and the first four columns can be divided into one block as another sub-convolution kernel; the elements in the last four rows and the last three columns can be divided into one block as another sub-convolution kernel. The situation after grouping is as follows Figure 8 As shown, in order to distinguish, different sub-convolution kernels are represented by different grayscales. For each sub-convolution kernel, find the traversal range of the elements in the sub-convolution kernel in the original input matrix, and use the elements in the traversal range as a corresponding sub-input matrix. For example, taking the first sub-convolution kernel as an example, in the width direction, the elements 1 to 25 of the sub-convolution kernel can be traversed from the first column of the original input matrix to the column where element 6 is located in the input matrix (i.e., the 7th column), and in the height direction, the elements of the sub-convolution kernel can be traversed from the first row of the original input matrix to the row where element 22 is located in the input matrix (i.e., the 5th row). Therefore, the elements in the 1st to 7th columns and the 1st to 5th rows in the original input matrix can be segmented out as the sub-input matrix corresponding to the first sub-convolution kernel. Based on a similar method, the sub-input matrix corresponding to each sub-convolution kernel can be obtained.

[0076] For example, when KH or KW is greater than 7, the weight needs to be split. After the weight is split into GH·GW parts, analyze the sliding range of each part in the original input matrix, and split the input matrix into GH·GW parts according to its range, as shown above. Figure 8As shown in the figure, the sub-input matrices and sub-weights of the same color are convoluted and added separately, and the effect is equivalent to the convolution of the original input matrix and the original weight. Finally, the split data are superimposed on their respective channels, and the resulting new matrices and new weights are subjected to ordinary convolution to obtain the final result. The implementation process of splitting the convolution kernel and the input matrix can be shown in the following steps B1 to B6.

[0077] Step B1: Determine the splitting parameters GH and GW, which are incremented from 1 until KH / GH<7 and KW / GW<7. In this way, the sub-weights obtained by splitting can meet the size constraint and minimize the number of weight divisions.

[0078] Step B2: Apply for transformed matrix and weight memory, let TKH=(KH+GH-1) / GH, TKW=(KW+GW-1) / GW, the shape of sub-weight is [OC, IC·GH·GW, TKH, TKW], let OH=(H+PHT+PHB-KH) / SH+1, OW=(W+PWL+PWR-KW) / SW+1, then the shape of sub-input matrix is ​​[N, IC·GH·GW, OH+TKH-1, OW+TKW-1]. Among them, PHT represents the padding amount at the top of the input matrix, Figure 7 PHT=1; PHB represents the padding amount at the bottom of the input matrix. Figure 7 PHB=2; PWL indicates the padding amount on the left side of the input matrix. Figure 7 PWL=1; PWR indicates the padding amount of the input matrix on the right side. Figure 7 Medium PWR=2.

[0079] Step B3: Copy each pixel point [oc, ic, kh, kw] in the original weight to [oc,ic·GH·GW + kh / TKH·GW + kw / TKW, kh%TKH, kw%TKW] of the new weight.

[0080] Step B4: Determine the upper, lower, left and right boundaries of each matrix slice (i.e., sub-input matrix). When gh is traversed at [0, GH] and gw is traversed at [0, GW], the boundaries of the [gh, gw]th slice [top, bottom, left, right] = [gh·TKH, gh·TKH+TH, gw·TKW, gw·TKW+TW]. Among them, TH represents the height of the sub-input matrix after splitting, and TW represents the width of the sub-input matrix after splitting. GH and GW are mentioned in the above embodiments. In order to distinguish them from gh and gw, it is explained as follows: GH and GW represent the number of parts that the weight is split in the height direction and the width direction, respectively. The number of parts that the input matrix is ​​split is the same as the number of parts that the weight is split. Therefore, GH and GW also represent the number of parts that the input matrix is ​​split in the height direction and the width direction. [gh, gw] represents the coordinates of the matrix slice in multiple slices, which is used to indicate which slice of the multiple slices the slice is, wherein gh and gw represent the positions of the matrix slice in the height direction and the width direction respectively, which is used to indicate which slice the slice is in the height direction and which slice the slice is in the width direction.

[0081] Step B5: traverse all slices and copy the data within the slice range to the new matrix respectively. When gh is traversed in [0, GH] and gw is traversed in [0, GW], for each pixel point [oc, ic, kh, kw] in the weight, if [kh+PHT, kw+PWL] is within the slice boundary [top, bottom, left, right], copy the data to [oc, ic·GH·GW+gh·GW+gw, kh+ph-top, kw+pw-left]. Where PHT represents the padding amount of the input matrix at the top, and PWL represents the padding amount of the input matrix on the left.

[0082] Step B6: Convolve the transformed matrix and weights with padding=0 to obtain the final result.

[0083] For example, when the spacing condition is not met, the convolution kernel is split in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix. This may include: taking each element in the convolution kernel as a sub-convolution kernel; for each sub-convolution kernel, determining the traversal range of the sub-convolution kernel in the input matrix, and dividing the elements within the traversal range into sub-input matrices corresponding to the sub-convolution kernels.

[0084] Fig. 9 and 10 A schematic diagram showing another transformation method provided by at least one embodiment of the present disclosure is shown.

[0085] like Fig. 9As shown in the figure, in this example, the dilation of the dilated convolution is 2. A new weight is constructed, and each pixel in the original weight is split into a sub-weight, and each sub-weight has only 1x1 pixels. For example, if the weight width and height are both 3, it can be split into 9 sub-weights (weight[0,0]~weight[2,2]). A new matrix is ​​constructed, and for each sub-weight, the pixels that need to be multiplied in the matrix corresponding to the dilated convolution are copied to the new matrix without holes in the middle. The situation after grouping is as follows Fig.10 As shown, for each sub-weight, find the traversal range of each sub-weight in the original input matrix, and use the elements in the traversal range as a corresponding sub-input matrix. For example, taking the first sub-weight weight[0,0] as an example, in the width direction, the sub-weight can be traversed from the first column of the original input matrix to the column where element 5 is located in the input matrix (i.e., the 6th column), and in the height direction, the elements of the sub-weight can be traversed from the first row of the original input matrix to the row where element 22 is located in the input matrix (i.e., the 5th row). Therefore, the elements in the 1st to 6th columns and the 1st to 5th rows in the original input matrix can be split out as the sub-input matrix corresponding to the first sub-weight. Based on a similar method, the sub-input matrix corresponding to each sub-weight can be obtained. At this time, the new matrix (i.e., the sub-input matrix) and the new weight (i.e., the sub-weight) are subjected to ordinary convolution calculation with dilation=1, and the result is consistent with the original algorithm. The implementation process of splitting the convolution kernel and the input matrix can be shown in the following steps C1 to C4.

[0086] Step C1: Apply for transformed matrix and weight memory, the weight shape is [OC, IC·KH·KW, 1, 1], let KRH=(KH-1)·(DH-1)+KH, KRW=(KW-1)·(DW-1)+KW, then the matrix shape is [N, IC·KH·KW, H-KRH+2·PH+1, W-KRW+2·PW+1]. DH is the spacing of convolution points in the H direction, and DW is the spacing of convolution points in the W direction.

[0087] Step C2: Copy each pixel point [oc, ic, kh, kw] in the original weight to the corresponding position of the new weight [oc, ic·KH·KW+kh·KW+kw, 0, 0].

[0088] Step C3: Repeat the above steps B4 and B5, where TKH=1, TKW=1, GH=DH, GW=DW.

[0089] Step C4: Perform ordinary convolution with dilation=1 and padding=0 on the transformed matrix and weights to obtain the final result.

[0090] For example, based on the above embodiments, the dilated convolution can be converted into a normal convolution with a spacing of 1, which can overcome the hardware limitations on the dilated convolution.

[0091] According to at least one embodiment of the present disclosure, hardware limitations are overcome, and the originally unfeasible convolution operator input conditions can be converted into a range acceptable to the hardware through a corresponding splitting method.

[0092] According to at least one embodiment of the present disclosure, the coverage of convolution parameter input situations is relatively comprehensive, and almost all conventional deep learning networks are applicable.

[0093] According to at least one embodiment of the present disclosure, a general process for calculating a convolution operator is implemented without having to implement operators separately for specific input parameter combinations.

[0094] According to at least one embodiment of the present disclosure, this process provides strong scalability for operator modification at the algorithm level.

[0095] According to at least one embodiment of the present disclosure, the process can be extended to 3D convolution.

[0096] Fig.11 A schematic block diagram of a data processing device 800 provided by at least one embodiment of the present disclosure is shown.

[0097] For example, Fig.11 As shown, the data processing device 800 includes a data acquisition unit 810, a condition acquisition unit 820, a processing unit 830 and an operation unit 840. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, these modules can be implemented by hardware (such as circuit) modules, software modules, or any combination of the two. The following embodiments are the same and will not be repeated here. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions. It should be noted that Fig.11 The components and structure of the data processing device 800 shown are merely exemplary and non-limiting. The data processing device 800 may also have other components and structures as required.

[0098] The data acquisition unit 810 is configured to acquire initial data for a convolution operation, wherein the initial data includes an input matrix and a convolution kernel for a convolution operation to be performed, and initial convolution parameters set for the convolution operation; the data acquisition unit 810 may, for example, execute Figure 1 Step S110 described.

[0099] The condition acquisition unit 820 is configured to acquire at least one constraint condition for constraining the convolution operation; the condition acquisition unit 820 may, for example, execute Figure 1 Step S120 described.

[0100] The processing unit 830 is configured to determine whether the initial data satisfies the at least one constraint condition, and if any of the constraints are not satisfied, perform a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies the at least one constraint condition, wherein the transformation operation includes: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each of the transformed convolution kernels from the input matrix; the processing unit 830 may, for example, perform Figure 1 Step S130 described.

[0101] The operation unit 840 is configured to perform a convolution operation based on the transformed data to obtain a calculation result. The operation unit 840 may, for example, perform Figure 1 Step S140 described.

[0102] For example, the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830 and the operation unit 840 can be hardware, software, firmware and any feasible combination thereof. For example, the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830 and the operation unit 840 can be dedicated or general circuits, chips or devices, etc., or a combination of a processor and a memory. The embodiments of the present disclosure do not limit the specific implementation forms of the above-mentioned units.

[0103] For example, the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830, and the operation unit 840 may include codes and programs stored in a memory; the processor may execute the codes and programs to implement some or all functions of the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830, and the operation unit 840 as described above. For example, the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830, and the operation unit 840 may be dedicated hardware devices, used to implement some or all functions of the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830, and the operation unit 840 as described above. For example, the data acquisition unit 810, the condition acquisition unit 820, the processing unit 830, and the operation unit 840 may be a circuit board or a combination of multiple circuit boards, used to implement the functions as described above. In the embodiment of the present disclosure, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware stored in the memory that can be executed by the processor.

[0104] It should be noted that in the embodiment of the present disclosure, each unit of the data processing device 800 corresponds to each step of the aforementioned data processing method. For the specific functions of the data processing device 800, please refer to the relevant description of the data processing method, which will not be repeated here. Fig.11 The components and structures of the data processing device 800 shown are only exemplary and non-restrictive. The data processing device 800 may also include other components and structures as needed. The data processing device 800 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or can be composed in other applicable ways.

[0105] At least one embodiment of the present disclosure further provides an electronic device, the electronic device comprising a processor and a memory, wherein the memory stores one or more computer program modules. The one or more computer program modules are configured to be executed by the processor to implement the above data processing method.

[0106] Fig.12 A schematic block diagram of a first electronic device provided in some embodiments of the present disclosure. Fig.12 As shown, the first electronic device 900 includes a processor 910 and a memory 920. The memory 920 stores non-temporary computer-readable instructions (e.g., one or more computer program modules). The processor 910 is used to run non-temporary computer-readable instructions, and the non-temporary computer-readable instructions are executed by the processor 910 to execute one or more steps in the data processing method described above. The memory 920 and the processor 910 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown). For the specific implementation of each step of the data processing method and the related explanation content, please refer to the above-mentioned embodiment of the data processing method, and the repetitions will not be repeated here.

[0107] It should be noted that Fig.12 The components of the first electronic device 900 shown are merely exemplary and non-limiting. The first electronic device 900 may also have other components according to actual application requirements.

[0108] For example, the processor 910 and the memory 920 may communicate with each other directly or indirectly.

[0109] For example, the processor 910 and the memory 920 may communicate via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 910 and the memory 920 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0110] For example, the processor 910 and the memory 920 may be arranged on a server side (or a cloud side).

[0111] For example, the processor 910 can control other components in the first electronic device 900 to perform desired functions. For example, the processor 910 can be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) can be an X86 or ARM architecture, etc. The processor 910 can be a general-purpose processor or a dedicated processor, and can control other components in the first electronic device 900 to perform desired functions.

[0112] For example, the memory 920 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 910 may run one or more computer program modules to implement various functions of the first electronic device 900. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0113] For example, in some embodiments, the first electronic device 900 may be a mobile phone, a tablet computer, an electronic paper, a television, a monitor, a laptop computer, a digital photo frame, a navigator, a wearable electronic device, a smart home device, etc.

[0114] For example, the first electronic device 900 may include a display panel, and the display panel may be used to segment an image, etc. For example, the display panel may be a rectangular panel, a circular panel, an elliptical panel, or a polygonal panel, etc. In addition, the display panel may be not only a flat panel, but also a curved panel, or even a spherical panel.

[0115] For example, the first electronic device 900 may have a touch function, that is, the first electronic device 900 may be a touch device.

[0116] It should be noted that, in the embodiments of the present disclosure, the specific functions and technical effects of the first electronic device 900 can refer to the above description of the data processing method, which will not be repeated here.

[0117] Fig.13 This is a schematic block diagram of a second electronic device provided in some embodiments of the present disclosure. The second electronic device 1000 is suitable for implementing the data processing method provided in the embodiments of the present disclosure. The second electronic device 1000 may be a terminal device, etc. It should be noted that: Fig.13 The second electronic device 1000 shown is merely an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Fig.13 As shown, the second electronic device 1000 may include a processing device 1010, which is, for example, a central processing unit, a graphics processing unit, etc., which can perform various appropriate actions and processes according to the program stored in the ROM 1020 or the program loaded from the storage device 1080 to the RAM 1030, wherein ROM represents a read-only memory and RAM represents a random access memory. In the RAM 1030, various programs and data required for the operation of the second electronic device 1000 are also stored. The processing device 1010, the ROM 1020 and the RAM 1030 are connected to each other via a bus 1040. The I / O interface 1050 is also connected to the bus 1040, wherein I / O represents input / output.

[0119] Typically, the following devices may be connected to the I / O interface 1050: an input device 1060 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1070 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1080 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1090. The communication device 1090 may allow the second electronic device 1000 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.13 The second electronic device 1000 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the second electronic device 1000 may alternatively implement or possess more or fewer devices.

[0120] For example, according to an embodiment of the present disclosure, the above-mentioned data processing method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned data processing method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1090, or installed from the storage device 1080, or installed from the ROM 1020. When the computer program is executed by the processing device 1010, the functions defined in the data processing method provided in the embodiment of the present disclosure can be implemented.

[0121] At least one embodiment of the present disclosure further provides a computer-readable storage medium, which stores non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a computer, the above-mentioned data processing method can be implemented. Using the computer-readable storage medium, the tediousness of users preparing video clips of different image styles can be reduced, and the switching between video clips of different image styles can be made smoother.

[0122] Fig.14 A schematic diagram of a storage medium provided in some embodiments of the present disclosure. Fig.14 As shown, the storage medium 1100 stores non-transitory computer-readable instructions 1110. For example, when the non-transitory computer-readable instructions 1110 are executed by a computer, one or more steps in the data processing method described above are performed.

[0123] For example, the storage medium 1100 may be applied to the first electronic device 900. Fig.12 The memory 920 in the first electronic device 900 is shown. For example, the relevant description of the storage medium 1100 can refer to Fig.12 The corresponding description of the memory 920 in the first electronic device 900 is shown and will not be repeated here.

[0124] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0125] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0126] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0127] There are a few points to note about this disclosure:

[0128] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure. Other structures may refer to the general design.

[0129] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0130] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Acquire initial data for a convolution operation, wherein the initial data includes an input matrix and a convolution kernel for a convolution operation to be performed, and initial convolution parameters set for the convolution operation; Obtaining at least one constraint condition for constraining the convolution operation; Determine whether the initial data satisfies the at least one constraint condition, and if any of the constraints is not satisfied, perform a transformation operation on the input matrix and the convolution kernel until the transformed data satisfies the at least one constraint condition, wherein the transformation operation includes: transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each of the transformed convolution kernels from the input matrix; Performing a convolution operation based on the transformed data to obtain a calculation result; Wherein, the at least one constraint condition includes at least one of a group number condition, a step length condition and a spacing condition; The group number condition includes: the group number of the input matrix and the convolution kernel is not greater than the group number threshold; The step size condition includes: the step size of the convolution operation is not greater than a step size threshold; The spacing condition includes: in the case of a dilated convolution during the convolution operation, the spacing of convolution kernel points in the dilated convolution is not greater than a spacing threshold.

2. The data processing method according to claim 1, wherein: The at least one constraint condition further comprises a size condition; The size condition includes: the size of the convolution kernel is not greater than a size threshold, wherein the size of the convolution kernel includes the height and width of the convolution kernel.

3. The data processing method according to claim 2, wherein: When the group number condition is not met, the convolution kernel includes M groups of convolution kernel data, and the input matrix includes M groups of matrix data, wherein the number of input channels of each group of the convolution kernel data and the number of input channels of each group of the matrix data are both 1 / M of the target input channel number, and M is an integer greater than 1; The step of transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each of the transformed convolution kernels from the input matrix, comprises: Performing data expansion on each group of the convolution kernel data so that the number of input channels of each group of the expanded convolution kernel data is consistent with the target number of input channels, and combining multiple groups of the expanded convolution kernel data in the output channel direction, and the combined convolution kernel is used as a transformation convolution kernel; The M groups of matrix data are superimposed along the input channel direction to obtain the transformation matrix.

4. The data processing method according to claim 2, wherein: In the case that any one of the step size condition, the spacing condition and the size condition is not satisfied, transforming the convolution kernel to obtain at least one transformed convolution kernel, and determining a transformation matrix corresponding to each of the transformed convolution kernels from the input matrix, comprises: The convolution kernel is split in the width and height directions of the convolution kernel to obtain multiple sub-convolution kernels, and a sub-input matrix corresponding to each sub-convolution kernel is split from the input matrix, wherein each sub-convolution kernel serves as a transformation convolution kernel, and each sub-input matrix serves as a transformation matrix.

5. The data processing method according to claim 4, wherein: When the step size condition is not met, at least one of the width direction step size and the height direction step size in the initial convolution parameters is greater than the step size threshold; The convolution kernel is split in the width direction and the height direction of the convolution kernel to obtain a plurality of sub-convolution kernels, and a sub-input matrix corresponding to each of the sub-convolution kernels is split from the input matrix, including: Determine a first sub-width of each of the sub-convolution kernels based on the width direction step size and the width of the convolution kernel, and determine a first sub-height of each of the sub-convolution kernels based on the height direction step size and the height of the convolution kernel; Based on the first sub-width and the first sub-height, determine the sub-convolution kernel corresponding to each element in the convolution kernel and the position in the corresponding sub-convolution kernel; Determine the second sub-width of each of the sub-input matrices based on the width direction step size, the width of the convolution kernel, the width of the input matrix, and the padding width of the input matrix, and determine the second sub-height of each of the sub-input matrices based on the height direction step size, the height of the convolution kernel, the height of the input matrix, and the padding height of the input matrix; Based on the second sub-width and the second sub-height, the sub-input matrix corresponding to each element in the input matrix and the position in the corresponding sub-input matrix are determined.

6. The data processing method according to claim 4, wherein: When the size condition is not met, the convolution kernel is split in the width direction and the height direction of the convolution kernel to obtain a plurality of sub-convolution kernels, and a sub-input matrix corresponding to each of the sub-convolution kernels is split from the input matrix, including: Divide the convolution kernel into blocks in the width direction and the height direction of the convolution kernel, so that the width and height of each sub-convolution kernel obtained by the block satisfy the size condition, and the position of each element of the sub-convolution kernel in the convolution kernel is continuous; For each of the sub-convolution kernels, determine the traversal range of the sub-convolution kernel in the input matrix, and divide the elements in the traversal range into sub-input matrices corresponding to the sub-convolution kernel.

7. The data processing method according to claim 4, wherein: When the spacing condition is not met, the convolution kernel is split in the width direction and the height direction of the convolution kernel to obtain a plurality of sub-convolution kernels, and a sub-input matrix corresponding to each of the sub-convolution kernels is split from the input matrix, including: Taking each element in the convolution kernel as a sub-convolution kernel; For each of the sub-convolution kernels, determine the traversal range of the sub-convolution kernel in the input matrix, and divide the elements in the traversal range into sub-input matrices corresponding to the sub-convolution kernel.

8. The data processing method according to claim 4, wherein: The transformed data includes multiple groups of data, each group of data includes a sub-convolution kernel and a sub-input matrix corresponding to the sub-convolution kernel, and a convolution parameter of a convolution operation between each sub-convolution kernel and a corresponding sub-input matrix satisfies the at least one constraint condition; The convolution operation is performed based on the multiple sub-convolution kernels and the sub-input matrix corresponding to each of the sub-convolution kernels to obtain a calculation result, including: For each group of data, use a sliding window to select elements from the upper left corner of the sub-input matrix and slide through the sub-input matrix to calculate the multiplication and addition value of the sub-convolution kernel and the elements selected by the sliding window each time; The multiplication and addition values ​​at the same position of the plurality of groups of data are accumulated as the elements at the corresponding positions of the output matrix.

9. The data processing method according to claim 2, wherein: Determining whether the initial data satisfies the at least one constraint condition includes: Determine whether the group number condition is met; If the group number condition is not met, performing a first transformation operation on the input matrix and the convolution kernel; If the group number condition is met, then determine whether the step length condition is met; If the step size condition is not met, performing a second transform operation on the input matrix and the convolution kernel; If the step size condition is met, determining whether the spacing condition is met; If the spacing condition is not met, performing a third transformation operation on the input matrix and the convolution kernel; If the spacing condition is met, determining whether the size condition is met; If the size condition is not met, performing a fourth transform operation on the input matrix and the convolution kernel; If the size condition is met, the transformed data is obtained.

10. An electronic device, comprising: processor; A memory storing one or more computer program modules; The one or more computer program modules are configured to be executed by the processor to implement the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing non-transitory computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the data processing method according to any one of claims 1 to 8 is implemented.

Images