Image data processing method and device

Abstract

The present application relates to an image data processing method and apparatus. The method comprises: obtaining a first image matrix A m×p And the second image matrix B p×n ; Determine the first image matrix A m×p Corresponding first feature map; Determine the second image matrix B p×n The corresponding X convolution kernels, each of which has a size of f×f×p, where f is an integer greater than 1, X is an integer greater than or equal to 1, and each convolution kernel contains the second image matrix B p×n One or more columns of elements in ; Convolve the first feature map with the X convolution kernels to obtain a second feature map, the second feature map corresponds to a third image matrix, and the third image matrix is ​​the first image matrix A m×p And the second image matrix B p×n The result of matrix multiplication is obtained. The solution provided in this application can improve the flexibility of the Transformer module.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image data processing method and device. Background Art

[0002] Current mainstream machine translation is primarily based on neural network machine translation, a system employing an encoder-decoder architecture. The encoder encodes the source language sequence and extracts information, which is then converted into the target language by the decoder, completing the translation process. The Deep Self-Attention Transformer model, designed based on this encoder-decoder architecture, has become a mainstream model in the field of machine translation due to its superior performance, significantly impacting the field of deep learning.

[0003] In a neural network with Transformer as the main module, there is a matrix multiplication operation between two two-dimensional data tensors. In some solutions, the Transformer model is generally deployed in a chip that supports matrix multiplication.

[0004] However, some chips on the market only support convolution calculations, so Transformer modules that require matrix multiplication operations cannot be deployed on these chips. This limits the use of Transformer modules and makes them less flexible. Summary of the Invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides an image data processing method that can improve the flexibility of the Transformer module.

[0006] The first aspect of the present application provides an image data processing method, comprising obtaining a first image matrix A m×p And the second image matrix B p×n ;

[0007] Determine the first image matrix A m×p the corresponding first feature map;

[0008] Determine the second image matrix B p×n The corresponding X convolution kernels, each of which has a size of f×f×p, where f is an integer greater than 1, X is an integer greater than or equal to 1, and each convolution kernel contains the second image matrix B p×n One or more columns of elements in ;

[0009] Convolve the first feature map with the X convolution kernels to obtain a second feature map, where the second feature map corresponds to a third image matrix, which is the first image matrix A.m×p And the second image matrix B p×n The result of matrix multiplication.

[0010] The second aspect of the present application provides an image data processing device, comprising

[0011] Acquisition module, used to obtain the first image matrix A m×p And the second image matrix B p×n ;

[0012] A first determining module, configured to determine the first image matrix A m×p The corresponding first feature map;

[0013] The second determining module is used to determine the second image matrix B p×n The corresponding X convolution kernels, each of which has a size of f×f×p, where f is an integer greater than 1, X is an integer greater than or equal to 1, and each convolution kernel contains the second image matrix B p×n One or more columns of elements in ;

[0014] A convolution module is used to convolve the first feature map with the X convolution kernels to obtain a second feature map, where the second feature map corresponds to a third image matrix, and the third image matrix is ​​the first image matrix A. m×p And the second image matrix B p×n The result of matrix multiplication.

[0015] A third aspect of the present application provides an electronic device, including:

[0016] processor; and

[0017] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method described above.

[0018] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method described above.

[0019] The technical solution of this application can obtain the first image matrix A m×p And the second image matrix B p×n , and determine the first image matrix A m×p The corresponding first feature map and the second image matrix B p×nCorresponding X convolution kernels of size f×f×p are then convolved with the first feature map and the X convolution kernels to obtain a feature map corresponding to the third image matrix, where f is an integer greater than 1, and the third image matrix is ​​obtained by matrix multiplication of the first image matrix and the second image matrix. In other words, this embodiment can use convolution operations to replace matrix multiplication operations to obtain the corresponding feature map, solving the problem that the Transformer module cannot be deployed on a chip that only supports convolution calculations, and improving the flexibility of the Transformer module.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0022] Figure 1 is a flowchart of an image data processing method shown in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a third feature map in the image data processing method shown in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a first feature map in the image data processing method shown in an embodiment of the present application;

[0025] Figure 4 is another schematic diagram of the first feature map in the image data processing method shown in an embodiment of the present application;

[0026] Figure 5 is another schematic diagram of the first feature map in the image data processing method shown in an embodiment of the present application;

[0027] Figure 6 Schematic diagram of X convolution kernels in the image data processing method shown in an embodiment of the present application;

[0028] Figure 7 is another schematic diagram of X convolution kernels in the image data processing method shown in an embodiment of the present application;

[0029] Figure 8 is a structural diagram of an image data processing device shown in an embodiment of the present application;

[0030] Figure 9 is another structural diagram of the image data processing device shown in an embodiment of the present application;

[0031] Figure 10 It is a structural diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although the accompanying drawings illustrate embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0033] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0035] To address the above problems, an embodiment of the present application provides an image data processing method that can improve the flexibility of the Transformer module.

[0036] To facilitate understanding, some terms involved in the embodiments of this application are introduced below.

[0037] Image Matrix: Digital image data can be represented using matrices, so matrix theory and matrix algorithms can be used to analyze and process digital images. Because digital images can be represented as matrices, two-dimensional arrays are often used to store image data in computer digital image processing programs.

[0038] Convolution kernel: When processing an image, a convolution kernel is a weighted average of pixels in a small area of ​​the input image, which becomes each corresponding pixel in the output image. The weight is defined by a function called a convolution kernel.

[0039] Feature Map: In the convolutional layers of a neural network, data is presented in three-dimensional form. Think of it as stacking multiple two-dimensional images, each of which is called a feature map. In the input layer, if the image is grayscale, there is only one feature map; if it is a color image, there are typically three feature maps (red, green, and blue). Between layers are several convolution kernels. Each feature map of the previous layer is convolved with each kernel to produce a feature map for the next layer.

[0040] Data module: In the image field, it is usually a three-dimensional array used to represent the pixel value of the image. The length represents the height of the image, the width represents the width of the image, and the depth represents the number of color channels of the image.

[0041] It should be noted that the image data processing method in this embodiment can be used for image processing of a neural network including a Transformer module, and can also be used for other neural networks that require matrix multiplication operations, and this embodiment does not limit this.

[0042] It should be noted that the image data processing device in this embodiment may include a Transformer module, or other modules that require matrix multiplication, which is not specifically limited in this embodiment.

[0043] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 It is a flowchart of the image data processing method shown in an embodiment of the present application.

[0045] See also Figure 1 , the image data processing method in this embodiment includes:

[0046] 101. An image data processing device obtains a first image matrix and a second image matrix;

[0047] The image data processing device obtains the first image matrix A that needs to be multiplied by the matrix m×p And the second image matrix B p×n . Among them, the first image matrix A m×p Or the second image matrix B p×n It can be the original image matrix or the image matrix obtained after matrix operation, which is not limited in this embodiment.

[0048] 102. The image data processing device determines a first feature map corresponding to the first image matrix;

[0049] The image data processing device obtains the first image matrix A m×p Then, the first image matrix A is generated according to the first preset rule m×pThe corresponding first feature map.

[0050] Specifically, the image data processing device can generate the first image matrix A in the following manner: m×p The corresponding first feature map:

[0051] S1. Determine the first image matrix A m×p The m row vectors correspond to the m data modules;

[0052] The image data processing device is for the first image matrix A m×p For each row vector of , a data module corresponding to the row vector is generated, where the length of the data module is 1, the width is 1, and the depth is p. The p elements contained in the data module correspond one-to-one to the p elements of the row vector. Specifically, the image data processing device can generate the data module corresponding to the row vector using a matrix transformation function, or can generate the data module corresponding to the row vector using other methods, which are not limited in this embodiment.

[0053] S2. Splicing the m data modules according to a preset splicing rule to obtain a third feature map;

[0054] After the image data processing device generates m data modules, it splices the m data modules according to a preset splicing rule to obtain a third feature map, where the length of the third feature map is a, the width is b, and the depth is p, and a×b=m.

[0055] Specifically, the splicing rule includes a splicing order, which can be from top to bottom and from left to right. That is, a data module is spliced ​​in order from top to bottom, the first column of the third feature map is arranged, and then the second column, the third column, ..., the bth column are arranged in order from top to bottom to obtain the third feature map. The splicing order can also be from top to bottom, from right to left; or from bottom to top, from left to right; or from bottom to top, from right to left; or other splicing orders, which are not limited in this embodiment.

[0056] Exemplarily, the first image matrix a=2,b=2,the splicing order is from top to bottom and from left to right. The image data processing device generates 4 row vectors [a 1，1 a 1，2 a 1，3 ],[a 2，1 a 2，2 a 2，3 ],[a 3，1 a 3，2 a 3，3 ],[a 4，1 a 4，2 a 4，3] corresponds to the four data modules M1, M2, M3 and M4, and then the image data processing device splices M2 below M1 in order from top to bottom to obtain the first column, then splices M3 to the left of M1, and then splices M4 below M3 to obtain the second column, as shown in FIG. Figure 2 shown.

[0057] S3. Insert a zero data module into the third feature map according to a preset zero-padding rule to obtain a first feature map.

[0058] It should be understood that in this embodiment, the length of the zero data module is 1, the width is 1, and the depth is n, and the values ​​of all elements in the zero data module are zero. p×n The number of column vectors and the position of the column vector in the convolution kernel are related.

[0059] As an optional method, the second image matrix B p×n Each corresponding convolution kernel contains the second image matrix B p×n , the corresponding image data processing device can insert zero data modules in the following manner: the image data processing device inserts zero data modules at the upper edge, lower edge, left edge and right edge of the third feature map to obtain the first feature map

[0060] Specifically, if each convolution kernel contains a column vector, f is an odd number, and the column vector is located at the center of the convolution kernel, the image data processing device can insert the upper edge and the lower edge of the third feature map. Row zero data module, inserted at the left and right edges of the third feature map The zero data module obtains the first feature map, that is, the front of the 1st to nth channels of the first feature map. Column, last column, front Row and last The values ​​of the rows are all zero. For example, f=3, the first image matrix The third feature map obtained by the image data processing device is as follows Figure 2 As shown, then insert one row of zero data modules at the upper edge and the lower edge of the third feature map, and insert one column of zero data modules at the left edge and the right edge of the third feature map to obtain the first feature map, as shown Figure 3 shown.

[0061] If each convolution kernel contains a column vector, f is an odd number, and the column vector is not located at the center of the convolution kernel, the image data processing device can insert the third feature map at the upper edge. Row zero data module, bottom edge inserted Row zero data module, inserted on the left edge Column zero data module, inserted at the right edge The column zero data module obtains the first feature map, and the values ​​of L1, L2, L3 and L4 are related to the position of the column vector in the convolution kernel. For example, f = 3, the column vector is located at the 1st row and 1st column of the convolution kernel, then the image data processing device inserts 1 row of zero data modules at the upper edge, 2 rows of zero data modules at the lower edge, 1 row of zero data modules at the left edge, and 2 rows of zero data modules at the right edge, as shown in FIG. Figure 4 shown.

[0062] As an optional method, the system sets the second image matrix B p×n Each corresponding convolution kernel contains the second image matrix B p×n If there are multiple column vectors in the convolution kernel, the corresponding image data processing device can insert a zero data module in the following manner: the image data processing device inserts a zero data module at the target position of the third feature map to obtain a first feature map, where the target position includes an internal position and an edge position, and the target position is related to the corresponding positions of the multiple column vectors in the convolution kernel.

[0063] For example, f=3, and each convolution kernel includes two column vectors W1 and W2, where W1 is located at the 2nd row and 2nd column of the convolution kernel, and W2 is located at the 2nd row and 3rd column of the convolution kernel. The image data processing device inserts one column of zero data modules between each column of the third feature map, and inserts one row of zero data modules at the upper edge and the lower edge of the third feature map, and one column of zero data modules at the left edge and the right edge to obtain the first feature map, as shown in FIG. Figure 5 shown.

[0064] It should be understood that the zero-padding rule is related to the number of column vectors contained in the convolution kernel and the position of each column vector in the convolution kernel. It can be set by the user based on the set number of column vectors and the position of the column vector in the convolution kernel. It can also be set in other ways, which is not limited in this embodiment.

[0065] 103. The image data processing device determines X convolution kernels corresponding to the second image matrix;

[0066] The image data processing device obtains the second image matrix B p×n Then, the second image matrix B is generated according to the second preset rule. p×n The corresponding X convolution kernels are of size f×f×p, each of which contains the second image matrix B p×n One or more column vectors in, that is, each convolution kernel contains the second image matrix B p×n One or more columns of elements in , f is an integer greater than 1, and X is an integer greater than or equal to 1.

[0067] Specifically, the image data processing device can determine the second image matrix B in the following manner:p×n Corresponding X convolution kernels: The image data processing device determines the values ​​of n groups of target elements in the X convolution kernels, and determines that the values ​​of other elements in the sum of the X convolution kernels except the n groups of target elements are 0, wherein each group of target elements corresponds to the second image matrix B p×n A column vector, the value of the target element of the i-th group is the same as the second image matrix B p×n The values ​​of the elements in the i-th column correspond one to one, i is an integer from 1 to n, and each convolution kernel contains one or more groups of target elements among these n groups of target elements.

[0068] It should be noted that the size of the convolution kernel is pre-set, that is, the value of f is a preset value. The system can also set the number of column vectors contained in each convolution kernel, that is, set the number of column vectors contained in each convolution kernel to be the second image matrix B. p×n Furthermore, the system can also set the position of the column vector contained in each convolution kernel in the convolution kernel.

[0069] In some embodiments, the system assumes that each convolution kernel contains a column vector, then X=n, and each convolution kernel contains a set of target elements. Specifically, for each convolution kernel, the target elements Z1, Z2, ..., Z p are located in the jth row and kth column of the 1st...,pth channel of the convolution kernel respectively. More specifically, the system assumes that the column vector contained in the convolution kernel is located at the center of the convolution kernel, and f is an odd number. Then the target elements Z1, Z2,..., Z contained in the convolution kernel are p Located in the 1st..., pth channel of the convolution kernel respectively Rank Column. For example, f = 3, the second image matrix The image data processing device determines the second image matrix B 3×3 Corresponding to 3 groups of target elements b in 3 convolution kernels 1，1 , b 2，1 , b 3，1 ; b 1，2 , b 2，2 , b 3，2 ; b 1，3 , b 2，3 , b 3，3 ; And determine that the values ​​of other elements in each convolution kernel except the target element are 0, where each convolution kernel contains 1 set of target elements, and the target elements contained in each convolution kernel are located in the 2nd row and 2nd column of the 1st, 2nd and 3rd channels of the convolution kernel respectively, such as Figure 6 shown.

[0070] In some embodiments, the system pre-sets that each convolution kernel includes u column vectors, where u is an integer greater than 1 and less than or equal to f.

[0071] If u is divisible by n, then X = n / u, and each convolution kernel contains u groups of target elements. Specifically, for each of the X convolution kernels, the convolution kernel contains the vth group of target elements located at the jth channel of the 1st, 2nd, ..., pth channel of the convolution kernel. v row k v Column, where v is an integer from 1 to u;

[0072] If u is not divisible by n, then X is the smallest integer greater than n / u. The 1st to X-1th convolution kernels in the X convolution kernels each contain u groups of target elements, and the Xth convolution kernel contains nu(X-1) groups of target elements. The 1st to X-1th convolution kernels contain the vth group of target elements in the u groups of target elements, which are located in the jth channel of the 1st, 2nd, ..., pth channel of the convolution kernel respectively. v row k v The rth target element in the nu(X-1)th target element contained in the Xth convolution kernel is located at the jth channel of the 1st, 2nd, ..., pth channel of the convolution kernel. r row k r Column, where r is an integer from 1 to nu(X-1).

[0073] For example, f=3, each convolution kernel contains two column vectors W1 and W2, W1 is located at the 2nd row and 2nd column of the convolution kernel, and W2 is located at the 2nd row and 3rd column of the convolution kernel.

[0074] Second image matrix The image data processing device determines the second image matrix B 3×3 Corresponding to 3 groups of target elements b in 3 convolution kernels 1，1 , b 2，1 , b 3，1 ; b 1，2 , b 2，2 , b 3，2 ; b 1，3 , b 2，3 , b 3，3 ; And determine that the values ​​of all elements except the target element in each convolution kernel are 0, where the first convolution kernel contains the second image matrix B 3×3 The first column element b in 1，1 , b 2，1 , b 3，1 and the second column element b 1，2 , b 2，2 , b 3，2 , the second convolution kernel contains the second image matrix B 3×3 The third column element b in 1，3 , b 2，3 , b 3，3 ,like Figure 7 shown.

[0075] 104. The image data processing device convolves the first feature map with X convolution kernels to obtain a second feature map.

[0076] After the image data processing device generates the first feature map and X convolution kernels, the first feature map is convolved with the X convolution kernels to obtain a second feature map. The second feature map is convolved with the third image matrix C m×n Correspondingly, the third image matrix C m×n That is, the first image matrix A m×p With the second image matrix B p×n The result of matrix multiplication.

[0077] In some embodiments, f is an odd number, each convolution kernel contains a column vector of the second image matrix, and the column vector is located at the center of the convolution kernel. Then, the image data processing device convolves the first feature map with n convolution kernels to obtain a second feature map with a length of a, a width of b, and a depth of n. Each data module of the second feature map in the depth direction of 1×1×n corresponds to the third image matrix C m×n The column vector of C, and the order of the column vectors in the second feature map corresponds to the order of splicing the third feature map, that is, the second feature map and C m×n The mapping method and the third feature map and A m×p The mapping is the same.

[0078] In some embodiments, each convolution kernel contains multiple column vectors of the second image matrix. After the image data processing device convolves the first feature map with n convolution kernels to obtain the second feature map, it rearranges the data modules in the second feature map according to a preset rearrangement rule to obtain a fourth feature map. The length of the fourth feature map is a, the width is b, and the depth is n. Each of the 1×1×n data in the depth direction of the fourth feature map corresponds to the third image matrix C m×n , and the distribution order of the column vectors in the fourth feature map corresponds to the splicing order of the third feature map.

[0079] In some embodiments, each convolution kernel includes a column vector of a second image matrix, and the column vector is not located at the center of the convolution kernel. Then, after the image data processing device convolves the first feature map with n convolution kernels to obtain the second feature map, it deletes the zero data module at the preset position to obtain the fifth feature map. The length of the fifth feature map is a, the width is b, and the depth is n. Each of the fifth feature maps with 1×1×n data in the depth direction corresponds to the third image matrix C. m×n , and the distribution order of the column vectors in the fifth feature map corresponds to the splicing order of the third feature map.

[0080] The technical solution of this application can obtain the first image matrix A m×pAnd the second image matrix B p×n , and determine the first image matrix A m×p The corresponding first feature map and the second image matrix B p×n Corresponding X convolution kernels of size f×f×p are then convolved with the first feature map and the X convolution kernels to obtain a feature map corresponding to the third image matrix, where f is an integer greater than 1, and the third image matrix is ​​obtained by matrix multiplication of the first image matrix and the second image matrix. In other words, this embodiment can use convolution operations to replace matrix multiplication operations to obtain the corresponding feature map, solving the problem that the Transformer module cannot be deployed on a chip that only supports convolution calculations, and improving the flexibility of the Transformer module.

[0081] Secondly, the embodiments of the present application provide multiple ways to determine the first feature map and multiple ways to determine the convolution kernel, which improves the flexibility of the solution.

[0082] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides an image data processing device, an electronic device and corresponding embodiments.

[0083] Figure 8 It is a structural diagram of an image data processing device shown in an embodiment of the present application.

[0084] See also Figure 8 , the image data processing device 800 in this embodiment includes:

[0085] Acquisition module 801, used to obtain the first image matrix A m×p And the second image matrix B p×n ;

[0086] The first determining module 802 is used to determine the first image matrix A m×p The corresponding first feature map;

[0087] The second determining module 803 is used to determine the second image matrix B p×n The corresponding X convolution kernels have a size of f×f×p, where f is an integer greater than 1 and X is an integer greater than or equal to 1. Each convolution kernel contains the second image matrix B p×n One or more columns of elements in ;

[0088] Convolution module 804 is used to convolve the first feature map with X convolution kernels to obtain a second feature map, the second feature map corresponds to the third image matrix, and the third image matrix is ​​the first image matrix A m×p And the second image matrix B p×n The result of matrix multiplication.

[0089] The technical solution acquisition module 801 of the present application can obtain the first image matrix A m×p And the second image matrix B p×n , the first determination module 802 can determine the first image matrix A m×p Corresponding to the first feature map, the second determination module 803 can determine the second image matrix B p×n Corresponding X convolution kernels of size f×f×p, the convolution module 804 can then convolve the first feature map with the X convolution kernels to obtain a feature map corresponding to the third image matrix, where f is an integer greater than 1, and the third image matrix is ​​obtained by matrix multiplication of the first image matrix and the second image matrix. In other words, this embodiment can use convolution operations to replace matrix multiplication operations to obtain the corresponding feature map, solving the problem that the Transformer module cannot be deployed on a chip that only supports convolution operations, thereby improving the flexibility of the Transformer module.

[0090] For ease of understanding, the image data processing device in this application is described in detail below. Figure 9 , the image data processing device 900 in this embodiment includes:

[0091] Acquisition module 901, used to obtain the first image matrix A m×p And the second image matrix B p×n ;

[0092] The first determining module 902 is used to determine the first image matrix A m×p The corresponding first feature map;

[0093] The second determining module 903 is used to determine the second image matrix B p×n The corresponding X convolution kernels have a size of f×f×p, where f is an integer greater than 1 and X is an integer greater than or equal to 1. Each convolution kernel contains the second image matrix B p×n One or more columns of elements in ;

[0094] Convolution module 904 is used to convolve the first feature map with X convolution kernels to obtain a second feature map, the second feature map corresponds to the third image matrix, and the third image matrix is ​​the first image matrix A m×p And the second image matrix B p×n The result of matrix multiplication;

[0095] The second determining module 903 includes:

[0096] A first determining unit 9031 is configured to determine values ​​of n groups of target elements in the X convolution kernels, where the value of the i-th group of target elements in the n groups of target elements corresponds to the value of the element in the i-th column of the second image matrix, where i is an integer from 1 to n, and each convolution kernel includes one or more groups of target elements in the n groups of target elements;

[0097] A second determining unit 9032 is configured to determine that the values ​​of the other elements in the X convolution kernels except the n groups of target elements are 0;

[0098] Optionally, each convolution kernel contains a set of target elements Z1, Z2, ..., Z among n sets of target elements. p , for each convolution kernel, the convolution kernel contains the target elements Z1, Z2, ..., Z p They are located in the jth row and kth column of the 1st, 2nd, ..., pth channels of the convolution kernel respectively;

[0099] The first determining module 902 includes:

[0100] The third determining unit 9021 is used to determine the first image matrix A m×p The m row vectors correspond to the m data modules, and the length of these m data modules is 1, the width is 1, and the depth is p;

[0101] The first splicing unit 9022 is configured to splice the m data modules according to a preset splicing rule to obtain a third feature map. The third feature map has a length a, a width b, and a depth p, and a×b=m:

[0102] The first zero-padding unit 9023 is used to insert zero data modules at the upper edge, lower edge, left edge and right edge of the third feature map to obtain the first feature map. The length of the zero data module is 1, the width is 1, and the depth is p. The values ​​of all elements in the zero data module are zero.

[0103] Optionally, each convolution kernel contains u groups of target elements in n groups of target elements, where u is an integer greater than 1 and less than or equal to f. For each convolution kernel, the p target elements in the v-th group of target elements contained in the convolution kernel are located at the j-th channel of the 1st, 2nd, ..., p-th channel of the convolution kernel, respectively. v row k v Column, where v is an integer from 1 to u;

[0104] The first determining module 902 includes:

[0105] The fourth determining unit 9024 is used to determine the first image matrix A m×p The m row vectors correspond to the m data modules, and the length of these m data modules is 1, the width is 1, and the depth is p;

[0106] The second splicing unit 9025 is used to splice the m data modules according to a preset splicing rule to obtain a third feature map, where the length of the third feature map is a, the width is b, and the depth is p, where a×b=m;

[0107] The second zero-padding unit 9026 is used to insert a zero data module at the target position of the third feature map to obtain the first feature map, where the target position includes an internal position and an edge position. The target position is related to the corresponding positions of multiple groups of target elements in the convolution kernel. The length of the zero data module is 1, the width is 1, and the depth is n. The values ​​of all elements in the zero data module are zero.

[0108] The technical solution acquisition module 901 of the present application can obtain the first image matrix A m×p And the second image matrix B p×n , the first determining module 902 can determine the first image matrix A m×p Corresponding to the first feature map, the second determination module 903 can determine the second image matrix B p×n Corresponding X convolution kernels of size f×f×p, the convolution module 904 can then convolve the first feature map with these X convolution kernels to obtain a feature map corresponding to the third image matrix, where f is an integer greater than 1, and the third image matrix is ​​obtained by matrix multiplication of the first image matrix and the second image matrix. In other words, this embodiment can use convolution operations to replace matrix multiplication operations to obtain the corresponding feature map, solving the problem that the Transformer module cannot be deployed on a chip that only supports convolution calculations, thereby improving the flexibility of the Transformer module.

[0109] Secondly, the embodiments of the present application provide a method for determining the first feature map and the convolution kernel in multiple ways, thereby improving the flexibility of the solution.

[0110] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0111] Figure 10 It is a structural diagram of an electronic device shown in an embodiment of the present application.

[0112] See also Figure 10 , the electronic device 1000 includes a memory 1010 and a processor 1020.

[0113] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0114] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. ROM may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage may be a readable and writable storage device. The permanent storage may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (e.g., a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (e.g., a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0115] The memory 1010 stores executable codes. When the executable codes are processed by the processor 1020 , the processor 1020 may execute part or all of the above-mentioned methods.

[0116] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0117] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0118] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing image data, characterized in that: include: Get the first image matrix and the second image matrix ; Determine the first image matrix the corresponding first feature map; Determine the second image matrix The corresponding X convolution kernels, the size of each convolution kernel is , wherein the is an integer greater than 1, X is an integer greater than or equal to 1, and each convolution kernel contains the second image matrix One or more columns of elements in ; Convolve the first feature map with the X convolution kernels to obtain a second feature map, where the second feature map corresponds to a third image matrix, which is the first image matrix. and the second image matrix The result of matrix multiplication; Wherein, the second image matrix is ​​determined The corresponding X convolution kernels include: Determine values ​​of n groups of target elements in the X convolution kernels, where the value of the i-th group of target elements in the n groups of target elements corresponds to the value of the element in the i-th column of the second image matrix, where i is an integer from 1 to n, and each convolution kernel includes one or more groups of target elements in the n groups of target elements; Determine that values ​​of other elements in the X convolution kernels except the n groups of target elements are 0.

2. The image data processing method according to claim 1, wherein: Each convolution kernel contains a set of target elements in the n sets of target elements , for each convolution kernel, the target element contained in the convolution kernel They are located in the jth row and kth column of the 1st, 2nd, ..., pth channels of the convolution kernel respectively; determining the first image matrix The corresponding first feature map includes: Determine the first image matrix m data modules corresponding to the m row vectors of , wherein the length of the data module is 1, the width is 1, and the depth is p; According to the preset splicing rules, the m data modules are spliced ​​to obtain a third feature map, wherein the length of the third feature map is a, the width is b, and the depth is p. ; A zero data module is inserted into the upper edge, lower edge, left edge and right edge of the third feature map to obtain a first feature map, where the length of the zero data module is 1, the width is 1, and the depth is p, and the values ​​of all elements in the zero data module are zero.

3. The image data processing method according to claim 1, wherein: Each convolution kernel contains u groups of target elements in the n groups of target elements, where u is an integer greater than 1 and less than or equal to f. For each convolution kernel, the p target elements in the vth group of target elements contained in the convolution kernel are respectively located at the 1st, 2nd..., and pth channels of the convolution kernel. Rank Column, where v is an integer from 1 to u; determining the first image matrix The corresponding first feature map includes: Determine the first image matrix m data modules corresponding to the m row vectors of , wherein the length of the data module is 1, the width is 1, and the depth is p; According to the preset splicing rules, the m data modules are spliced ​​to obtain a third feature map, wherein the length of the third feature map is a, the width is b, and the depth is p. ; A zero data module is inserted into the target position of the third feature map to obtain a first feature map, where the target position includes an internal position and an edge position, and the target position is related to the corresponding positions of the multiple groups of target elements in the convolution kernel. The length of the zero data module is 1, the width is 1, and the depth is n, and the values ​​of all elements in the zero data module are zero.

4. An image data processing device, characterized in that: include: Acquisition module, used to obtain the first image matrix and the second image matrix ; A first determining module, configured to determine the first image matrix the corresponding first feature map; A second determining module is used to determine the second image matrix The corresponding X convolution kernels, the size of each convolution kernel is , wherein the is an integer greater than 1, X is an integer greater than or equal to 1, and each convolution kernel contains the second image matrix One or more columns of elements in ; A convolution module is used to convolve the first feature map with the X convolution kernels to obtain a second feature map, where the second feature map corresponds to a third image matrix, and the third image matrix is ​​the first image matrix. and the second image matrix The result of matrix multiplication; The second determining module includes: a first determining unit, configured to determine values ​​of n groups of target elements in the X convolution kernels, where the value of the i-th group of target elements in the n groups of target elements corresponds to the value of the element in the i-th column of the second image matrix, where i is an integer from 1 to n, and each convolution kernel includes one or more groups of target elements in the n groups of target elements; The second determining unit is configured to determine that the values ​​of other elements in the X convolution kernels except the n groups of target elements are 0.

5. The image data processing device according to claim 4, characterized in that Each convolution kernel contains a set of target elements in the n sets of target elements , for each convolution kernel, the target element contained in the convolution kernel They are located in the jth row and kth column of the 1st, 2nd, ..., pth channels of the convolution kernel respectively; The first determining module includes: The third determining unit is used to determine the first image matrix m data modules corresponding to the m row vectors of , wherein the length of the data module is 1, the width is 1, and the depth is p; The first splicing unit is used to splice the m data modules according to a preset splicing rule to obtain a third feature map, wherein the length of the third feature map is a, the width is b, and the depth is p. ; The first zero-padding unit is used to insert zero data modules into the upper edge, lower edge, left edge and right edge of the third feature map to obtain a first feature map, where the length of the zero data module is 1, the width is 1, and the depth is p, and the values ​​of all elements in the zero data module are zero.

6. The image data processing device according to claim 4, wherein: Each convolution kernel contains u groups of target elements in the n groups of target elements, where u is an integer greater than 1 and less than or equal to f. For each convolution kernel, the p target elements in the vth group of target elements contained in the convolution kernel are respectively located at the 1st, 2nd..., and pth channels of the convolution kernel. Rank Column, where v is an integer from 1 to u; The first determining module includes: The fourth determining unit is used to determine the first image matrix m data modules corresponding to the m row vectors of , wherein the length of the data module is 1, the width is 1, and the depth is p; The second splicing unit is used to splice the m data modules according to a preset splicing rule to obtain a third feature map, wherein the length of the third feature map is a, the width is b, and the depth is p. ; A second zero-padding unit is used to insert a zero data module at a target position of the third feature map to obtain a first feature map, where the target position includes an internal position and an edge position, and the target position is related to the corresponding positions of the multiple groups of target elements in the convolution kernel. The length of the zero data module is 1, the width is 1, and the depth is n, and the values ​​of all elements in the zero data module are zero.

7. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having executable code stored thereon, wherein when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 3.

Images