Image convolution calculation method and device, computer program product and terminal equipment

Abstract

The invention belongs to the technical field of neural networks, and particularly relates to an image convolution calculation method and device, a computer program product and terminal equipment. The method comprises the following steps: quantizing a target initial weight of a target convolution kernel to obtain a target quantized weight; performing convolution kernel transformation on the target quantization weight by using a neural network processor to obtain a target transformation weight; and based on the target transformation weight, performing convolution calculation on the target image blocks by using a neural network processor to obtain a target convolution calculation result. According to the method, the quantization process of the weight can be set before the convolution kernel transformation, so that the convolution kernel transformation of the weight can be carried out under the condition that the precision of the weight is kept, and the transformation weight with relatively high precision is obtained; according to the transformation weight with high precision, accurate convolution calculation can be carried out on the image blocks of any initial pixel, and the availability of the Winograd algorithm can be improved.

Description

Technical Field

[0001] This application belongs to the field of neural network technology, and in particular relates to a method, apparatus, computer program product and terminal device for image convolution calculation. Background Technology

[0002] With the development of artificial intelligence technology, various neural network models are widely used. In neural networks, convolution is a common operation that can use convolution kernels to slide windows through an image, generating output data for one pixel at a time.

[0003] When performing convolution operations, the Winograd algorithm can be used to improve the computation speed. Specifically, when using the Winograd algorithm, the weights of the convolution kernel need to be transformed to obtain the transformed weights required by the Winograd algorithm. Then, the transformed weights can be quantized to obtain fixed-point quantized weights. Based on the quantized weights, convolution calculations can be performed on the image. In addition, when the image width is large, it is usually necessary to segment the image to obtain individual image blocks, and convolution calculations can be performed on each image block based on the quantized weights.

[0004] However, the process of quantizing the weights causes a loss of weight precision. The precision of quantized weights can only guarantee that image blocks starting with even-numbered pixels (i.e., pixels with even-numbered indices) can obtain relatively accurate convolution calculation results. If the image blocks start with odd-numbered pixels (i.e., pixels with odd-numbered indices), it is difficult to obtain high-precision convolution results. Therefore, when using the Winograd algorithm, it is necessary to restrict the starting pixels of the image blocks to even-numbered pixels, resulting in low usability of the Winograd algorithm. Summary of the Invention

[0005] In view of this, embodiments of this application provide an image convolution calculation method, apparatus, computer program product, and terminal device to solve the problem that the starting pixels of image blocks are limited in existing image convolution calculation methods, resulting in low usability of the Winograd algorithm.

[0006] The first aspect of this application provides a method for calculating the convolution of an image, which may include:

[0007] The initial weights of the target convolution kernel are quantized to obtain the target quantized weights; wherein, the target convolution kernel is a convolution kernel used to perform convolution calculation on the target image block, and the target image block is an image block of the target image;

[0008] The target quantization weights are transformed using a neural network processor through convolution kernel transformation to obtain the target transformed weights;

[0009] Based on the target transformation weights, the target image is divided into blocks by a neural network processor for convolution calculation to obtain the target convolution calculation result.

[0010] A second aspect of this application provides an image convolution calculation apparatus, which may include:

[0011] The quantization module is used to quantize the initial weights of the target convolution kernel to obtain the target quantized weights; wherein, the target convolution kernel is a convolution kernel used to perform convolution calculation on the target image block, and the target image block is an image block of the target image;

[0012] The transformation module is used to perform convolution kernel transformation on the target quantized weights using a neural network processor to obtain target transformed weights.

[0013] The calculation module is used to perform convolution calculations on the target image blocks based on the target transformation weights using a neural network processor to obtain the target convolution calculation results.

[0014] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described image convolution calculation methods.

[0015] A fourth aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the terminal device implements the steps of any of the above-described image convolution calculation methods.

[0016] A fifth aspect of this application provides a computer program product, including a computer program that, when run, causes the convolution calculation method for any of the above-described images to be executed.

[0017] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment quantizes the initial weights of the target convolution kernel to obtain target quantized weights; wherein, the target convolution kernel is a convolution kernel used for convolution calculation on target image blocks, and the target image block is an image block of the target image; the target quantized weights are transformed by the convolution kernel using a neural network processor to obtain target transformed weights; based on the target transformed weights, the target image block is convolved using a neural network processor to obtain the target convolution calculation result. In this application embodiment, the weight quantization process can be set before the convolution kernel transformation, so that the weight convolution kernel transformation can be performed while preserving the accuracy of the weights, resulting in highly accurate transformed weights; based on the highly accurate transformed weights, accurate convolution calculation can be performed on image blocks with arbitrary starting pixels, which helps to improve the usability of the Winograd algorithm. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the convolution calculation process;

[0020] Figure 2 This is a schematic diagram of the image;

[0021] Figure 3 This is a schematic diagram of the image convolution calculation process;

[0022] Figure 4 This is a flowchart of one embodiment of an image convolution calculation method according to the present application.

[0023] Figure 5 This is a structural diagram of one embodiment of an image convolution calculation device according to the present application.

[0024] Figure 6 This is a schematic block diagram of a terminal device in an embodiment of this application. Detailed Implementation

[0025] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0030] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] With the development of artificial intelligence technology, various neural network models are widely used in image processing scenarios such as object detection, image retrieval, and image segmentation. In neural networks, convolution is a common operation that uses a convolution kernel to slide a window across an image, generating output data for one pixel at a time. Figure 1Taking a single convolution calculation of the kernel shown as an example, the weights of a certain row in the kernel are g0, g1, and g2. In traditional convolution calculations, four pixels of the image (d0, d1, d2, and d3 in this case) can be used as input to obtain the 0th output data r0 and the 1st output data r1. Specifically, the calculation process for the 0th output data r0 and the 1st output data r1 is as follows:

[0032] r0 = d0·g0 + d1·g1 + d2·g2,

[0033] r1 = d1·g0 + d2·g1 + d3·g2.

[0034] To improve the speed of convolution operations, the Winograd algorithm can be used to replace the traditional convolution process. The Winograd algorithm reduces the number of multiplication operations during convolution, thus improving the speed. Specifically, when using the Winograd algorithm, the weights of the convolution kernel need to be transformed to obtain the transformed weights required by the Winograd algorithm. Then, the transformed weights can be quantized to obtain integer quantized weights. Based on the quantized weights, convolution calculations can be performed on the image.

[0035] Continue with Figure 1 Taking a single convolution calculation of the convolution kernel as an example, the weights of a certain row in the convolution kernel are g0, g1, and g2. When performing convolution calculation using the Winograd algorithm, four pixels of the image (d0, d1, d2, and d3 in this case) can be used as input to obtain the 0th output data r0 and the 1st output data r1. Specifically, the weights g0, g1, and g2 of the convolution kernel can first be transformed to obtain the transformed weights, which are g0, (g0+g1+g2) / 2, (g0-g1+g2) / 2, and g2, respectively. Then, each transformed weight can be quantized to generate the fixed-point numbers (here, pure integers with the decimal point after all digits) corresponding to g0, (g0+g1+g2) / 2, (g0-g1+g2) / 2, and g2, respectively, thus obtaining the quantized weights. Based on the quantization weights and input pixels, convolution calculations can be performed. In this example, the intermediate result of the convolution operation calculated using the transformation weights is as follows:

[0036] m1 = (d0 - d2)·g0,

[0037] m2=(d1+d2)·((g0+g1+g2) / 2),

[0038] m3=(d2-d1)·((g0-g1+g2) / 2),

[0039] m4 = (d1 - d3)·g2.

[0040] Based on the intermediate results m1, m2, m3, and m4, the output data r0 and output data r1 can be calculated:

[0041] r0 = m1 + m2 + m3

[0042] r1 = m2 - m3 - m4.

[0043] When an image is wide, it is usually divided into individual image tiles for better processing. During convolution calculations, each image tile can be calculated separately based on quantization weights. For example, if an image is 258 pixels wide, it can be divided into two image tiles for convolution processing. Specifically, pixels with indices 0-129 can be defined as one image tile, and pixels with indices 128-257 as the other image tile, and convolution calculations can be performed on these two image tiles separately.

[0044] In the process of quantifying the weights described above, the weights are rounded or truncated, which causes a loss of precision in the quantized weights. Considering the impact of the loss of precision in the weights on the calculation results, the quantization analysis process usually makes corresponding adaptation settings to ensure that relatively accurate calculation results can still be obtained even when there is a loss of precision in the quantized weights.

[0045] However, the quantization analysis process only adapts to the case where the starting pixel of the image block is an even number of pixels (i.e., a pixel with an even index). If the image block starts with an odd number of pixels (i.e., a pixel with an odd index), it is difficult to obtain a convolution calculation result with high accuracy. Therefore, when using the Winograd algorithm, it is necessary to restrict the index of the starting pixel of the image block to an even number to ensure that the convolution calculation result meets the accuracy requirements. However, this also leads to the low usability of the Winograd algorithm.

[0046] In view of this, embodiments of this application provide an image convolution calculation method, apparatus, computer program product, and terminal device to solve the problem that existing image convolution calculation methods have limitations on the starting pixels of image blocks, resulting in low usability of the Winograd algorithm.

[0047] It should be noted that the subject of execution of the method in this application is a terminal device. Specifically, the terminal device can be a desktop computer, workstation, laptop, handheld computer, mobile phone, server, or other computing device.

[0048] The image convolution calculation method provided in this application embodiment is an image convolution calculation method based on the Winograd algorithm. For ease of understanding, the calculation process of the Winograd algorithm will be introduced below with specific examples.

[0049] Figure 2 The diagram shows a 10*10 image. When performing convolution calculations on this image using the Winograd algorithm, the pixels are calculated in ascending order of height (vertical direction). Specifically, if the kernel height is 3 and the vertical stride is 1, the Winograd algorithm is first applied to pixels with heights of 0-2, resulting in output data row 0. Then, the Winograd algorithm is applied to pixels with heights of 1-3, resulting in output data row 1. This process continues until the final Winograd algorithm is applied to pixels with heights of 7-9, resulting in output data row 8.

[0050] When performing a Winograd convolution calculation on pixels at a certain height, the convolution calculation needs to be performed based on the weights of the convolution kernel corresponding to that height. Here, the height of the image can correspond to the weights of a certain row of the convolution kernel; for example, please refer to... Figure 3 When calculating output data row 0, each pixel with height 0 in the image corresponds to the weight of row 0 in the convolution kernel, specifically g0, g1, and g2. When performing convolution calculation on each pixel with height 0, the calculation can be based on g0, g1, and g2. Each pixel with height 1 in the image corresponds to the weight of row 1 in the convolution kernel, specifically g3, g4, and g5. When performing convolution calculation on each pixel with height 1, the calculation can be based on g3, g4, and g5.

[0051] When performing convolution calculations using the Winograd algorithm on pixels at a certain height, the convolution calculation can be decomposed into one or more cycles of calculation, depending on the actual number of pixels and the size of the convolution kernel. For example, the convolution calculation can usually be decomposed into two cycles of calculation.

[0052] For example, each row of the convolution kernel includes 3 weights, and 4 output results can be calculated in each computation cycle. If the image width is 10 (pixel indices from 0 to 9), convolution can be performed in 2 computation cycles. Specifically, in the first computation cycle, pixels with indices 0, 1, 2, and 3 in the image can be used as input to calculate two output results. Then, pixels with indices 4, 5, 6, and 7 in the image can be used as input to calculate another two output results, resulting in a total of 4 output results in the first computation cycle. In the second computation cycle, pixels with indices 2, 3, 4, and 5 in the image can be used as input to calculate two output results. Then, pixels with indices 6, 7, 8, and 9 in the image can be used as input to calculate another two output results, resulting in a total of 4 output results in the second computation cycle. After two computation cycles, a total of 8 output results can be obtained.

[0053] During the Winograd algorithm's computation, the weights of the convolution kernel transformation need to be quantized, converting them into fixed-point numbers (pure integers). This process typically involves rounding or truncation, causing quantization errors and resulting in a loss of precision. For example, if the convolution kernel weights are g0, g1, and g2, the Winograd algorithm transforms these weights to obtain g0, (g0+g1+g2) / 2, (g0-g1+g2) / 2, and g2. Then, quantization analysis is performed on the transformed weights. (g0+g1+g2) / 2 and (g0-g1+g2) / 2 will be quantized to pure integers, causing a loss of precision in (g0+g1+g2) / 2 and (g0-g1+g2) / 2. For image blocks starting with an odd number of pixels, the resulting convolution calculation cannot meet the corresponding precision requirements, leading to inaccurate convolution calculation results.

[0054] Therefore, the embodiments of this application can utilize a Neural Processing Unit (NPU) to perform the calculation of the Winograd algorithm. Since the calculation of the NPU can preserve the precision of the weights, the restriction that the starting pixel of the image block must be an even number of pixels can be removed, thereby improving the usability and scene adaptability of the Winograd algorithm.

[0055] Please see Figure 4 An embodiment of an image convolution calculation method in this application may include the following steps S401 to S403:

[0056] Step S401: Quantize the initial weights of the target convolution kernel to obtain the target quantized weights.

[0057] As a common operation, convolution can be used as a sub-process in most image processing applications. For example, in specific image processing applications such as object detection, image segmentation, and image retrieval, convolution can be used to efficiently extract image features (convolution output), and subsequent image processing operations can be performed based on these features.

[0058] In the embodiments of this application, an image that needs to be processed (denoted as the target image) can be obtained, and the target image can be processed. For example, in a target detection scenario (such as face detection), an image that needs to be detected can be obtained and used as the target image. In another example, in an image segmentation scenario, an image that needs to be segmented can be obtained and used as the target image. In yet another example, in an image retrieval scenario, an image that needs to be retrieved can be obtained and used as the target image.

[0059] Specifically, during the image processing of the target image, the Winograd algorithm can be used to perform convolution calculations on the target image.

[0060] It should be understood that before performing convolution calculations on the target image, necessary processing can be performed on the target image according to the specific application scenario. For example, in the application scenario of object detection, the target image can be scaled to adjust its resolution to match the requirements of the model input. In the application scenario of image segmentation, the target image can be enhanced by adjusting contrast, brightness, and sharpness to improve the distinction between the foreground and background, making the segmentation more accurate. In the application scenario of image retrieval, the target image can be preprocessed for feature extraction, such as performing histogram equalization and contrast enhancement to highlight the features of the target image.

[0061] In this embodiment, the weights of the convolution kernel can be quantized before using the Winograd algorithm for convolution calculation; then, the Winograd algorithm is used to transform the quantized weights into convolution kernels. This decouples the quantization process from the Winograd algorithm, allowing convolution transformation to be performed while maintaining weight accuracy, resulting in a more accurate convolution calculation.

[0062] In this embodiment, the convolution kernel used to perform convolution calculations on the target image can be referred to as the target convolution kernel, and the initial weights of the target convolution kernel can be referred to as the target initial weights; alternatively, the quantized target initial weights can be referred to as the target quantized weights. It should be understood that the target initial weights in this embodiment can be the weights of a certain row in the target convolution kernel, or the weights of each row in the target convolution kernel.

[0063] In this embodiment of the application, a toolchain can be used to quantize the initial weights of the target convolution kernel to obtain the target quantized weights.

[0064] For example, if the initial target weights are g0, g1, and g2, then g0, g1, and g2 can be directly quantized and converted into fixed-point numbers to obtain the target quantized weights. Here, fixed-point numbers specifically refer to pure integer fixed-point numbers, that is, the decimal point is placed after each number.

[0065] Step S402: Use a neural network processor to perform convolution kernel transformation on the target quantization weights to obtain the target transformation weights.

[0066] Before executing the Winograd algorithm, the target image can be segmented as needed to obtain various image blocks (denoted as target image blocks). If the width of the target image is greater than the image width threshold that the NPU can process, the target image can be segmented to obtain various target image blocks. The image width threshold that the NPU can process is the maximum image width that the NPU can process. The specific value of the image width threshold is related to the number of processing units actually included in the NPU, and this application embodiment does not limit it.

[0067] For each target image block, the Winograd algorithm can be used by the NPU to perform convolution calculations, obtaining the target convolution calculation results. Then, the target convolution calculation results of each target image block can be concatenated to obtain the convolution output result of the target image.

[0068] Since the accuracy of the weights is guaranteed, in this embodiment of the application, when segmenting the target image, there is no need to restrict the index of the starting pixel (denoted as the target starting pixel) of the target image block. That is, the index of the target starting pixel can be either even or odd.

[0069] Specifically, the starting pixels of each target can be determined based on the width of the target image; then, the target image can be segmented based on each starting pixel to obtain the corresponding target image blocks.

[0070] For example, for a target image with a width of 16, the target image can be divided into two target image blocks by taking any two pixels within the width range (0-15) as the target starting pixels. Here, the pixel with index 0 can be taken as the target starting pixel of the first target image block, and the pixel with index 7 can be taken as the target starting pixel of the second target image block, so as to obtain the first target image block with an index range of 0-8 and the second target image block with an index range of 7-15 respectively; or, the pixel with index 0 can be taken as the target starting pixel of the first target image block, and the pixel with index 4 can be taken as the target starting pixel of the second target image block, so as to obtain the first target image block with an index range of 0-5 and the second target image block with an index range of 4-15 respectively.

[0071] For example, for a target image with a width of 32, the target image can be divided into three target image blocks by using any three pixels within the width range (0-31) as the target starting pixels. Here, the pixel with index 0 can be used as the target starting pixel of the first target image block, the pixel with index 11 as the target starting pixel of the second target image block, and the pixel with index 25 as the target starting pixel of the third target image block, respectively cutting to obtain the first target image block with an index range of 0-12, the second target image block with an index range of 11-26, and the third target image block with an index range of 25-31; or, the pixel with index 0 can be used as the target starting pixel of the first target image block, the pixel with index 16 as the target starting pixel of the second target image block, and the pixel with index 27 as the target starting pixel of the third target image block, respectively cutting to obtain the first target image block with an index range of 0-17, the second target image block with an index range of 16-28, and the third target image block with an index range of 27-31.

[0072] In this embodiment of the application, the NPU can be used to perform convolution kernel transformation on the target quantization weights to obtain the target quantization weights after convolution kernel transformation. Here, the target quantization weights after convolution kernel transformation can be denoted as the target transformation weights.

[0073] In existing technologies, the convolution kernel transformation process is typically performed by the toolchain, while the NPU directly obtains the target quantization weights from memory (e.g., Double Data Rate SDRAM, DDR). In this embodiment, the convolution kernel transformation process is performed by the NPU, which obtains the target quantization weights from memory and performs the convolution kernel transformation. Since the number of target quantization weights is less than the number of target transformation weights, this embodiment reduces the number of parameters that the NPU obtains from memory, thereby reducing the memory bandwidth requirement.

[0074] For example, the number of target quantization weights is 3, while the number of target transformation weights is 4. In the prior art, the NPU needs to obtain 4 target transformation weights from memory. In this embodiment, the NPU needs to obtain 3 target quantization weights from memory. Compared with the prior art, the number of parameters obtained by the NPU from memory is reduced by (4-3) / 4 = 25%, which reduces the demand for memory bandwidth.

[0075] In one specific implementation of this application, the target quantized weight g can be transformed by a convolution kernel based on the convolution kernel transformation matrix G determined by the Winograd algorithm to obtain the target transformed weight g'; wherein, the convolution kernel transformation matrix is ​​a matrix that transforms the initial weights into the weights required by the Winograd algorithm, and the process of convolution kernel transformation is: g' = G·g.

[0076] For example, if the target quantization weights are g = [g0, g1, g2], then a convolution kernel transformation can be performed on the target quantization weights based on the convolution kernel transformation matrix G. Here, the convolution kernel transformation matrix determined based on the Winograd algorithm is as follows:

[0077]

[0078] Then, the target quantization weights can be transformed using the convolution kernel transformation matrix described above:

[0079]

[0080] Based on this, the target transformation weights g0, (g0+g1+g2) / 2, (g0-g1+g2) / 2, and g2 can be obtained.

[0081] During the convolution kernel transformation of the target quantized weights, it is necessary to ensure the precision of the weights, that is, the decimal part of the weights cannot be ignored. However, during the convolution kernel transformation, the target transformed weights may exceed the representation range of the original data format; therefore, in the embodiments of this application, a data format with a larger representation range can be used to represent the target transformed weights to avoid data overflow of the target transformed weights.

[0082] Specifically, the target quantization weights can be bit-width extended to obtain the bit-width extended target quantization weights; then, the NPU can be used to perform convolution kernel transformation on the bit-width extended target quantization weights to obtain the target transform weights; by extending the bit width of the target quantization weights, data overflow of the target transform weights can be prevented, and the complete target transform weight data can be preserved.

[0083] For example, if the original data format of the target quantization weight is an 8-bit signed number (int8), the bit width of the target quantization weight can be expanded to obtain a 10-bit target quantization weight. As a result, the subsequent target transformation weight is also a 10-bit signed number, and its decimal point can be located between bit0 and bit1 (that is, one bit in the target transformation weight is used to represent the decimal part), so that its decimal part can be completely preserved.

[0084] Step S403: Based on the target transformation weights, use a neural network processor to perform convolution calculations on the target image blocks to obtain the target convolution calculation results.

[0085] The following section will take any convolution calculation in the Winograd algorithm as an example to describe in detail the process of using a neural network processor to perform convolution calculations on blocks of the target image in the embodiments of this application.

[0086] In this embodiment of the application, the target pixel can be transformed by input to obtain the target transformed pixel required by the Winograd algorithm; wherein, the target pixel is the pixel in the target image block that is covered by the target transform weight.

[0087] For example, convolution calculations can be performed on d0, d1, d2, d3, d4, d5, d6, d7, d8, and d9 in the target image block, with target transformation weights g0, g1, and g2. At the beginning of the convolution calculation, d0, d1, d2, and d3 can be used as inputs. At this time, the pixels in the target image block covered by the target transformation weights are d0, d1, d2, and d3. Afterward, input transformations can be performed on d0, d1, d2, and d3 to obtain the target transformed pixels.

[0088] In one specific implementation of this application, the input transformation matrix B can be determined based on the Winograd algorithm. T The target pixel d is transformed to obtain the target transformed pixel d'; where the input transformation matrix is ​​the matrix that transforms the initial input into the input required by the Winograd algorithm, and the formula for the input transformation is as follows: d' = B T ·d.

[0089] For example, the target pixel d = [d0, d1, d2, d3] can be determined based on the input transformation matrix B.T The input transformations d0, d1, d2, and d3 are applied to obtain the target transformed pixel d'; here, the input transformation matrix B is determined based on the Winograd algorithm. T as follows:

[0090]

[0091] Then, the target transformed pixel d' can be calculated according to the following formula:

[0092]

[0093] Based on this, the target transformed pixels can be obtained as d0-d2, d1+d2, d2-d1, and d1-d3.

[0094] Then, based on the target transformation weights and the target transformation pixels, the first convolution calculation result corresponding to the target transformation pixels can be calculated.

[0095] Specifically, the target transformation weight g' can be multiplied by the target transformation pixel d' to obtain the transformation convolution calculation result M.

[0096] For example, the target transformation weights are The target transformed pixels are Then, the target transformation weight g' can be multiplied by the target transformation pixel d' to obtain the transformation convolution calculation result M:

[0097]

[0098] Then, the inverse transformation of the transformed convolution result M can be performed to obtain the first convolution result M'.

[0099] In one specific implementation of this application, the output transformation matrix A can be determined based on the Winograd algorithm. T The inverse transformation of the transformed convolution result M is performed to obtain the first convolution result M'; where the output transformation matrix is ​​to convert the transformed convolution result to the conventional one, M' = A. T ·M.

[0100] For example, the result of transform convolution is The output transformation matrix A is determined based on the Winograd algorithm. T for:

[0101]

[0102] Here, the output transformation matrix A can be... T Multiplying the result of the transform convolution M, we obtain the first convolution result M' of the target image block:

[0103]

[0104] The values ​​of r0 and r1 are as follows:

[0105] r0 = m1 + m2 + m3

[0106] r1 = m2 - m3 - m4.

[0107] Here, m1 is (d0-d2)·g0, and m2 is m3 is If m4 is (d1-d3)·g2, then we can obtain:

[0108] r0 = m1 + m2 + m3

[0109] r1 = m2 - m3 - m4.

[0110] The above r0 and r1 are the results of the first convolution calculation for the target image block.

[0111] In this embodiment of the application, if there are pixels in the target image block that have not been convolutionally calculated (denoted as pixels to be calculated), it indicates that the convolution calculation of the target image block has not yet ended. At this time, the target convolution kernel can be slid. After the target convolution kernel is slid, the pixels in the target image block covered by the target transform weight will also be updated accordingly. At this time, the step of performing input transformation on the target pixel to obtain the target transformed pixel and its subsequent steps can be returned to so that the target convolution kernel can calculate the pixels to be calculated.

[0112] Specifically, if there are still pixels to be calculated in a certain height (i.e., a certain row) of the target image block, the target convolution kernel can be slid horizontally to enter the next calculation cycle and calculate the pixels to be calculated in the horizontal direction.

[0113] In one specific implementation, since each convolution calculation result generates two pixels, the target convolution kernel can be slid two pixels horizontally in the direction of the pixel to be calculated (e.g., horizontally to the right) each time the target convolution kernel is slid.

[0114] After sliding the target convolution kernel, the convolution calculation can continue to be performed on the pixels to be calculated in the target image blocks by referring to the above method.

[0115] If there are no pixels to be calculated in a certain height (i.e., a certain row) of the target image block, it means that the convolution calculation of the target image block at a certain height has been completed. At this time, the convolution calculation of the next height (i.e., the next row) can be performed by referring to the above method.

[0116] It is important to note that when performing convolution calculations on pixels at a certain height, the weights of the corresponding row in the target convolution kernel must be used. For example, if there are no pixels with a height of 0 (i.e., row 0) to be calculated in the target image block, it means that the target image block has completed the convolution calculations on all pixels with a height of 0. In this case, convolution calculations can be performed on the next height after height 0 (i.e., height 1). When performing convolution calculations on pixels with a height of 0, the weights of row 0 in the target convolution kernel are specifically used. Therefore, when performing convolution calculations on pixels with a height of 1, the weights of the next row after row 0 in the target convolution kernel (i.e., row 1) must be used.

[0117] If there are no pixels to be calculated in the target image block, it indicates that the convolution calculation for the target image block has ended. At this time, the convolution calculation result of the target image block (denoted as the target convolution calculation result) can be determined based on the first convolution calculation results of each target image block. For example, if there are 4 first convolution calculation results, they can be concatenated in the order of the 0th first convolution calculation result, the 1st first convolution calculation result, the 2nd first convolution calculation result, and the 3rd first convolution calculation result to obtain the target convolution calculation result.

[0118] By sliding the target convolution kernel, convolution calculations can be performed on different pixels in the target image block, thereby efficiently completing the convolution operation on the target image block.

[0119] After obtaining the target convolution calculation results for each target image block, these results can be merged to obtain the convolution output result of the target image. Through multiple high-precision convolution calculations, a more accurate convolution output result can be obtained, laying the foundation for correct inference in neural networks (such as convolutional neural networks).

[0120] In one specific implementation, the convolutional calculation results of each target image can be concatenated to obtain the convolutional output of the target image. After obtaining the convolutional output of the target image, corresponding image processing can be performed on the target image based on the convolutional output to obtain the image processing result of the target image. For example, in the application scenario of object detection, the convolutional output of the target image can be input into a region proposal network to generate candidate regions; then, region pooling and fully connected layers can be used to refine the candidate regions, and finally output the category and specific location of the target contained in the target image (the image processing result of the target image in this scenario). As another example, in the application scenario of image segmentation, the convolutional output of the target image can be upsampled and skip connections can be applied to fuse features; based on the fused features, a classification probability map of each pixel in the target image can be generated; using a preset threshold, a segmentation mask can be determined from this classification probability map (the image processing result of the target image in this scenario). For example, in image retrieval applications, the convolutional output of the target image can be feature-encoded to obtain a fixed-length feature vector. Then, the similarity between this feature vector and the feature vectors corresponding to each image in the database can be calculated. Based on the similarity scores, the images in the database can be sorted, and the top N images with the highest similarity scores can be used as the retrieval results for the target image (the image processing result of the target image in this scenario).

[0121] In summary, this application embodiment quantizes the initial weights of the target convolution kernel to obtain target quantized weights. The target convolution kernel is a kernel used for convolution calculations on target image blocks, and the target image block is an image block of the target image. A neural network processor is used to transform the target quantized weights using the convolution kernel to obtain target transformed weights. Based on the target transformed weights, the neural network processor is used to perform convolution calculations on the target image block to obtain the target convolution calculation result. In this application embodiment, the weight quantization process can be set before the convolution kernel transformation, thereby allowing the weight convolution kernel transformation to be performed while preserving the accuracy of the weights, resulting in highly accurate transformed weights. Based on the highly accurate transformed weights, accurate convolution calculations can be performed on image blocks with arbitrary starting pixels, which helps improve the usability of the Winograd algorithm.

[0122] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0123] Corresponding to the image convolution calculation method described in the above embodiment, Figure 5This illustration shows a structural diagram of an embodiment of an image convolution calculation apparatus provided in this application.

[0124] In this embodiment of the application, an image convolution calculation device may include:

[0125] The quantization module 501 is used to quantize the target initial weights of the target convolution kernel to obtain the target quantized weights; wherein, the target convolution kernel is a convolution kernel used to perform convolution calculation on the target image block, and the target image block is an image block of the target image;

[0126] Transformation module 502 is used to perform convolution kernel transformation on the target quantized weights using a neural network processor to obtain target transformed weights;

[0127] The calculation module 503 is used to perform convolution calculation on the target image blocks using a neural network processor based on the target transformation weights to obtain the target convolution calculation result.

[0128] In one specific implementation of this application embodiment, the image convolution calculation device further includes:

[0129] The determining module is used to determine the starting pixel of each target based on the width of the target image; wherein the index of the starting pixel of the target is an odd number or an even number, and the starting pixel of the target is the starting pixel of the block of the target image;

[0130] The cutting module is used to cut the target image based on each of the target starting pixels to obtain the corresponding target image blocks.

[0131] In one specific implementation of this application embodiment, the transformation module includes:

[0132] The widening submodule is used to extend the bit width of the target quantization weight to obtain the target quantization weight after bit width extension.

[0133] The transformation submodule is used to perform convolution kernel transformation on the target quantized weights after bit-width expansion using a neural network processor to obtain the target transformed weights.

[0134] In one specific implementation of this application embodiment, the calculation module includes:

[0135] An input transformation submodule is used to perform input transformation on target pixels to obtain target transformed pixels; wherein, the target pixels are the pixels in the target image block that are covered by the target transformation weight;

[0136] The calculation submodule is used to obtain the first convolution calculation result corresponding to the target transformed pixel based on the target transformed weight and the target transformed pixel;

[0137] The result determination submodule is used to determine the target convolution calculation result based on the first convolution calculation results of each block of the target image.

[0138] In one specific implementation of this application embodiment, the calculation submodule includes:

[0139] The multiplication unit is used to multiply the target transformation weights with the target transformation pixels to obtain the transformation convolution calculation result;

[0140] The inverse transform unit is used to perform an inverse transform on the transformed convolution calculation result to obtain the first convolution calculation result.

[0141] In one specific implementation of this application embodiment, the calculation submodule includes:

[0142] The calculation result determination unit is used to determine the target convolution calculation result based on the first convolution calculation results of each of the target image blocks if there is no pixel to be calculated; wherein the pixel to be calculated is the pixel in the target image block that has not undergone convolution calculation;

[0143] The execution unit is returned to the target convolution kernel if the pixel to be calculated exists, and then returns to execute the step of performing input transformation on the target pixel to obtain the target transformed pixel and its subsequent steps, until the pixel to be calculated no longer exists.

[0144] In one specific implementation of this application embodiment, the image convolution calculation device further includes:

[0145] The merging module is used to stitch together the target convolution calculation results of each target image block to obtain the convolution output result of the target image;

[0146] The image processing module is used to perform image processing on the target image based on the convolution output result to obtain the image processing result of the target image. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus, modules, and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0149] like Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the above-described embodiments of the image convolution calculation methods, for example... Figure 4 Steps S401 to S403 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 503 are shown.

[0150] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the terminal device 6.

[0151] The terminal device 6 can be a desktop computer, laptop, handheld computer, smartphone, or smart TV, etc. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 6 may also include input / output devices, network access devices, buses, etc.

[0152] The processor 60 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 60 can be the nerve center and command center of the terminal device 6. The processor 60 can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0153] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0154] The terminal device 6 may further include a communication module, which can provide communication solutions for network devices, including Wireless Local Area Networks (WLANs) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have a single element or be an antenna array with multiple elements. The communication module can receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module can also receive signals to be transmitted from the processor, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0155] The terminal device 6 may further include a power management module, which can receive input from an external power source, battery, and / or charger to power the processor, the memory, and the communication module.

[0156] The terminal device 6 may further include a display module, which can be used to display information input by the user or information provided to the user. The display module may include a display panel, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Furthermore, a touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0158] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0164] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0165] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calculating image convolution, characterized in that, include: The initial weights of the target convolution kernel are quantized to obtain the target quantized weights; wherein, the target convolution kernel is a convolution kernel used to perform convolution calculation on the target image block, and the target image block is an image block of the target image; The target quantization weights are transformed using a neural network processor through convolution kernel transformation to obtain the target transformed weights; Based on the target transformation weights, the target image is divided into blocks by a neural network processor for convolution calculation to obtain the target convolution calculation result.

2. The image convolution calculation method according to claim 1, characterized in that, Before performing convolution calculations on the target image blocks using a neural network processor based on the target transformation weights to obtain the target convolution calculation result, the method further includes: Based on the width of the target image, the starting pixel of each target is determined; wherein, the index of the starting pixel of the target is an odd number or an even number, and the starting pixel of the target is the starting pixel of the target image block; Based on each of the target starting pixels, the target image is segmented to obtain the corresponding target image blocks.

3. The image convolution calculation method according to claim 2, characterized in that, The step of using a neural network processor to perform convolution kernel transformation on the target quantized weights to obtain target transformed weights includes: The target quantization weight is extended in bit width to obtain the target quantization weight after bit width extension; The target quantization weights after bit-width expansion are transformed by a convolution kernel using a neural network processor to obtain the target transformed weights.

4. The image convolution calculation method according to claim 1, characterized in that, The step of performing convolution calculations on the target image blocks using a neural network processor based on the target transformation weights to obtain the target convolution calculation results includes: The target pixel is subjected to an input transformation to obtain the target transformed pixel; wherein, the target pixel is the pixel in the target image block that is covered by the target transformation weight; Based on the target transformation weights and the target transformation pixels, the first convolution calculation result corresponding to the target transformation pixels is obtained; The target convolution calculation result is determined based on the first convolution calculation results of each of the target image blocks.

5. The image convolution calculation method according to claim 4, characterized in that, The step of obtaining the first convolution calculation result corresponding to the target transformed pixel based on the target transformed weight and the target transformed pixel includes: Multiply the target transformation weights by the target transformation pixels to obtain the transformation convolution calculation result; The inverse transformation of the transformed convolution calculation result is performed to obtain the first convolution calculation result.

6. The image convolution calculation method according to claim 4, characterized in that, The determination of the target convolution calculation result based on each of the first convolution calculation results of the target image blocks includes: If there are no pixels to be calculated, the target convolution calculation result is determined based on the first convolution calculation results of each of the target image blocks; wherein, the pixels to be calculated are the pixels in the target image blocks that have not undergone convolution calculation. If the pixel to be calculated exists, then slide the target convolution kernel and return to the step of performing input transformation on the target pixel to obtain the target transformed pixel and its subsequent steps, until the pixel to be calculated no longer exists.

7. The method for calculating the convolution of an image according to any one of claims 1 to 6, characterized in that, After performing convolution calculations on the target image blocks using a neural network processor based on the target transformation weights to obtain the target convolution calculation results, the method further includes: The convolution calculation results of each target image block are concatenated to obtain the convolution output result of the target image; Based on the convolution output, image processing is performed on the target image to obtain the image processing result of the target image.

8. An image convolution calculation device, characterized in that, include: The quantization module is used to quantize the initial weights of the target convolution kernel to obtain the target quantized weights; wherein, the target convolution kernel is a convolution kernel used to perform convolution calculation on the target image block, and the target image block is an image block of the target image; The transformation module is used to perform convolution kernel transformation on the target quantized weights using a neural network processor to obtain target transformed weights. The calculation module is used to perform convolution calculations on the target image blocks based on the target transformation weights using a neural network processor to obtain the target convolution calculation results.

9. A computer program product, characterized in that, Includes a computer program, which, when run, causes the convolution calculation method for the image as described in any one of claims 1 to 7 to be performed.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the terminal device to implement the steps of the image convolution calculation method as described in any one of claims 1 to 7.

Images