Image processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides an image processing method, an image processing device, an electronic device, a storage medium and a program product, which can be used in the field of artificial intelligence, in particular in the field of image processing and the like. The method comprises: obtaining an input image, the input image comprising a plurality of rows of pixels; for each row of pixels in the plurality of rows of pixels of the input image, performing pixel extraction in parallel using a plurality of dedicated processing units to obtain row data for each row of pixels; and splicing the row data for each row of pixels to obtain an output image. By using the above method, various scale input image format processing can be flexibly supported, the parallelism characteristics of hardware can be fully utilized, pixel extraction can be efficiently performed, efficient weight gradient calculation can be supported, the throughput of the overall calculation process can be improved, and therefore the user experience can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the computer technology, and more particularly, to an image processing method, an image processing apparatus, an electronic device, a computer readable storage medium and a computer program product, which can be used in the field of artificial intelligence, in particular, the field of image processing, etc. BACKGROUND

[0002] Nowadays, deep learning has been widely applied. With the diversification of models and the increase of demand for computing power, considering the performance, cost, etc. of general-purpose processors, graphics processing units, etc. in the original deep learning hardware platform, the industry has begun to develop deep learning accelerators.

[0003] The convolution layer is a very common layer in the neural network. In the forward calculation of the convolution layer, a plurality of convolution kernels need to be convolved on the input image. A common implementation method is to perform im2col operation on the input image, so that the convolution operation can be converted into matrix multiplication. In the training process, backward calculation is also needed, and the weight gradient and the input image gradient of the current layer are calculated according to the output image gradient of the current layer, and are sequentially propagated to the front layer. The solving method of the weight gradient is the output image gradient multiplied by the input image after the im2col operation. In order to obtain high throughput in hardware calculation, the dimension needs to be arranged into continuous data and output to the matrix operation unit, which is equivalent to performing im2col operation and transpose operation.

[0004] However, the pixel extraction technology in the prior art is low in efficiency, and it is difficult to meet the needs of users. SUMMARY

[0005] According to an embodiment of the present disclosure, an image processing method, an image processing apparatus, an electronic device, a computer readable storage medium and a computer program product are provided.

[0006] In a first aspect of the present disclosure, an image processing method is provided, comprising: obtaining an input image, the input image comprising a plurality of rows of pixels; for each row of pixels in the plurality of rows of pixels of the input image, performing pixel extraction in parallel using a plurality of special-purpose processing units to obtain row data for each row of pixels; and splicing the row data for each row of pixels to obtain an output image.

[0007] In a second aspect of the present disclosure, an image processing apparatus is provided, comprising: an obtaining module configured to obtain an input image, the input image comprising a plurality of rows of pixels; a first calculation module configured to, for each row of pixels in the plurality of rows of pixels of the input image, perform pixel extraction in parallel using a plurality of special-purpose processing units to obtain row data for each row of pixels; and a first splicing module configured to splice the row data for each row of pixels to obtain an output image.

[0008] In a third aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the method according to the first aspect of the present disclosure.

[0009] In a fourth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to implement the method according to the first aspect of the present disclosure.

[0010] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, and the computer program, when executed by a processor, performs the method according to the first aspect of the present disclosure.

[0011] With the technology according to the present application, a method suitable for input image data processing in CHW format convolution layer weight gradient calculation is provided, and the technical scheme using the method can flexibly support format processing of input images of various scales, can fully utilize the parallelism characteristics of hardware, efficiently perform pixel extraction, so as to support efficient weight gradient calculation, can improve the throughput of the overall calculation process, and therefore can improve the user experience.

[0012] It should be understood that the content described in the summary section is not intended to limit the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to the like elements throughout the drawings, and in which:

[0014] Figure 1 A schematic block diagram of an image processing environment 100 in which an image processing method in certain embodiments of the present disclosure can be implemented is shown;

[0015] Figure 2 A schematic block diagram of an image processing environment 200 in which an image processing method in certain embodiments of the present disclosure can be implemented is shown;

[0016] Figure 3 A flowchart of an image processing method 300 according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic diagram of an input image 400 is shown, according to an embodiment of the disclosure;

[0018] Figure 5 A flowchart of an image processing method 500 is shown, according to an embodiment of the disclosure;

[0019] Figures 6A to 6D A schematic diagram of input data changes 610 to 640 in an image processing method is shown, according to an embodiment of the disclosure;

[0020] Figure 7 A schematic diagram of an output matrix 700 is shown, according to an embodiment of the disclosure;

[0021] Figure 8 A schematic block diagram of an image processing apparatus 800 is shown, according to an embodiment of the disclosure; and

[0022] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the disclosure is shown.

[0023] In the various drawings, like or corresponding elements are denoted by like or corresponding reference numerals. DETAILED DESCRIPTION

[0024] Preferred embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0025] The term "comprising" and variations thereof as used herein are intended to mean "including but not limited to." The term "or" as used herein is intended to mean "and / or." The term "based on" means "based, at least in part, on." The terms "one example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "a first," "a second," etc. can refer to different or the same objects. Other explicitly and implicitly recited definitions can also be included below.

[0026] As described above in the BACKGROUND, the techniques for pixel extraction in conventional techniques are inefficient and difficult to meet the needs of users.

[0027] Specifically, the convolution layer is a very common layer in the neural network. In the forward calculation of the convolution layer, a plurality of convolution kernels need to be convolved on the input image. In the CHW format, assuming that the size of the image is c*in_h*in_w, the size of the convolution kernel, which can also be referred to as the weight, is f*c*r*s, and the size of the output image after the convolution operation is f*out_h*out_w, where c is the number of channels of the input image, in_h is the height of the input image, in_w is the width of the input image, f is the number of convolution kernels, r is the height of the convolution kernel, s is the width of the convolution kernel, out_h is the height of the output image, and out_w is the width of the output image. The units of these heights and widths can be pixels.

[0028] A common implementation method is to perform an im2col operation on the input image to expand the part passed through the convolution kernel into out_h*out_w one-dimensional vectors, each with a length of c*r*s, that is, converted into [out_h*out_w, c*r*s]. At this time, the convolution operation can be converted into matrix multiplication, and the specific matrix multiplication size is [f, c*r*s]·[c*r*s, out_h*out_w]=[f, out_h*out_w].

[0029] In the training process, reverse calculation is also needed, in which the weight gradient and the input image gradient of the current layer are calculated according to the output image gradient of the current layer, and are sequentially propagated to the front layer. The solving method of the weight gradient is the output image gradient multiplied by the input image after the im2col operation, and the specific size is [f, out_h*out_w]·[out_h*out_w, c*r*s]=[f, c*r*s], where out_w*out_h is the dimension of multiplication and accumulation.

[0030] In hardware calculation, in order to obtain high throughput, the dimension needs to be arranged into continuous data and output to the matrix operation unit. The output image gradient is [f, out_h*out_w], which does not need additional processing, while the input image needs to be processed from [c, in_h*in_w] to [c*r*s, out_h*out_w] format, which is equivalent to performing im2col operation and transpose operation, so the improvement of the processing efficiency is of great significance to the weight gradient calculation.

[0031] In the traditional technology, two schemes are generally used to solve the weight gradient calculation in the deep learning reverse calculation. One scheme is to use a general-purpose processor to execute general-purpose instructions by using a register stack and a general-purpose calculation unit to complete the deep learning reverse calculation, and the other scheme is to use a special-purpose processing unit using a graphics processing unit to execute single instruction multiple data stream (SIMD) instructions by using a register stack and a stream processing unit to complete the deep learning reverse calculation.

[0032] However, the above two solutions have serious shortcomings.

[0033] For the first solution, the implementation is relatively simple, but the general-purpose processor is often suitable for scalar calculation, and the computing power is insufficient in deep learning related calculation, and the overall performance is low.

[0034] For the first solution, there is still a lack of specific implementation details.

[0035] To at least partially solve one or more of the above problems and other potential problems, embodiments of the present disclosure propose an image processing method. With the technical solutions according to the embodiments of the present disclosure, various scale input image format processing can be flexibly supported, the parallelism characteristics of hardware can be fully utilized, pixel extraction can be efficiently performed, thereby supporting efficient weight gradient calculation, the throughput of the overall calculation process can be improved, and therefore the user experience can be improved.

[0036] Figure 1 A schematic block diagram of an image processing environment 100 in which an image processing method in certain embodiments of the present disclosure can be implemented is shown. According to one or more embodiments of the present disclosure, the image processing environment 100 can be a cloud environment.

[0037] As Figure 1 shown in FIG. 1, the image processing environment 100 includes a computing device 110, and the computing device 110 includes a dedicated processing unit 111-1, a dedicated processing unit 111-2, and a dedicated processing unit 111-3, which can be collectively referred to as dedicated processing units 111. It should be understood that the three dedicated processing units 111 are shown in FIG. 1 for the purpose of example only, and more or fewer dedicated processing units 111 can be included in the image processing environment 100 according to different functions and user needs. According to one or more embodiments of the present disclosure, the dedicated processing units 111 can be, for example, dedicated processing units such as graphics processing units, or processing cores in the aforementioned dedicated processing units. Figure 1

[0038] In the image processing environment 100, the computing device 110 acquires an input image 120. According to one or more embodiments of the present disclosure, the input image 120 can be an input image 120 in CHW format, and the image scale of the input image 120 can be c*in_h*in_w. Therefore, the input image 120 can include multiple channels, and each channel includes in_w rows of pixels.

[0039] ​According to one or more embodiments of the present disclosure, for an input image 120 of CHW, the width dimension, which can be referred to as a row dimension, is contiguous, and the order of storage is to store all pixels of a row first, then store data associated with the height dimension, which can be referred to as a column dimension, and finally store data associated with the channel. For example, if the input image 120 is an RGB image file, the entire image for the R channel is stored first, then the entire image for the G channel is stored, and finally the entire image for the B channel is stored.

[0040] After the input image 120 is obtained, the computing device 110 can use the dedicated processing unit 111 in the computing device 110 to perform pixel extraction in parallel for each row of pixels in the in_w row pixels in each channel of the input image 120 to obtain row data for each row of pixels. Then, the computing device 110 can use the dedicated processing unit 111 in the computing device 110 to stitch the row data for each row of pixels to obtain the output image 130.

[0041] It should be understood that the image processing environment 100 is merely exemplary and not limiting, and it is scalable, in which more computing devices 110 can be included, and the computing devices 110 can obtain more input images 120, and more dedicated processing units 111 can be used to obtain the input images 120 so that more users can simultaneously use more computing devices 110 to perform pixel extraction in parallel to obtain more output images 130, so that the needs of users to use more computing devices 110 and the dedicated processing units 111 in the computing devices to process more input images 120 simultaneously or non-simultaneously to obtain more output images 130 can be met.

[0042] In Figure 1 In the image processing environment 100 shown, the process of obtaining the input image 120 and providing the output image 130 by the computing device 110 can be performed through a network.

[0043] Figure 2 A schematic block diagram of an image processing environment 200 in which an image processing method in certain embodiments of the present disclosure can be implemented is shown. According to one or more embodiments of the present disclosure, the image processing environment 200 can be a cloud environment.

[0044] As Figure 2 As shown in the image processing environment 200, the computing device 210 includes the dedicated processing unit 211-1, the dedicated processing unit 211-2, and the dedicated processing unit 211-3, which can be collectively referred to as the dedicated processing unit 211. It should be understood that in Figure 2Three special processing units 211 are shown in FIG. 2 for the purpose of example only, and more or less special processing units 211 can be included in the image processing environment 200 depending on the functionality and user requirements. According to one or more embodiments of the present disclosure, the special processing units 211 can be, for example, graphics processing units.

[0045] In addition to the special processing units 211, the computing device 210 also includes an instruction storage unit 212, an instruction decoding unit 213, a control unit 214, a synchronization unit 215, a data reading unit 216, and a data processing unit 217.

[0046] In the image processing environment 200, there are also included a source memory 220 from which the computing device 210 can obtain the input image 120, a target memory 230 to which the computing device 210 can store the output image 130, processing instructions 240 that can be input to the computing device 210 for performing various operations, and a scheduler 250 that schedules various operations performed by the computing device 210. According to one or more embodiments of the present disclosure, the input image 120 can be an input image 120 in the CHW format as described with reference to Figure 1 FIG. 1.

[0047] According to one or more embodiments of the present disclosure, the source memory 220 and the target memory 230 can be various types of memories, such as static random access memory (SRAM). In addition, the source memory 220 and the target memory 230 can also be combined into the same memory.

[0048] The instruction storage unit 212 is configured to store the obtained processing instructions 240. For data conversion, the processing instructions 240 include, but are not limited to, parameter configuration instructions, data operation instructions, synchronization instructions, and the like. The parameter configuration instructions are configured to configure parameters including, but not limited to, data types, sizes of input images, sizes of convolution kernels or weights, convolution step sizes, padding sizes, memory addresses and offsets, and the like. The data operation instructions are configured to initiate subsequent hardware execution actions. The synchronization instructions are configured to ensure that all instructions before the instruction are executed and data is flushed, for the scheduler to synchronize the modules.

[0049] The instruction decoding unit 213 reads and parses the processing instructions from the instruction storage unit 212 when detecting that the instruction storage unit 212 is not empty and the current executable instruction, and sends the parsed content to the control unit 214.

[0050] The control unit 214 generates corresponding control signals according to the configuration parameters, which can be used to control the following: read request operations of the data reading unit 216, operations of the data processing unit 217 and the special processing units 211, and operations of the synchronization unit 215.

[0051] The data reading unit 216 sends a read request to the source memory 220 according to the control signal from the control unit 214. The data reading unit 216 can initiate multiple read requests to the source memory 220 according to the control signal from the control unit 214.

[0052] The data processing unit 217 and the special processing unit 211 perform certain operations on the read data according to the control signal from the control unit 214. For example, pixel extraction is performed in parallel for each row of pixels in the in_w row of pixels in each channel of the read input image 120.

[0053] The synchronization unit 215 outputs a synchronization completion signal to the dispatcher 250 after detecting that the current instruction is completed and the data is written to disk after receiving a synchronization request.

[0054] Specifically, according to one or more embodiments of the present disclosure, the specific operation flow of the computing device 210 can include: acquiring the processing instruction 240 including the configuration instruction, the data operation instruction and the synchronization instruction to the instruction storage unit 212, performing instruction decoding by the instruction decoding unit 213, and the control unit 214 outputting control signals to the subsequent modules such as the data reading unit 216, the data processing unit 217, the special processing unit 211 and the synchronization unit 215 according to the parameter configuration. The data reading unit 216 initiates a continuous read operation on the source memory 220 in a certain way according to the control signal. After the read data is returned, the data processing unit 216 performs operations such as shifting and padding on the data according to the control signal. The special processing unit 211 acquires data from the data processing unit 217, and synchronously performs extraction and splicing operations on the data belonging to the part responsible for it according to the control signal, and finally writes to the target memory 230. When the special processing unit 211 processes the data, different special processing units 211 are responsible for processing different points on the convolution kernel, and the control unit 214 performs several cycles during the entire processing process until all points of the entire convolution kernel are completed. After all the data is written to disk, the synchronization module 215 judges according to the control signal of the control unit 214 and the control signal returned by the special processing unit 211, and returns an operation completion signal to the dispatcher 250 to indicate that the data operation has been completed.

[0055] It should be understood that the image processing environment 200 is also merely exemplary and not limiting, and it is scalable. In addition, in Figure 2 In the illustrated image processing environment 200, the processes of the computing device 210 acquiring the input image 120 from the source memory 220, providing the output image 130 to the target memory 230, acquiring the instruction 240 and interacting with the dispatcher 250 can be performed through a network.

[0056] Figure 3A flowchart of an image processing method 200 according to an embodiment of the present disclosure is shown. Specifically, the image processing method 300 can be performed by the computing device 110 in the image processing environment 100 shown in Figure 1 or the computing device 210 in the image processing environment 200 shown in Figure 2 It should be appreciated that the image processing method 300 can also include additional operations not shown and / or can omit some of the operations shown, without limitation to the scope of the present disclosure. The following describes each step of the image processing method 300 with the computing device 110 as the performing subject.

[0057] At block 302, the computing device 110 acquires an input image. According to one or more embodiments of the present disclosure, the input image is an input image in CHW format, and includes multiple rows of pixels.

[0058] The specific form of the input image is described below with reference to Figure 4 .

[0059] Figure 4 A schematic diagram of an input image 400 according to an embodiment of the present disclosure is shown. As shown in Figure 4 , the input image 400 has one convolution kernel represented by the hatched part of the horizontal line. The label 401 represents the number of channels c of the input image 400, and at the same time also represents the number of channels of the convolution kernel. The labels 402 and 403 represent the height in_h and the width in_w of the input image 400, respectively, which can also be considered as the column dimension and the row dimension of the input image 400, respectively. The labels 404 and 405 represent the height r and the width s of the convolution kernel, respectively. According to one or more embodiments of the present disclosure, the units of the height in_h and the width in_w of the input image can be pixels of the input image 400.

[0060] In the method 300, it is first described with the input image 400 and the number of channels c of the convolution kernel being 1 as an example.

[0061] At block 304, the computing device 110 performs pixel extraction in parallel using the multiple special-purpose processing units 111 for each row of pixels in the multiple rows of pixels of the input image, to obtain row data for each row of pixels.

[0062] In the forward calculation of the convolution layer, the convolution kernel moves in the row direction by a step and traverses the entire row, then returns to the beginning of the row, moves down by a step, and continues to traverse the entire row in the row direction, and the above operation is repeated until the entire image is traversed. One implementation is to extract the data covered by the position of the convolution kernel each time it moves in sequence and expand it into a one-dimensional vector, that is, the im2col operation, and the scale of the expanded matrix is [out_h*out_w, c*r*s]. Then the data after the im2col operation and the convolution kernel are multiplied by the matrix, and the scale is [f, c*r*s]·[c*r*s, out_h*out_w] = [f, out_h*out_w]. It should be noted that the accumulation dimension in the matrix multiplication formula is [c*r*s], and the two matrices input into the matrix multiplication module are generally row storage and column storage, so the data after the im2col operation does not need to be transposed.

[0063] In the backward weight gradient calculation, the output data weight and the input data after the im2col operation need to be multiplied by the matrix, and the scale is [f, out_h*out_w]·[out_h*out_w, c*r*s] = [f, c*r*s]. Since the accumulation dimension is [out_h*out_w], the input data needs to be arranged in the format [c*r*s, out_h*out_w], and the continuous storage of the out_h*out_w dimension can ensure efficient data supply to the matrix multiplication unit.

[0064] According to one or more embodiments of the present disclosure, the computing device 110 uses multiple special-purpose processing units 111 to perform pixel extraction in parallel to improve the efficiency of pixel extraction in the above process.

[0065] According to one or more embodiments of the present disclosure, the computing device 110 can add padding to at least one side edge of the input image to obtain a padded input image. According to the algorithm used and actual requirements, the above padding can be pixels with values of all 0 or all 1, or can be values according to other preset rules, such as copying the values in the outermost pixels of the input image. In the case of adding padding, the computing device 110 uses multiple special-purpose processing units 111 to perform pixel extraction in parallel for each row of pixels included in the padded input image to obtain row data for each row of pixels.

[0066] In block 306, the computing device 110 splices the row data for each row of pixels obtained in block 304 to obtain an output image. According to one or more embodiments of the present disclosure, the operation in block 306 can also be performed by the computing device 110 using the special-purpose processing units 111.

[0067] According to one or more embodiments of the present disclosure, as previously described, the input image can include multiple channels, and each channel can include multiple rows of pixels. In this case, in block 304, the computing device 110 can perform, in parallel using the multiple dedicated processing units 111, pixel extraction for each row of pixels in the multiple rows of pixels of each channel of the input image to obtain row processing data for each row of pixels of the channel. Then, in block 306, the computing device 110 can stitch the row processing data for each row of pixels of the channel to obtain channel data for the channel, and stitch the channel data for each channel to obtain the output image.

[0068] The above is described with reference to Figure 3 An image processing method 300 according to an embodiment of the present disclosure is described. According to one or more embodiments of the present disclosure, the image processing method 300 can also be extended with additional steps. The above is described with reference to Figure 5 The above is described with reference to

[0069] Figure 5 A flowchart of an image processing method 500 according to an embodiment of the present disclosure is shown. Specifically, the image processing method 500 can be performed by the computing device 110 in the image processing environment 100 shown in Figure 1 or the computing device 210 in the image processing environment 200 shown in Figure 2 It should be understood that the image processing method 500 can also include additional operations not shown and / or can omit some of the operations shown, and the scope of the present disclosure is not limited in this regard. The following describes each step of the image processing method 500 with the computing device 110 as an example.

[0070] In block 502, the computing device 110 obtains an input image. According to one or more embodiments of the present disclosure, the input image is an input image in CHW format and includes multiple rows of pixels. The specific content of the steps involved in block 502 is the same as that of the steps involved in block 302, and is not described here again.

[0071] In block 504, the computing device 110 determines the number of pixels each of the multiple dedicated processing units 111 processes each time when performing pixel extraction for each row of pixels based on the width of the convolution kernel used to perform the pixel extraction.

[0072] According to one or more embodiments of the present disclosure, the width of the convolution kernel used to perform the pixel extraction can be pre-configured according to an algorithm or user demand, and the number of pixels each of the dedicated processing units processes each time when performing pixel extraction for each row of pixels can be equal to the width of the convolution kernel. For example, if the width of the convolution kernel is 3, each of the dedicated processing units can process 3 pixels each time when performing pixel extraction for each row of pixels.

[0073] At block 506, the computing device 110 determines, based on the width stride of the convolution kernel used to perform the pixel extraction, the pixels that are to be targeted each time by each of the plurality of special-purpose processing units 111 when performing the pixel extraction for each row of pixels.

[0074] According to one or more embodiments of the present disclosure, the width stride of the convolution kernel used to perform the pixel extraction can be pre-configured according to an algorithm or user demand. For example, if the width stride of the convolution kernel used to perform the pixel extraction is 2, then in each row of pixels, each of the special-purpose processing units 111 will select one pixel every 1 pixels for performing the pixel extraction.

[0075] At block 508, the computing device 110 determines, based on the height stride of the convolution kernel used to perform the pixel extraction, the order in which each of the plurality of special-purpose processing units 111 performs the pixel extraction for the plurality of rows of pixels.

[0076] According to one or more embodiments of the present disclosure, the height stride of the convolution kernel used to perform the pixel extraction can be pre-configured according to an algorithm or user demand. For example, if the height stride of the convolution kernel used to perform the pixel extraction is 2, then when performing the pixel extraction, each of the special-purpose processing units 111 will select one row of pixels every 1 rows of pixels for performing the pixel extraction.

[0077] According to one or more embodiments of the present disclosure, the steps involved in blocks 504, 506 and 508 in the method 500 can not all be included, but only a part of them, or even none of them.

[0078] At block 510, the computing device 110 performs, for each row of pixels in the plurality of rows of pixels of the input image, the pixel extraction in parallel using the plurality of special-purpose processing units 111 to obtain row data for each row of pixels. The specific content of the steps involved in block 510 is the same as that of the steps involved in block 304, which will not be repeated here.

[0079] At block 512, the computing device 110 stitches the row data for each row of pixels obtained in block 304 to obtain the output image. The specific content of the steps involved in block 512 is the same as that of the steps involved in block 306, which will not be repeated here.

[0080] The image processing method 300 according to an embodiment of the present disclosure and the image processing method 500 according to an embodiment of the present disclosure are described above respectively with reference to Figure 3 and Figure 5 The image processing method 300 according to an embodiment of the present disclosure and the image processing method 500 according to an embodiment of the present disclosure are described above respectively with reference to Figures 6A to 6DThe input data changes in the image processing method brought by the image processing method 300 or the image processing method 500 according to the embodiments of the present disclosure are described.

[0081] Figure 6A A schematic diagram of input data changes 610 in the image processing method according to the embodiments of the present disclosure is shown. In the input data changes 610 shown in Figure 6A In the input data changes 610 shown in

[0082] In the input data changes 610 shown in Figures 6A to 6D In the input data changes 610 shown in, the preset channel number c = 1, the input image height in_h = 3, the input image width in_w = 5, the convolution kernel height r = 2, the convolution kernel width s = 3, the width stride stride_w = 2, the height stride stride_h = 1, the upper padding pad_up = 0, the lower padding pad_down = 0, the left padding pad_left = 1, and the right padding pad_right = 1 are taken as examples to be described.

[0083] In the input data changes 610 shown in Figure 6A In the input data changes 610 shown in, the first row of the input data 601 is added with a dashed box to indicate that the pixels in this row will be processed, the first row of the convolution kernel 602 is added with a dashed box to indicate that the convolution kernel in this row will be processed, the label 611 indicates that the data 611 in the first row is extracted from the input data 601, the label 612 indicates that the computing device 110 adds padding pad to the data 611 in the first row extracted from the input data 601 and performs division, and the labels 613, 614, and 615 indicate three row data for the data 611 in the first row extracted from the input data 601 processed by the three special data processing units respectively, wherein the block r0s0 at the upper left corner of the convolution kernel is associated with the row data of the label 613, the block r0s1 in the upper middle of the convolution kernel is associated with the row data of the label 614, and the block r0s2 at the upper right corner of the convolution kernel is associated with the row data of the label 615. As described above, since the upper padding pad_up and the lower padding pad_down are both equal to 0, and the left padding pad_left and the right padding pad_right are both equal to 1, it can be seen that one pixel of padding is added to the left and right of the row indicated by the label 612.

[0084] In the input data changes 610 shown in Figure 6AIt can be seen that since padding is added to the left and right sides of the data 611 of the first row extracted from the input data 601, and the width step is 2, the three special data processing units process the 613, 614 and 615 formed every 1 pixel in the padded first row data 612 in the input data 601, respectively.

[0085] In Figure 6B , the second row of the input data 601 is added with a dashed box indicating that the pixels in this row will be processed, the first row of the convolution kernel 602 is added with a dashed box indicating that the convolution kernel in this row will be processed, the number 621 indicates that the data 621 of the second row is extracted from the input data 601, the number 622 indicates that the computing device 110 adds padding pad to the data 621 of the second row extracted from the input data 601 and performs division, and the numbers 623, 624 and 625 indicate three row data for the data 621 of the second row extracted from the input data 601 after being processed by the three special data processing units, respectively, wherein the block r0s0 at the upper left corner of the convolution kernel is associated with the row data of the number 623, the block r0s1 in the upper middle of the convolution kernel is associated with the row data of the number 624, and the block r0s2 at the upper right corner of the convolution kernel is associated with the row data of the number 625.

[0086] By Figure 6B It can be seen that since padding is added to the left and right sides of the data 621 of the second row extracted from the input data 601, and the width step is 2, the three special data processing units process the 623, 624 and 625 formed every 1 pixel in the padded first row data 622 in the input data 601, respectively.

[0087] In Figure 6C , the second row of the input data 601 is added with a dashed box indicating that the pixels in this row will be processed, the second row of the convolution kernel 602 is added with a dashed box indicating that the convolution kernel in this row will be processed, the number 631 indicates that the data 631 of the second row is extracted from the input data 601, the number 632 indicates that the computing device 110 adds padding pad to the data 631 of the second row extracted from the input data 601 and performs division, and the numbers 633, 634 and 635 indicate three row data for the data 631 of the second row extracted from the input data 601 after being processed by the three special data processing units, respectively, wherein the block r1s0 at the lower left corner of the convolution kernel is associated with the row data of the number 633, the block r1s1 in the lower middle of the convolution kernel is associated with the row data of the number 634, and the block r1s2 at the lower right corner of the convolution kernel is associated with the row data of the number 635.

[0088] By Figure 6CIt can be seen that since padding is added to the left and right sides of the data 631 of the 2nd row extracted from the input data 601, and the width step is 2, the three special data processing units process the 633, 634 and 635 formed by every other pixel in the padded first row data 632 in the input data 601 respectively.

[0089] In Figure 6D , the 3rd row in the input data 601 is added with a dashed box indicating that the pixels in this row will be processed, the 2nd row in the convolution kernel 602 is added with a dashed box indicating that the convolution kernel in this row will be processed, the label 641 indicates that the data 641 in the 3rd row is extracted from the input data 601, the label 642 indicates that the computing device 110 adds padding pad to the data 641 in the 3rd row extracted from the input data 601 and performs division, and the labels 643, 644 and 645 indicate three rows of data for the data 641 in the 3rd row extracted from the input data 601 after processing by the three special data processing units respectively, wherein the block r1s0 at the lower left corner of the convolution kernel is associated with the row data of the label 643, the block r1s1 in the middle of the convolution kernel is associated with the row data of the label 644, and the block r1s2 at the lower right corner of the convolution kernel is associated with the row data of the label 645.

[0090] By Figure 6D It can be seen that since padding is added to the left and right sides of the data 641 of the 3rd row extracted from the input data 601, and the width step is 2, the three special data processing units process the 643, 644 and 645 formed by every other pixel in the padded first row data 642 in the input data 601 respectively.

[0091] It should be noted that since Figures 6A to 6D In the example of the convolution kernel with a height of 2 and the convolution kernel moving as a whole on the input image 601, in order to avoid the convolution kernel moving beyond the range of the input image 601, the first row of the convolution kernel only moves through the first and second rows of the input data 601, and the second row of the convolution kernel only moves through the second and third rows of the input data 601.

[0092] After Figures 6A to 6D the row data 613, 614, 615, 623, 624, 625, 633, 634, 635, 643, 644 and 645 are obtained respectively, they can be spliced to obtain an output matrix, which will be described below with reference to Figure 7 .

[0093] Figure 7A schematic diagram of an output matrix 700 according to an embodiment of the present disclosure is shown. In matrix 700, the first row contains values ​​in row data 613 and 623 associated with the block r0s0 at the upper left corner of the convolution kernel; the second row contains values ​​in row data 614 and 624 associated with the block r0s1 above the convolution kernel; the third row contains values ​​in row data 615 and 625 associated with the block r0s2 at the upper right corner of the convolution kernel; the fourth row contains values ​​in row data 633 and 643 associated with the block r1s0 at the lower left corner of the convolution kernel; the fifth row contains values ​​in row data 634 and 644 associated with the block r1s1 below the convolution kernel; and the sixth row contains values ​​in row data 635 and 645 associated with the block r1s2 at the lower right corner of the convolution kernel. Therefore, the size of the output matrix 700 is [c*r*s,out_h*out_w]=[6,6], where each row of the output matrix 700 represents the data taken out when the corresponding element of the convolution kernel slides on the image.

[0094] It should be understood that, in reference Figures 6A to 6D as well as Figure 7 In the described example, the top-left block r0s0 and bottom-left block r1s0 of the convolution kernel are processed by one dedicated processing unit, the top block r0s1 and bottom block r1s1 of the convolution kernel are processed by another dedicated processing unit, and the top-right block r0s2 and bottom-right block r1s2 of the convolution kernel are processed by yet another dedicated processing unit. However, in practical applications, this correspondence is not limited. Furthermore, for scenarios where the number of channels c is greater than 1, the above operations can be performed on each channel of the input image, and then the output matrices obtained for each channel can be concatenated to obtain the output matrix for an input image with multiple channels. For example, if the number of channels c in the above example is 2, the resulting output matrix can be a 12x6 matrix.

[0095] Furthermore, it should be understood that the rows and columns stored in the above examples are merely logical concepts and do not represent their actual locations in memory.

[0096] Furthermore, according to one or more embodiments of this disclosure, parallel computing is most efficient when the number of dedicated processing units is not less than the width of the convolution kernel.

[0097] The above is for reference only. Figures 1 to 7 Image processing environment 100, image processing environment 200, image processing method 300, and image processing method 500 according to embodiments of the present disclosure are described in relation to image processing methods that may be implemented in certain embodiments of the present disclosure. It should be understood that the above description is intended to better illustrate the content set forth in this disclosure and is not intended to be limiting in any way.

[0098] It should be understood that the number of various elements and the size of physical quantities employed in the above-described various drawings of the present disclosure are merely examples, and are not a limitation on the scope of protection of the present disclosure. The above-mentioned numbers and sizes can be arbitrarily set as needed without affecting the normal implementation of the embodiments of the present disclosure.

[0099] The details of the image processing method 300 and the image processing method 500 according to the embodiments of the present disclosure have been described above. In the following, the details of the image processing apparatus according to the embodiments of the present disclosure will be described with reference to Figures 1 to 7 The details of the image processing method 300 and the image processing method 500 according to the embodiments of the present disclosure have been described above. In the following, the details of the image processing apparatus according to the embodiments of the present disclosure will be described with reference to Figure 4 The details of the image processing method 300 and the image processing method 500 according to the embodiments of the present disclosure have been described above. In the following, the details of the image processing apparatus according to the embodiments of the present disclosure will be described with reference to

[0100] Figure 8 is a schematic block diagram of an image processing apparatus 800 according to an embodiment of the present disclosure. As shown in Figure 8 The image processing apparatus 800 includes an acquisition module 810 configured to acquire an input image, the input image including a plurality of rows of pixels; a first calculation module 820 configured to, for each row of pixels of the plurality of rows of pixels of the input image, perform pixel extraction in parallel using a plurality of dedicated processing units to obtain row data for each row of pixels; and a first stitching module 830 configured to stitch the row data for each row of pixels to obtain an output image.

[0101] In one or more embodiments, wherein the first calculation module 820 includes a padding module (not shown) configured to add padding at an edge of at least one side of the input image to obtain a padded input image; and a second calculation module (not shown) configured to, for each row of pixels of the plurality of rows of pixels included in the padded input image, perform pixel extraction in parallel using the plurality of dedicated processing units to obtain row data for each row of pixels.

[0102] In one or more embodiments, wherein the first calculation module 820 includes a pixel number determination module (not shown) configured to determine, based on a width of a convolution kernel used to perform the pixel extraction, a number of pixels each of the plurality of dedicated processing units targets each time when performing the pixel extraction for each row of pixels.

[0103] In one or more embodiments, wherein the first calculation module 820 includes a pixel determination module (not shown) configured to determine, based on a width step of a convolution kernel used to perform the pixel extraction, a number of pixels each of the plurality of dedicated processing units targets each time when performing the pixel extraction for each row of pixels.

[0104] In one or more embodiments, the first computing module 820 comprises: a sequence determining module (not shown) configured to determine a sequence in which the pixel extraction is performed by each of the plurality of special-purpose processing units for the multiple rows of pixels based on a height step of a convolution kernel used to perform the pixel extraction.

[0105] In one or more embodiments, the input image comprises at least one channel, each of the at least one channel comprising a plurality of rows of pixels; the first computing module 820 comprises: a third computing module (not shown) configured to, for each row of pixels in the plurality of rows of pixels of each channel of the input image, perform the pixel extraction using the plurality of special-purpose processing units in parallel to obtain row processing data for each row of pixels of the channel; and the first stitching module 830 comprises: a second stitching module (not shown) configured to stitch the row processing data for each row of pixels of the channel to obtain channel data for the channel; and a third stitching module (not shown) configured to stitch the channel data for each channel to obtain the output image.

[0106] By the above description with reference to Figures 1 to 8 the technical solution according to the embodiments of the present disclosure has many advantages over the conventional solution. For example, with the technical solution according to the embodiments of the present disclosure, various scale input image format processing can be flexibly supported, the parallelism feature of hardware can be fully utilized, and the pixel extraction can be efficiently performed, so that efficient weight gradient calculation can be supported, the throughput of the overall calculation process can be improved, and thus the user experience can be improved.

[0107] Specifically, with the technical solution according to the embodiments of the present disclosure, the parameters can be flexibly configured based on the received parameter configuration instruction, so that various input image scales and convolution kernel scales can be adapted. Meanwhile, since the plurality of special-purpose processing units work in parallel, and each special-purpose processing unit can obtain multiple data at a time and perform stitching, the input data processing work can be efficiently completed in general. In addition, since the technical solution according to the embodiments of the present disclosure is equivalent to performing the im2col operation and the transpose operation at a time, it does not need to occupy additional storage space for implementation.

[0108] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a computer readable storage medium, and a computer program product.

[0109] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. For example, the computing device 110 as shown in Figure 1 , the computing device 210 as shown in Figure 2 , and the computing device 310 as shown in Figure 8The image processing apparatus 800 shown can be implemented by electronic device 900. Electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 900 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0111] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the methods 300 and 500. For example, in some embodiments, the methods 300 and 500 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the methods 300 and 500 described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the methods 300 and 500 by other any appropriate means, such as by means of firmware.

[0113] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0114] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0117] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0118] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0119] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0120] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method of image processing, comprising: obtaining an input image based on a first control signal generated according to configuration parameters, the input image comprising a plurality of rows of pixels; for each row of pixels of the plurality of rows of pixels of the input image, performing pixel extraction based on the first control signal using a plurality of dedicated processing units in parallel to obtain row data for each row of pixels, wherein performing the pixel extraction in parallel comprises determining an order in which the pixel extraction is performed by each of the plurality of dedicated processing units for the plurality of rows of pixels based on a height stride of a convolution kernel used to perform the pixel extraction, and comprises one of: determining a number of pixels for which the pixel extraction is performed by each of the plurality of dedicated processing units for each row of pixels at a time based on a width of the convolution kernel used to perform the pixel extraction, and determining a number of pixels for which the pixel extraction is performed by each of the plurality of dedicated processing units for each row of pixels at a time based on a width stride of the convolution kernel used to perform the pixel extraction; providing a plurality of second control signals based on completion of performance of the pixel extraction by the plurality of dedicated processing units; generating an operation completion signal based on the first control signal and the plurality of second control signals; and stitching the row data for each row of pixels based on the first control signal and the operation completion signal to obtain an output image.

2. The method of claim 1, wherein performing the pixel extraction in parallel comprises: adding padding to edges of at least one side of the input image to obtain a padded input image; and performing the pixel extraction in parallel using the plurality of dedicated processing units for each row of pixels of a plurality of rows of pixels comprised by the padded input image to obtain row data for each row of pixels.

3. The method of claim 1, wherein: the input image comprises at least one channel, each channel of the at least one channel comprising a plurality of rows of pixels; performing the pixel extraction in parallel comprises: performing the pixel extraction in parallel using the plurality of dedicated processing units for each row of pixels of the plurality of rows of pixels of each channel of the input image to obtain row processing data for each row of pixels of the channel; and stitching the row data for each row of pixels comprises: stitching the row processing data for each row of pixels of the channel to obtain channel data for the channel; and stitching the channel data for each channel to obtain the output image.

4. An apparatus for image processing, comprising: an obtaining module configured to obtain an input image based on a first control signal generated according to configuration parameters, the input image comprising a plurality of rows of pixels; ​ ​ a first computing module configured to, for each row of pixels of the multiple rows of pixels of the input image, perform pixel extraction in parallel using multiple dedicated processing units based on the first control signal to obtain row data for each row of pixels, wherein the first computing module comprises a sequential determination module configured to determine a sequence in which each of the multiple dedicated processing units performs the pixel extraction for the multiple rows of pixels based on a height stride of a convolution kernel used to perform the pixel extraction, and comprises one of: a pixel number determination module configured to determine a number of pixels for which each of the multiple dedicated processing units performs the pixel extraction for each row of pixels at each time based on a width of the convolution kernel used to perform the pixel extraction, and a pixel determination module configured to determine a pixel for which each of the multiple dedicated processing units performs the pixel extraction for each row of pixels at each time based on a width stride of the convolution kernel used to perform the pixel extraction; an execution completion indication module configured to provide multiple second control signals based on a completion of the pixel extraction by the multiple dedicated processing units; an operation completion signal generation module configured to generate an operation completion signal based on the first control signal and the multiple second control signals; and a first splicing module configured to splice the row data for each row of pixels based on the first control signal and the operation completion signal to obtain an output image.

5. The apparatus of claim 4, wherein the first computing module comprises: a padding module configured to add padding to an edge of at least one side of the input image to obtain a padded input image; and a second computing module configured to, for each row of pixels of multiple rows of pixels comprised by the padded input image, perform the pixel extraction in parallel using the multiple dedicated processing units to obtain row data for each row of pixels.

6. The apparatus of claim 4, wherein: the input image comprises at least one channel, each of the at least one channel comprising multiple rows of pixels; the first computing module comprises: a third computing module configured to, for each row of pixels of the multiple rows of pixels of each channel of the input image, perform the pixel extraction in parallel using the multiple dedicated processing units to obtain row processing data for each row of pixels of the channel; and the first splicing module comprises: a second splicing module configured to splice the row processing data for each row of pixels of the channel to obtain channel data for the channel; and a third splicing module configured to splice the channel data for each channel to obtain the output image. comprises: at least one processor; and 7. An electronic device, comprising: a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3. ​ ​ ​ 8. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-3.

9. A computer program product comprising computer instructions which, when executed by a processor, perform the method of any one of claims 1-3.

Images