An image processing method, apparatus and computing device

Abstract

An image processing method comprises: acquiring a first image; for any one pixel point in the first image, using a plurality of look-up tables associated with the any one pixel point to determine an image block corresponding to the any one pixel point, the image block comprising one or more pixel points, the index region of each look-up table containing the any one pixel point, and the index regions of different look-up tables being different and at least partially the same; obtaining a second image according to the image blocks corresponding to each pixel point in the first image, the resolution of the second image being higher than or equal to the resolution of the first image; and outputting the second image. In this way, in the image processing process, a plurality of look-up tables are used to work cooperatively for each pixel point, thereby expanding the receptive field while maintaining linear growth of the calculation amount and the storage amount, and improving the image processing performance.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an image processing method, apparatus and computing device. Background Technology

[0002] Image processing is a major research area in low-level computer vision. In recent years, methods based on convolutional neural networks (CNNs) have achieved remarkable results in image processing performance. Generally, more complex neural networks can achieve better results. However, due to limitations in operating speed and power consumption, networks for image processing on terminal devices with relatively weak computing power, such as mobile phones, generally cannot be too complex. Although this constraint can be alleviated through carefully designed network structures or dedicated computing engines (such as graphics processing units), the corresponding hardware costs and power consumption still increase the difficulty of deploying existing CNN methods on terminal devices. On the other hand, in terminal devices such as smart cars, application scenarios such as lane changing during high-speed driving and handling of sudden emergencies place high demands on the runtime latency of image processing methods. Therefore, how to design an efficient image processing method that meets the requirements of low power consumption and low latency when deployed on terminal devices, while also possessing good image processing performance, is a technical problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides an image processing method, apparatus, computing device, computing device cluster, computer storage medium, and computer product that can meet the requirements of low power consumption and low latency when deployed on terminal devices, while also having good image processing performance.

[0004] In a first aspect, this application provides an image processing method, the method comprising: acquiring a first image; for any pixel in the first image, using N (N≥2) lookup tables associated with the arbitrary pixel to determine the image block corresponding to the arbitrary pixel, wherein the image block includes one or more pixels, wherein the index area of ​​each lookup table contains the arbitrary pixel, and the index areas of different lookup tables are different and at least partially the same; obtaining a second image based on the image blocks corresponding to each pixel in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and outputting the second image.

[0005] In this way, multiple lookup tables are used collaboratively for each pixel during image processing, thereby expanding the receptive field and improving image processing performance while maintaining a linear increase in computational and storage requirements. In contrast, existing lookup table techniques that utilize spatial information use only a single lookup table, and the expansion of the receptive field leads to an exponential increase in storage requirements, which is limited by memory size and hardware cost.

[0006] In one possible implementation, N lookup tables associated with any given pixel are used to determine the image block corresponding to that pixel. Specifically, this involves: inputting the index region of each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, each containing one or more pixels; summing the pixels at the same position in the N first sub-image blocks and averaging the sums to obtain the image block corresponding to the given pixel. Since the index region corresponding to the output of the lookup table increases, the receptive field of the lookup table also increases accordingly. Simultaneously, the dimension of the lookup table remains unchanged, thus increasing the receptive field of the lookup table without increasing its dimension.

[0007] In one possible implementation, the index region of each of the N lookup tables is input into the corresponding lookup table to obtain N first sub-image blocks. Specifically, this includes: for any index region of any of the N lookup tables, rotating the index region of any lookup table around any pixel point as the center and along a first direction M times to obtain M index regions, where M≥1; based on the lookup table corresponding to each of the M index regions, determining the second sub-image blocks corresponding to each of the M index regions to obtain M second sub-image blocks, each of which includes one or more pixels; based on the rotation angle corresponding to each of the M index regions, rotating the second sub-image blocks corresponding to each of the M index regions along the opposite direction to the first direction by the same angle to obtain M third sub-image blocks; adding the pixels at the same position in the first sub-image blocks and the M third sub-image blocks corresponding to the index region of any lookup table and averaging them, and using the calculation result as the first sub-image block corresponding to the index region of the arbitrary lookup table. This allows for a further increase in the receptive field of the lookup table without increasing its dimensions by rotating the index region.

[0008] In one possible implementation, the output of the i-th lookup table in the N lookup tables is the input of the (i+1)-th lookup table, the input of the first lookup table in the N lookup tables is the index region of the first lookup table, and the output of the N-th lookup table in the N lookup tables is the image patch corresponding to any given pixel. That is to say, the N lookup tables can be cascaded, thereby increasing the receptive field of the lookup tables.

[0009] In one possible implementation, N lookup tables are composed of lookup tables contained in M ​​lookup table sets, with each lookup table set containing multiple lookup tables. The input to each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, determined based on the output of the (j-1)-th lookup table set. The input to each lookup table in the 1-th lookup table set is the index region of the corresponding lookup table in the 1-th lookup table set, determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image patch corresponding to any of the aforementioned pixels. Thus, by using multiple complementary and cascaded lookup tables, the image patch corresponding to each pixel is determined, further improving image processing performance.

[0010] Secondly, this application provides an image processing apparatus, comprising an acquisition module and a processing module. The acquisition module is used to acquire a first image. The processing module is used to, for any pixel in the first image, use N (N≥2) lookup tables associated with that pixel to determine the corresponding image block, where each image block includes one or more pixels. The index area of ​​each lookup table contains the arbitrary pixel, and the index areas of different lookup tables are different but at least partially identical. The processing module is further used to obtain a second image based on the image blocks corresponding to each pixel in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and to output the second image.

[0011] In one possible implementation, when the processing module determines the image block corresponding to any given pixel using N lookup tables associated with that pixel, it specifically performs the following steps: inputting the index region of each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, each of which includes one or more pixels; and summing and averaging the pixels at the same position in the N first sub-image blocks to obtain the image block corresponding to that given pixel.

[0012] In one possible implementation, when the processing module inputs the index region of each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, it specifically performs the following steps: For any index region of any of the N lookup tables, it rotates the index region of any lookup table M times along a first direction with any pixel as the center to obtain M index regions, where M ≥ 1; Based on the lookup table corresponding to each of the M index regions, it determines the second sub-image blocks corresponding to each of the M index regions to obtain M second sub-image blocks, each of which includes multiple pixels; Based on the rotation angle corresponding to each of the M index regions, it rotates the second sub-image blocks corresponding to each of the M index regions along the same angle in the opposite direction to the first direction to obtain M third sub-image blocks; It adds up and averages the pixels at the same position in the first sub-image block and the M third sub-image blocks corresponding to the index region of any lookup table, and uses the calculation result as the first sub-image block corresponding to the index region of the arbitrary lookup table.

[0013] In one possible implementation, the output of the i-th lookup table in the N lookup tables is the input of the (i+1)-th lookup table, the input of the 1-th lookup table in the N lookup tables is the index region of the 1-th lookup table, and the output of the N-th lookup table in the N lookup tables is the image block corresponding to any given pixel.

[0014] In one possible implementation, the N lookup tables are composed of lookup tables contained in M ​​lookup table sets, with each lookup table set containing multiple lookup tables. The input to each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, determined based on the output of the (j-1)-th lookup table set. The input to each lookup table in the 1-th lookup table set is the index region of the corresponding lookup table in the 1-th lookup table set, determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image patch corresponding to any of the aforementioned pixels.

[0015] Thirdly, this application provides a computing device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation thereof.

[0016] Fourthly, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0017] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0018] In a sixth aspect, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0019] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0020] Figure 1 This is a comparative schematic diagram showing how CNN networks of different complexities process images, as provided in the embodiments of this application.

[0021] Figure 2 This is a schematic diagram illustrating a process of obtaining a lookup table using a neural network, as provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the architecture of an image processing system provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an index region and an equivalent index region provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram illustrating the rotation of an index region according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating the rotation of the output result of a lookup table according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a hierarchical table-valued reindex provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of a quanta-aware hierarchical neural network training method provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram illustrating the collaborative operation of multiple lookup tables provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram illustrating the adaptation process of a convolutional network to a Y-type indexing pattern, as provided in an embodiment of this application.

[0030] Figure 11 This is a schematic diagram illustrating a process of processing a Bayer pixel image according to an embodiment of this application;

[0031] Figure 12 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0032] Figure 13 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0033] Figure 14 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0034] Figure 15 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0035] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0036] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0037] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0038] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0039] For example, the image processing performance of complex neural networks can be improved because they have larger receptive fields (RF). For instance, as... Figure 1 As shown, Figure 1 (A) is the input to the neural network; Figure 1 (B) is the output of the neural network when it is a simple CNN network and its receptive field is 17x17. Figure 1 (C) is the output of the neural network when it is a complex CNN network and its receptive field is 45x45. Figure 1 (D) is the truth value, i.e., the original image. Through Figure 1 A comparison of (B), (C), and (D) shows that complex neural networks, due to their larger receptive field, can produce results that are closer to the true value, meaning they have better image processing performance.

[0040] Generally, neural networks can be converted into lookup tables (LUTs) and applied to image processing to meet deployment requirements on terminal devices. Specifically, the mapping relationship between input and output data can be obtained using the trained neural network and stored in a lookup table in key-value pairs. Then, image processing is performed using this lookup table. For example, after obtaining input data, the corresponding numerical result can be found in the lookup table using that input data, and then that result can be output. This completes the image processing.

[0041] For example, such as Figure 2 As shown in (a), a neural network for image processing can be trained first. This neural network can consist of several convolutional layers, with input and output being single-channel images. Unlike typical image processing neural networks, the receptive field of this network is strictly limited to the input range of 2x2, meaning that a set of output results depends only on the corresponding 4-dimensional inputs I0, I1, I2, I3. Parameter optimization of this neural network can be achieved using conventional backpropagation.

[0042] After training, the neural network can be converted into a lookup table. For example... Figure 2 As shown in (b), after training the network parameters, the input of the neural network is constructed by traversing the sampled numerical combinations of 4-dimensional pixels, and the corresponding output results are obtained through calculation. Then, the results obtained from the traversal are stored in a lookup table in key-value format, resulting in a 4-dimensional lookup table, i.e., 4DLUT.

[0043] Finally, image processing can be performed using transformed lookup tables. For example... Figure 2As shown in (c), given a set of inputs I0, I1, I2, I3 centered at I0, the system searches for the numerical results already stored in the table. If the results corresponding to sampling points I0, I1, I2, I3 are not directly stored in 4DLUT, the key values ​​of the sampling points adjacent to the input are first obtained, and then the output results are calculated through interpolation. This completes the image processing. For an example, please refer to [further details]. Figure 2 (c) When it is necessary to process pixel I0 in an image, the input can be I0, I1, I2, I3, and then the output pixel can be obtained. That is, the image block corresponding to I0 is obtained; after obtaining the image blocks corresponding to each pixel in the image, these image blocks are placed at the positions of their respective pixel points to obtain the processed image. In some embodiments, an image block may include one pixel or multiple pixels, depending on the actual situation.

[0044] While this approach can be deployed on terminal devices, the storage size required for the lookup table often increases exponentially with the linear increase of the receptive field of the neural network. For example, with 4DLUT, for an image sampled in 4 bits, 4DLUT requires a total of (2... 4 +1) 4 = 1.274MB of space is needed to store the image processing results. However, when the LUT dimension is increased by 1 dimension, a 5DLUT requires (2... 4 +1) 5 =21.665MB of storage space. Therefore, as the dimensions of the lookup table increase, the storage space requirements of the terminal device also increase, making it difficult to deploy on terminal devices with limited memory. In addition, the size of the receptive field (i.e., the index range) of a single lookup table is limited by memory size and hardware cost, so only lookup tables with smaller index ranges can be built, which leads to limited image processing performance.

[0045] In view of this, the embodiments of this application provide an image processing method that can increase the size of the receptive field of the lookup table without increasing the dimension of the lookup table during image processing, which significantly alleviates the constraints of memory size and hardware cost on the terminal device, thereby greatly improving its performance in various image processing tasks while maintaining the high efficiency and low power consumption characteristics of the lookup table method.

[0046] For example, Figure 3 A schematic diagram of the architecture of an image processing system is shown. Figure 3 As shown, the image processing system 300 may include: a multi-table collaborative processing unit 310 based on spatial complementary indexing and / or a multi-level processing unit 320 based on table value re-indexing.

[0047] The multi-table collaborative processing unit 310 based on spatial complementary indexing is mainly used to define the center pixel of any local region in the input image as I0, and construct multiple complementary index regions around the center pixel I0, where each index region includes the center pixel I0. Based on the constructed multiple complementary index regions, a lookup table corresponding to each index region is determined. For example, the constructed index regions can be input into a pre-trained neural network model to obtain the lookup table corresponding to each index region. For example, the complementarity described in this embodiment differs from the strict mathematical definition; in this embodiment, complementarity means that the index regions can overlap. The complementarity described in this embodiment only requires that each lookup table has an area that cannot be covered by other lookup table indexes. In addition, the multi-table collaborative processing unit 310 based on spatial complementary indexing can also determine the image block corresponding to each pixel based on the lookup table corresponding to each index region associated with each pixel, thereby completing image processing.

[0048] To construct multiple complementary index regions around the center point I0, taking the construction of a single index region as an example, the number of pixels contained in the index region can be determined by the dimensions of the pre-defined lookup table. For example, when the lookup table has a 4-dimensional dimension, the index region contains 4 pixels, and these 4 pixels include the center point I0.

[0049] The process of determining the lookup table, taking the determination of a 4D lookup table and a region in the input image as an example, is as follows: Figure 4 As shown, the center pixel of this region can be defined as I0. Then, multiple complementary indices, i.e., LUTs, are constructed around the center point I0. (1) Indexed regions, LUTs (2) Index region, and LUT (3) Index region. Where, LUT... (1) The index region is [I0][I1][I3][I4], and the LUT is... (2) The index region is [I0][I2][I6][I8], and the LUT is... (3) The index range is [I0][I4][I5][I7]. Figure 4 In the middle, only LUTs are available. (1) The index region contains {I1, I3} and contains only LUTs. (2) The index region contains {I2, I6, I8} and contains only LUTs. (3)The index region contains {I5, I7}, therefore, the index regions corresponding to each lookup table can complement each other to expand the receptive field. Of course, other complementary index lookup tables can also be used; different lookup tables can complement each other, and this is not a limitation here. After determining multiple complementary index regions, these index regions can be input into a pre-trained neural network model, which will then output the lookup table corresponding to each index region.

[0050] After determining the lookup tables for multiple complementary index regions, the average of the output results corresponding to the center point I0 in each lookup table can be calculated to obtain the desired result, i.e., the image patch corresponding to the center point I0. For example, see [link to relevant documentation]. Figure 4 The output result V0 corresponding to the center point I0 is (LUT) (1) [I0][I1][I3][I4]+LUT (2) [I0][I2][I6][I8]+LUT (3) [I0][I4][I5][I7]) / 3. When the input is [I0][I1][I3][I4], the index region corresponding to the output result can be equivalently composed of [I0][I1][I3][I4][I5][I6][I7][I8]. Since the index region corresponding to the output result of the lookup table increases, the receptive field of the lookup table also increases accordingly. At the same time, the dimension of the lookup table remains unchanged, thus increasing the receptive field of the lookup table without increasing its dimension. Specifically, when the input data is [I0][I1][I3][I4], the output result V0 can be found through the lookup table. For lookup tables of other dimensions, the method of increasing the receptive field of a 4-dimensional lookup table can be referred to, which will not be elaborated here.

[0051] In some embodiments, to further increase the receptive field of the lookup table without increasing its dimension, the index regions corresponding to each constructed lookup table can be rotated, and the final output result of the lookup table corresponding to the center point I0 can be determined based on the output results of the lookup table corresponding to the index region before rotation and the output results of the lookup table corresponding to the index region after rotation. Specifically, for Figure 4 Any lookup table in the LUT (1) For example, Figure 5 As shown in (A), we can first determine the output V corresponding to the center point I0 in the index region based on the lookup table with index range [I0][I1][I3][I4]. j (1). Then, as Figure 5 As shown in (B), Figure 5If the index region in (A) is rotated 90 degrees counterclockwise, the index region becomes [I0][I1][I9][I] 10 Then, based on the index range [I0][I1][I9][I], we can... 10 The lookup table is used to determine the output V corresponding to the center point I0 in the indexed region. j (2). Next, as... Figure 5 As shown in (C), Figure 5 If the index region in (B) is rotated 90 degrees counterclockwise, the index region becomes [I0][I9][I]. 11 ][I 12 Then, based on the index range [I0][I9][I], it can be determined that... 11 ][I 12 The lookup table is used to determine the output V corresponding to the center point I0 in the indexed region. j (3). Next, as... Figure 5 As shown in (D), Figure 5 If the index region in (C) is rotated 90 degrees counterclockwise, the index region becomes [I0][I3][I]. 12 ][I 13 Then, based on the index region [I0][I3][I], it can be determined that... 12 ][I 13 The lookup table is used to determine the output V corresponding to the center point I0 in the indexed region. j (4). Due to V j (2) and V j (1) Since they are not in the same computational region, for the sake of consistency, V can be... j (2) Rotate 90 degrees clockwise to obtain V j (2)′. Similarly, V can be... j (3) Rotate 180 degrees clockwise to obtain V j (3)′, and V j (4) Rotate 270 degrees clockwise to obtain V j (4)′. For example, such as Figure 6 As shown in (A), assume V j (3) is [1][2][3][4]. Rotating it 90 degrees clockwise, we can obtain the following: Figure 6 The result shown in (B); Figure 6 Based on (B), rotating it 90 degrees clockwise yields the following: Figure 6 The result shown in (C) is that, Figure 6 The result shown in (C) is the desired result V. j (3)'. Finally, V can be... j (1) V j(2)′、V j (3)′ and V j (4) Add them together, average them, and use the result as: when the input is [I0][I1][I3][I4], pass it through the LUT. (1) The output result obtained is V = (V j (1)+V j (2)′+V j (3)′+V j (4)′) / 4. For increasing the receptive field of other lookup tables, one can follow... Figure 5 The processing method described herein will not be repeated here. In this implementation, the rotation angle can be selected according to the actual situation and is not limited here. For example, a rotation of only 90 degrees, or only 180 degrees or 270 degrees, or a rotation of 90 degrees and 180 degrees, or a rotation of 180 degrees and 270 degrees, and so on, can be selected. Similarly, after determining the output results corresponding to the center point I0 in each lookup table, these results can be added together and averaged, and the calculated result can be used as the final required result. Furthermore, the association between the input and the output can be established in the lookup table to obtain the latest lookup table. For example, please refer to [link to relevant documentation]. Figure 4 When rotating each lookup table by 90 degrees, 180 degrees, and 270 degrees, the resulting equivalent index area can be as follows: Figure 4 The equivalent index region is shown in the diagram. In this way, the index region corresponding to the output results of the lookup table is further increased, thereby further increasing the receptive field of the lookup table.

[0052] The multi-level processing unit 320 based on table-valued re-indexing is mainly used to perform hierarchical processing on multiple lookup tables related to a certain center point I0, where the input of the k-th level lookup table is the output of the (k-1)-th level lookup table. This expands the equivalent receptive field by cascading the inputs and outputs of multiple lookup tables. For example, ... Figure 7 As shown, taking the cascading process of a two-level lookup table as an example, LUT (1) LUT represents the first lookup table. (2) This indicates a secondary lookup table. Figure 7 In the middle, LUT can be used (1) The index value in the LUT is used as the secondary LUT. (2) The index, this hierarchical indexing process can be represented as:

[0053] V = LUT (2) [LUT (1) [I * ]][LUT (1) [I * ]][LUT(1) [I * ]][LUT (1) [I * ]]

[0054] When a low-quality image is input into the multi-level processing unit 320 based on table-valued re-indexing, a high-quality image can be output through the hierarchical processing of the multi-level processing unit 320 based on table-valued re-indexing.

[0055] From the perspective of receptive field, this cascaded framework is similar in principle to cascading multiple convolutional layers in a deep convolutional neural network. However, unlike the cascading of convolutional layers in a CNN, due to the limitation of LUT size, the index of image data is usually stored in integer format. And as mentioned above... Figure 2 Before the conversion step from neural network to lookup table described in the previous section, training the neural network requires gradients of floating-point data type. To address the issues arising from this data type, this embodiment designs a quantization-aware re-indexing mechanism, introducing a hierarchical indexing pattern into the neural network training process. For example... Figure 8 As shown, LUT (1) The output value is quantized to an integer during the forward pass (i.e., in the direction pointed to by the dashed line pointing to Integer), while its gradient is preserved as a floating-point value during the backward pass (i.e., in the direction pointed to by the dashed line pointing to Float-Point). In this way, numerical quantization operations can be perceived and modeled during the hierarchical training process of the neural network, thereby compensating for the performance loss caused by numerical quantization.

[0056] In some embodiments, the multi-table collaborative processing unit 310 based on spatial complementary indexes and the multi-level processing unit 320 based on table-valued re-indexes can be used individually or in combination. For example, Figure 9 This demonstrates a process where both are used in combination. For example... Figure 9As shown, Multiple Lookup Tables (MULUTs) are abbreviated as MuLUTs. MuLUT-S / D / Y represent three LUTs with different pixel indexing modes. Block 1x1 indicates that the output image resolution is the same as the input image resolution, and Block 2x2 indicates that the output image resolution is twice that of the input image resolution. The neural networks corresponding to the three types of LUTs are all implemented by stacking several convolutional layers. The first layer of the convolutional network corresponding to the S-type LUT uses a standard convolution with a kernel size of 2x2, denoted as Conv2x2. The first layer of the convolutional network corresponding to the D-type LUT uses a dilated convolution with a kernel size of 2x2, denoted as Conv2x2-D2. The first layer of the convolutional network corresponding to the Y-type LUT uses a standard convolution with a kernel size of 1x4, denoted as Conv1x4. Since the Y-type indexing mode is not a conventional grid index, it cannot be directly called from existing function libraries for computation. Therefore, the computation of the corresponding convolutional network needs to be specially adapted. Figure 10 As shown, the input image is first divided into 3x3 image blocks using a sliding window method, i.e., an unfold operation. Then, within each resulting image block, the pixel coordinates corresponding to the Y-index pattern are selected, and the values ​​at those coordinates are rearranged into 1x4 image blocks. The image after splitting and rearranging can then be adapted to a convolutional network with a first layer of Conv1x4.

[0057] Continue reading Figure 9 ,exist Figure 9 First, three index ranges corresponding to multiple lookup tables associated with the center point I0 are selected. Then, the data from each index range is input into its respective convolutional network, and the output of each convolutional network is 1D data. Next, the outputs of all convolutional networks are merged into one, for example, by averaging. Then, this result is used as the center point, and the index ranges of multiple lookup tables associated with this result are determined. The data from each index range is input into its respective convolutional network, and the output of each convolutional network is data with the same dimension as the desired lookup table. Finally, this data is merged into one, and a relationship is established between it and an index range containing the center point I0, thus creating a new lookup table. This new lookup table has a larger receptive field than the previous lookup table, while maintaining the same dimension.

[0058] In some embodiments, when the input image pixels are arranged in a Bayer pixel arrangement, a sliding window with a step size of twice can be used to perform LUT lookups on the input within a 2x2 window, thereby avoiding the influence of the Bayer pixel arrangement. Then, multi-lookup table collaborative processing is performed using the image processing system 300 described above, and / or the values ​​of the first-stage lookup table are used as input for the second stage, achieving hierarchical processing. For example, as... Figure 11 As shown, the pixels of the red channel, green channel and blue channel can be extracted in a step size of 2. Then, the extracted pixels are processed by the image processing system 300 described above.

[0059] Next, based on the foregoing description, the image processing method provided in the embodiments of this application will be introduced.

[0060] For example, Figure 12 The flowchart of an image processing method is shown. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 12 The image processing method may include the following steps:

[0061] S1201. Obtain the first image to be processed.

[0062] In this embodiment, after the user inputs the image they wish to process, the first image to be processed can be obtained. Alternatively, the first image to be processed can be automatically retrieved from a user-defined storage area. The specific method depends on the actual situation.

[0063] S1202. For any pixel in the first image, N (N≥2) lookup tables associated with the pixel are used to determine the image block corresponding to the pixel. The image block includes one or more pixels. The index area of ​​each lookup table contains the pixel, and the index areas of different lookup tables are different and at least partially the same.

[0064] In this embodiment, for any pixel in the first image, multiple lookup tables associated with that pixel can be used to determine the corresponding image block. This image block includes one or more pixels. Furthermore, the index range of each lookup table includes the arbitrary pixel, and the index ranges of different first lookup tables are different but at least partially the same; that is, the index ranges of different lookup tables are complementary. For example, see [link to example]. Figure 4 Any pixel can be the center point I0, i.e., LUT (1) LUT (2) and LUT(3) This can be a lookup table associated with the center point I0. The method of determining the image block corresponding to any given pixel using multiple lookup tables associated with it can be found in the aforementioned process of "the multi-table collaborative processing unit 310 obtaining the image block corresponding to the center point I0 based on spatial complementary indexes," which will not be repeated here.

[0065] One possible implementation is to input the index region of each of the N lookup tables into the corresponding lookup table, resulting in N first sub-image blocks. Each first sub-image block contains one or more pixels. Then, the pixels at the same position in the N first sub-image blocks are summed and averaged to obtain the image block corresponding to any given pixel. Since the index region corresponding to the output result of the lookup table increases, the receptive field of the lookup table also increases accordingly. Simultaneously, the dimension of the lookup table remains unchanged, thus increasing the receptive field of the lookup table without increasing its dimension. See the aforementioned... Figure 4 The description will not be repeated here.

[0066] Furthermore, for the index region of any one of the N lookup tables, the index region can be rotated M times along a first direction, centered on any pixel, to obtain M index regions, where M ≥ 1. Then, based on the lookup table corresponding to each of the M index regions, the second sub-image blocks corresponding to each of the M index regions are determined, resulting in M ​​second sub-image blocks. Each second sub-image block includes one or more pixels. Next, based on the rotation angle corresponding to each of the M index regions, the second sub-image blocks corresponding to each of the M index regions are rotated by the same angle in the opposite direction to the first direction to obtain M third sub-image blocks. Finally, the pixels at the same position in the first sub-image block corresponding to the index region of any lookup table and the M third sub-image blocks are added together and averaged. The result is then used as the first sub-image block corresponding to the index region of any lookup table. Thus, by rotating the index region, the receptive field of the lookup table is further increased without increasing the dimension of the lookup table. See the aforementioned details. Figure 5 and Figure 6 The description will not be repeated here.

[0067] As another possible implementation, the output of the i-th lookup table in the N lookup tables becomes the input of the (i+1)-th lookup table, the input of the 1-th lookup table in the N lookup tables becomes the index region of the 1-th lookup table, and the output of the N-th lookup table in the N lookup tables becomes the image patch corresponding to any pixel. That is to say, the N lookup tables can be cascaded to increase the receptive field of the lookup tables.

[0068] As another possible implementation, the N lookup tables can be composed of lookup tables contained in M ​​lookup table sets, with each lookup table set containing multiple lookup tables. The input to each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, determined based on the output of the (j-1)-th lookup table set. The input to each lookup table in the 1-th lookup table set is the index region of the corresponding lookup table in the 1-th lookup table set, determined based on any of the aforementioned pixels. The output of the M-th lookup table set is the image patch corresponding to any of the aforementioned pixels. For example, this processing can be understood as the aforementioned... Figure 9 The processing steps are detailed in the preceding description and will not be repeated here.

[0069] S1203. Based on the image blocks corresponding to each pixel in the first image, a second image is obtained, wherein the resolution of the second image is higher than or equal to the resolution of the first image.

[0070] In this embodiment, after obtaining the image blocks corresponding to each pixel in the first image, these image blocks can be stitched together to obtain the second image. The resolution of the second image is higher than or equal to the resolution of the first image. In some embodiments, when stitching the image blocks, each image block can be placed at the position of its corresponding pixel. For example, if pixel 1 in the first image corresponds to image block 1, then image block 1 can be placed at the position of pixel 1.

[0071] S1204, Output the second image.

[0072] In this embodiment, after acquiring the second image, the second image can be output, for example, displayed on the device's screen.

[0073] Therefore, in image processing, multiple lookup tables are used collaboratively for each pixel, thereby expanding the receptive field and improving image processing performance while maintaining a linear increase in computational and storage requirements. In contrast, existing lookup table techniques utilizing spatial information use only a single lookup table, and the expansion of the receptive field leads to an exponential increase in storage requirements, which is limited by memory size and hardware cost.

[0074] Based on the methods in the above embodiments, this application also provides an image processing apparatus.

[0075] For example, Figure 13 An image processing apparatus is shown. For example... Figure 13As shown, the image processing apparatus 1300 may include an acquisition module 1301 and a processing module 1302. The acquisition module 1301 is used to acquire a first image. The processing module 1302 is used to, for any pixel in the first image, use N (N≥2) lookup tables associated with that pixel to determine the corresponding image block, where each image block includes one or more pixels. The index area of ​​each lookup table contains the pixel, and the index areas of different lookup tables are different but at least partially identical. The processing module 1302 is also used to obtain a second image based on the image blocks corresponding to each pixel in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and to output the second image.

[0076] In some embodiments, when the processing module 1302 determines the image block corresponding to any pixel using N lookup tables associated with any pixel, it specifically performs the following steps: inputting the index area of ​​each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, each of which includes multiple pixels; adding the pixels at the same position in the N first sub-image blocks and averaging them to obtain the image block corresponding to any pixel.

[0077] In some embodiments, when the processing module 1302 inputs the index region of each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, it specifically performs the following steps: for the index region of any one of the N lookup tables, rotate the index region of the arbitrary lookup table M times along the first direction with any pixel as the center to obtain M index regions, where M≥1; based on the lookup table corresponding to each of the M index regions, determine the second sub-image blocks corresponding to each of the M index regions to obtain M second sub-image blocks, each of which includes one or more pixels; based on the rotation angle corresponding to each of the M index regions, rotate the second sub-image blocks corresponding to each of the M index regions along the same angle in the opposite direction to the first direction to obtain M third sub-image blocks; add up and average the pixels at the same position in the first sub-image block and the M third sub-image blocks corresponding to the index region of the arbitrary lookup table, and use the calculation result as the first sub-image block corresponding to the index region of the arbitrary lookup table.

[0078] In some embodiments, the output of the i-th lookup table in the N lookup tables is the input of the (i+1)-th lookup table, the input of the 1-th lookup table in the N lookup tables is the index region of the 1-th lookup table, and the output of the N-th lookup table in the N lookup tables is the image block corresponding to any pixel.

[0079] In some embodiments, the N lookup tables are composed of lookup tables contained in M ​​lookup table sets, and each lookup table set includes multiple lookup tables. The input to each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, determined based on the output of the (j-1)-th lookup table set. The input to each lookup table in the 1-th lookup table set is the index region of the corresponding lookup table in the 1-th lookup table set, determined based on any single pixel. The output of the M-th lookup table set is the image patch corresponding to any single pixel.

[0080] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0081] In some embodiments, Figure 13 The acquisition module 1301 and processing module 1302 shown can both be implemented in software or in hardware. For example, the following will use... Figure 13 Taking the acquisition module 1301 shown as an example, the implementation method of the acquisition module 1301 is introduced. Similarly, Figure 13 The implementation of the processing module 1302 shown can be referred to Figure 13 The implementation of the acquisition module 1301 shown is illustrated.

[0082] As an example of a software functional unit, processing module 1302 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1302 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0083] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0084] As an example of a hardware functional unit, the processing module 1302 may include at least one computing device, such as a server. Alternatively, the processing module 1302 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0085] The processing module 1302 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing module 1302 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1302 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0086] It should be noted that, in other embodiments, the processing module 1302 can be used to perform... Figure 12 Any step in the method shown, Figure 13 The processing module 1302 shown can be used to perform the above. Figure 12 Any step in the methods provided. Figure 13 The steps implemented by the acquisition module 1301 and processing module 1302 shown can be specified as needed, and the above-mentioned steps can be implemented by the acquisition module 1301 and processing module 1302 respectively. Figure 12 The different steps in the method provided here are used to realize all the functions of the image processing device 1300.

[0087] This application also provides a computing device 1400. For example... Figure 14 As shown, the computing device 1400 includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402. The computing device 1400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1400.

[0088] Bus 1402 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 1404 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1404 may include a path for transmitting information between various components of the computing device 1400 (e.g., memory 1406, processor 1404, communication interface 1408).

[0089] The processor 1404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0090] The memory 1406 may include volatile memory, such as random access memory (RAM). The processor 144 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0091] The memory 1406 stores executable program code, and the processor 1404 executes the executable program code to implement the aforementioned functions. Figure 13The functions of the acquisition module 1301 and processing module 1302 shown are used to implement all or part of the steps of the method in the above embodiments. That is, the memory 146 stores instructions for executing all or part of the steps in the method of the above embodiments.

[0092] Alternatively, the memory 1406 may store executable code, which the processor 1404 executes to implement the functions of the aforementioned image processing apparatus 1300, thereby implementing all or part of the steps in the methods of the above embodiments. That is, the memory 1406 may store instructions for executing all or part of the steps in the methods of the above embodiments.

[0093] The communication interface 1403 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 140 and other devices or communication networks.

[0094] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0095] like Figure 15 As shown, the computing device cluster includes at least one computing device 1400. The memory 1406 of one or more computing devices 1400 in the computing device cluster may store the same instructions for performing all or part of the steps in the methods of the above embodiments.

[0096] In some possible implementations, the memory 1406 of one or more computing devices 1400 in the computing device cluster may also store partial instructions for performing all or part of the steps in the methods of the above embodiments. In other words, a combination of one or more computing devices 1400 can jointly execute instructions for performing all or part of the steps in the methods of the above embodiments.

[0097] It should be noted that the memory 1406 in different computing devices 1400 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the image processing device 1300. That is, the instructions stored in the memory 1406 of different computing devices 1400 can implement the aforementioned... Figure 13 The functions of one or more modules in the acquisition module 814 and processing module 820 shown.

[0098] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0099] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0100] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0101] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0102] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0103] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0104] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: Get the first image; For any pixel in the first image, an image block corresponding to the pixel is determined using N lookup tables associated with the pixel. The image block includes one or more pixels. Each lookup table's index area contains the pixel, and the index areas of different lookup tables are different but at least partially the same. The output of the i-th lookup table in the N lookup tables is the input of the (i+1)-th lookup table, the input of the 1-th lookup table in the N lookup tables is the index area of ​​the 1-th lookup table, and the output of the N-th lookup table in the N lookup tables is the image block corresponding to the pixel. Where N ≥ 2. The determination of the image block corresponding to the pixel using N lookup tables associated with the pixel specifically includes: For each of the N lookup tables, the index area of ​​the lookup table is input into the corresponding lookup table to obtain the first sub-image block; The index region is rotated M times along the first direction with any one of the pixels as the center to obtain M index regions; M≥1; Based on the lookup table corresponding to each of the M index regions, the second sub-image block corresponding to each of the M index regions is determined to obtain M second sub-image blocks; Based on the rotation angles corresponding to the M index regions, the second sub-image blocks corresponding to the M index regions are rotated by the same angle in the opposite direction to the first direction to obtain M third sub-image blocks; The pixels at the same position in the first sub-image block corresponding to the index region of any lookup table and the M third sub-image blocks are added together and averaged. The result is then used as the first sub-image block corresponding to the index region of any lookup table. The output of the Nth lookup table is the image block corresponding to any one of the pixels; A second image is obtained based on the image blocks corresponding to each pixel in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; Output the second image.

2. The method according to claim 1, characterized in that, The N lookup tables are composed of lookup tables contained in the M lookup table sets, and each lookup table set includes multiple lookup tables; Wherein, the input of each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, which is determined based on the output of the (j-1)-th lookup table set; The inputs to each lookup table in the first lookup table set are the index areas of the corresponding lookup table in the first lookup table set, determined based on any given pixel. The output of the Mth lookup table set is the image block corresponding to any given pixel.

3. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first image; The processing module is configured to, for any pixel in the first image, use N lookup tables associated with that pixel to determine the corresponding image block, wherein the image block includes one or more pixels, wherein the index area of ​​each lookup table contains the pixel, and the index areas of different lookup tables are different and at least partially the same; the output of the i-th lookup table in the N lookup tables is the input of the (i+1)-th lookup table, the input of the 1-th lookup table in the N lookup tables is the index area of ​​the 1-th lookup table, and the output of the N-th lookup table in the N lookup tables is the image block corresponding to the pixel; wherein, N≥2; The processing module is further configured to input the index region of each of the N lookup tables into the corresponding lookup table to obtain N first sub-image blocks, each of which includes one or more pixels; for any one of the N lookup tables, rotate the index region of that lookup table M times along a first direction with the arbitrary pixel as the center to obtain M index regions; M≥1; and based on the lookup table corresponding to each of the M index regions, determine the second sub-image block corresponding to each of the M index regions to obtain M... The second sub-image block; based on the rotation angle corresponding to each of the M index regions, the second sub-image blocks corresponding to each of the M index regions are rotated by the same angle in the opposite direction to the first direction to obtain M third sub-image blocks; the pixels at the same position in the first sub-image block corresponding to the index region of any lookup table and the M third sub-image blocks are added together and averaged, and the calculation result is used as the first sub-image block corresponding to the index region of any lookup table; the output of the Nth lookup table is the image block corresponding to any pixel. The processing module is further configured to obtain a second image based on the image blocks corresponding to each pixel in the first image, wherein the resolution of the second image is higher than or equal to the resolution of the first image; and to output the second image.

4. The apparatus according to claim 3, characterized in that, The N lookup tables are composed of lookup tables contained in the M lookup table sets, and each lookup table set includes multiple lookup tables; Wherein, the input of each lookup table in the j-th lookup table set is the index region of the corresponding lookup table in the j-th lookup table set, which is determined based on the output of the (j-1)-th lookup table set; The inputs to each lookup table in the first lookup table set are the index areas of the corresponding lookup table in the first lookup table set, determined based on any given pixel. The output of the Mth lookup table set is the image block corresponding to any given pixel.

5. A computing device, characterized in that, include: At least one memory for storing programs; At least one processor for executing the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the method as described in any one of claims 1-2.

6. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-2.

7. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as described in any one of claims 1-2.

8. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-2.

Images