Image processing method and apparatus, and non-transitory computer-readable storage medium

Abstract

This disclosure provides an image processing method for an edge device, comprising: extracting features from an input image at a first scale to generate a plurality of first initial feature maps at the first scale; performing a first processing or a second processing on the plurality of first initial feature maps to generate an output image at a second scale, wherein the second scale is larger than the first scale; the first processing includes: dividing the plurality of first initial feature maps into a plurality of first initial feature map groups; for each first initial feature map group, performing pixel rearrangement on the plurality of first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group, wherein the first feature map has a second scale; and processing the first feature maps corresponding to the plurality of first initial feature map groups to generate an output image. This disclosure also provides an image processing apparatus and a non-transient computer-readable storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of display, in particular to an image processing method, an image processing device and a non-transitory computer readable storage medium. BACKGROUND

[0002] Super Resolution (SR) reconstruction of images has become an important branch in the field of image processing. At present, the methods of image super resolution reconstruction mainly include interpolation-based methods and machine learning-based methods. SUMMARY

[0003] The present disclosure provides an image processing method, an image processing device and a non-transitory computer readable storage medium.

[0004] In a first aspect, the present disclosure provides an image processing method for an edge device, comprising:

[0005] performing feature extraction on an input image of a first scale to generate a plurality of first initial feature maps of the first scale;

[0006] performing first processing or second processing on the plurality of first initial feature maps to generate an output image of a second scale, the second scale being greater than the first scale;

[0007] The first processing comprises:

[0008] dividing the plurality of first initial feature maps into a plurality of first initial feature map groups, each of the first initial feature map groups comprising a plurality of first initial feature maps, and for each of the first initial feature map groups, performing pixel rearrangement on the plurality of first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group, the first feature map after pixel rearrangement having the second scale;

[0009] processing the first feature maps corresponding to the plurality of first initial feature map groups by using a preset first normalization rule to generate the output image;

[0010] The second processing comprises:

[0011] processing the plurality of first initial feature maps by using a preset second normalization rule to generate a plurality of fourth feature maps of the first scale;

[0012] performing pixel rearrangement on the plurality of fourth feature maps to obtain the output image.

[0013] In some embodiments, when the image processing method comprises performing the first processing on the plurality of first initial feature maps, a preset first normalization rule is used to process the first feature maps corresponding to the plurality of first initial feature map groups to generate the output image, specifically comprising:

[0014] A plurality of feature element groups are formed based on the feature elements at the corresponding positions in the plurality of first feature maps;

[0015] The output image is generated based on the maximum element value in each feature element group.

[0016] In some embodiments, when the image processing method comprises performing the first processing on the plurality of first initial feature maps, the first processing further comprises:

[0017] The input image is subjected to convolution processing to generate a plurality of second initial feature map groups and a plurality of third initial feature map groups, each of the second initial feature map groups comprising a plurality of second initial feature maps, and each of the third initial feature map groups comprising a plurality of third initial feature maps, each of the second initial feature maps and each of the third initial feature maps having the first scale;

[0018] For any one of the second initial feature map groups, the plurality of second initial feature maps in the second initial feature map group are subjected to pixel rearrangement to generate the second feature maps corresponding to the second initial feature map group;

[0019] For any one of the third initial feature map groups, the plurality of third initial feature maps in the third initial feature map group are subjected to pixel rearrangement to generate the third feature maps corresponding to the third initial feature map group;

[0020] A preset first normalization rule is used to process the first feature maps corresponding to the plurality of first initial feature map groups to generate the output image, specifically comprising:

[0021] A plurality of mask maps are generated based on the plurality of second feature maps and the plurality of third feature maps;

[0022] A plurality of first intermediate feature maps having a second scale are generated based on the plurality of mask maps and the plurality of first feature maps;

[0023] The output image is synthesized based on the plurality of first intermediate feature maps.

[0024] In some embodiments, the second feature maps and the third feature maps correspond to each other one by one, and each of the second initial feature map groups and the corresponding third initial feature map groups constitutes a feature map pair;

[0025] Multiple mask maps are generated based on multiple second feature maps and multiple third feature maps, specifically including:

[0026] For any feature map pair, the second and third feature maps in the feature map pair are multiplied to generate a second intermediate feature map corresponding to the feature map pair.

[0027] The normalization function is used to normalize multiple second intermediate feature maps to generate multiple mask maps.

[0028] In some embodiments, the mask map corresponds one-to-one with the first feature map, and each mask map represents the feature weight of the corresponding first feature map;

[0029] Based on the plurality of mask images and the plurality of first feature images, a plurality of first intermediate feature images with a second scale are generated, specifically including:

[0030] Based on the feature weights represented by each mask image, the corresponding first feature image is weighted to generate multiple first intermediate feature images.

[0031] In some embodiments, the feature elements of each second initial feature map and each third initial feature map are arranged in M ​​rows and N columns, where M and N are both positive integers greater than 1; both the second feature map and the third feature map include M rows and N columns of map blocks;

[0032] The pixels of multiple second initial feature maps in the second initial feature map group are rearranged to generate the second feature map corresponding to the second initial feature map group. Specifically, this includes: selecting the feature element of each of the multiple second initial feature maps in the i-th row and j-th column, and arranging them into a patch in the i-th row and j-th column of the second feature map; i takes a positive integer from 1 to M, and j takes a positive integer from 1 to N.

[0033] The pixels of multiple third initial feature maps in the third initial feature map group are rearranged to generate the third feature map corresponding to the third initial feature map group. Specifically, this includes: selecting the feature element of each of the multiple third initial feature maps in the i-th row and j-th column, and arranging them into a block in the i-th row and j-th column of the third feature map.

[0034] In some embodiments, the feature elements of each first initial feature map are arranged in P rows and Q columns, where P and Q are both positive integers greater than 1; the first feature map includes a P row and Q column block;

[0035] Pixel rearrangement is performed on multiple first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group, specifically including:

[0036] Select the feature element in row p and column q of each of the multiple first initial feature maps, and arrange them into a block in row p and column q of the first feature map; p takes a positive integer from 1 to P, and q takes a positive integer from 1 to Q.

[0037] In some embodiments, when the image processing method includes performing a second processing on a plurality of the first initial feature maps, the number of the first initial feature maps is d. V d V It is an integer greater than 1;

[0038] The second process further includes: performing convolution processing on the input image to generate multiple fifth initial feature maps and multiple sixth initial feature maps;

[0039] The first initial feature maps are processed using a preset second normalization rule to generate multiple fourth feature maps with a first scale, specifically including:

[0040] Based on multiple fifth initial feature maps and multiple sixth initial feature maps, multiple weighted feature map groups are generated. Each weighted feature map group corresponds one-to-one with the fourth feature map, and each weighted feature map group includes d. V Each weighted feature map;

[0041] For any one of the weighted feature map groups, a fourth feature map corresponding to the weighted feature map group is generated based on the weighted feature map group and multiple first initial feature maps.

[0042] In some embodiments, the magnification ratio of the second scale to the first scale is s.

[0043] Based on the multiple fifth initial feature maps and the multiple sixth initial feature maps, multiple weighted feature map groups are generated, specifically including:

[0044] A third matrix is ​​generated by multiplying a first matrix composed of multiple fifth initial feature maps and a second matrix composed of multiple sixth initial feature maps; wherein, the first matrix includes a matrix arranged in s... 2 line d K The first matrix element of the column, each of the first matrix elements being one of the fifth initial feature maps; the second matrix includes elements arranged in d... K line d V The second matrix element comprises a column, each of which is a sixth initial feature map; the third matrix comprises multiple third matrix elements, each of which is an initial weighted feature map; d K s and s are both integers greater than 1;

[0045] The multiple initial weight feature maps in the third matrix are normalized using a normalization function to generate multiple weight feature maps, wherein each weight feature map corresponds one-to-one with an initial weight feature map, and the weight feature maps corresponding to multiple initial weight feature maps located in the same row in the third matrix form a weight feature map group.

[0046] In some embodiments, within the same weighted feature map group, each weighted feature map corresponds to one first initial feature map, and different weighted feature maps correspond to different first initial feature maps;

[0047] Based on the weighted feature map group and multiple first initial feature maps, a fourth feature map corresponding to the weighted feature map group is generated, specifically including:

[0048] Each weighted feature map in the weighted feature map group is multiplied with its corresponding first initial feature map to generate multiple third intermediate feature maps.

[0049] Multiple third intermediate feature maps are superimposed to generate the fourth feature map corresponding to the weighted feature map group.

[0050] In some embodiments, when the image processing method includes performing a second processing on a plurality of first initial feature maps, the feature elements in the fourth feature map are arranged in H rows and K columns, where H and K are both positive integers greater than 1; the output image includes an image block of H rows and K columns.

[0051] The output image is obtained by pixel rearranging of multiple fourth feature maps, specifically including:

[0052] Select the feature element in the h-th row and k-th column of each of the fourth feature maps, and arrange them into a patch in the h-th row and k-th column of the output image; h takes a positive integer from 1 to H, and k takes a positive integer from 1 to K.

[0053] In some embodiments, a trained image processing model is used to perform the steps of the image processing method.

[0054] In some embodiments, the image processing model is trained through the following steps:

[0055] The degraded image generated based on the reference image is input into the image processing model to generate the training output image;

[0056] Based on the training output image and the reference image, the loss value of the image processing model is determined according to a predetermined loss function;

[0057] The parameters of the image processing model are adjusted based on the loss value of the image processing model.

[0058] Secondly, this disclosure also provides an image processing apparatus, comprising:

[0059] The image acquisition module is configured to acquire the input image at the first scale;

[0060] The image processing module is configured to extract features from the input image to generate multiple first initial feature maps with the first scale; and to perform a first processing or a second processing on the multiple first initial feature maps to generate an output image with a second scale, wherein the second scale is larger than the first scale.

[0061] The first process includes:

[0062] The plurality of first initial feature maps are divided into a plurality of first initial feature map groups, each first initial feature map group including a plurality of first initial feature maps. For each first initial feature map group, the plurality of first initial feature maps in the first initial feature map group are pixel rearranged to generate a first feature map corresponding to the first initial feature map group. The first feature map after pixel rearrangement has the second scale.

[0063] The first feature maps corresponding to multiple first initial feature map groups are processed using a preset first normalization rule to generate the output image;

[0064] The second process includes:

[0065] Multiple first initial feature maps are processed using a preset second normalization rule to generate multiple fourth feature maps with a first scale;

[0066] The output image is obtained by rearranging the pixels of multiple fourth feature maps.

[0067] Thirdly, this disclosure also provides an image processing apparatus, comprising:

[0068] The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the image processing method described above.

[0069] Fourthly, this disclosure also provides a non-transient computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described image processing method. Attached Figure Description

[0070] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0071] Figure 1A This is a flowchart of an image processing method provided in some embodiments of this disclosure.

[0072] Figure 1A This is a schematic diagram illustrating an optional approach to step S2 provided in some embodiments of this disclosure.

[0073] Figure 1A This is a schematic diagram illustrating an alternative approach to step S2 provided in some other embodiments of this disclosure.

[0074] Figure 2 This is a schematic diagram of an image processing model provided in some embodiments of this disclosure.

[0075] Figure 3 This is a schematic diagram of the pixel rearrangement process provided in some embodiments of this disclosure.

[0076] Figure 4 This is a schematic diagram of an image processing model provided in some other embodiments of this disclosure.

[0077] Figure 5 This is a schematic diagram illustrating the normalization operation of multiple second intermediate feature maps provided in some embodiments of this disclosure.

[0078] Figure 6 This is a schematic diagram of an image processing model provided in some further embodiments of the present disclosure.

[0079] Figure 7 A flowchart illustrating the training steps of an image processing model provided for some embodiments of this disclosure.

[0080] Figure 8 The processing time versus output quality curves for various image processing methods are shown.

[0081] Figure 9 A schematic diagram of the image processing model for executing the EdgeSR method.

[0082] Figure 10 This is a schematic block diagram of an image processing apparatus provided for some embodiments of the present disclosure.

[0083] Figure 11 A schematic block diagram of an image processing apparatus provided for other embodiments of this disclosure. Detailed Implementation

[0084] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0085] Super-resolution (SR) image reconstruction has become an important branch of image processing. Image super-resolution refers to the technique of reconstructing low-resolution (LR) images into high-resolution (HR) images using image processing methods. Currently, image super-resolution reconstruction methods are mainly based on interpolation and machine learning. Interpolation methods upsample the image by inserting new elements between pixels in the original image using appropriate interpolation algorithms, such as nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation. These methods do not provide much information about the original image, thus inevitably affecting the quality of the reconstructed image. Machine learning-based methods typically achieve image upsampling through machine learning models. These methods can learn more features about the original image. However, currently used machine learning models usually have complex structures, thus requiring significant computational resources and resulting in slow processing speeds.

[0086] This disclosure provides an image processing method for edge devices. An edge device refers to a device with computing and network resources located between a data source and a cloud service. For example, a user terminal device can be an edge device between a person and a cloud service, while a gateway can be an edge device between a smart home and a cloud center. Edge devices can communicate with cloud services via a network. Compared to the cloud, edge devices can provide faster, more real-time responses and higher information security; however, their storage capacity, computing power, and available energy (such as electricity) are often very limited, making them unsuitable for performing overly complex computational processing.

[0087] Edge devices can be computers, mobile phones, or any edge device or combination of edge devices that can be used to perform the image reconstruction technology solutions provided in this disclosure. Edge devices can include user terminal devices and Internet of Things (IoT) devices. End-user devices can include, for example, desktop computers, laptops, tablets, mobile phones and other mobile devices, as well as wearable devices (e.g., smartwatches, smart glasses, headphones). IoT devices can include any device capable of participating in and / or communicating with Internet of Things (IoT) systems or networks. For example, various devices in vehicles (such as navigation systems, autonomous driving systems, etc.), equipment, facilities and / or infrastructure associated with industrial manufacturing and production, and smart entertainment systems (e.g., televisions, audio systems, video game systems), smart home or office systems, security systems, etc.

[0088] Edge devices can be used to implement artificial intelligence algorithms, technologies, and products. According to embodiments of this disclosure, the edge device is configured to perform image processing on an input image, which can be super-resolution reconstruction to generate a scaled-up output image.

[0089] The image processing method in this embodiment can be used to perform super-resolution reconstruction on an input image, thereby generating a high-resolution output image based on a low-resolution input image. Figure 1A Here are flowcharts of image processing methods provided in some embodiments of this disclosure, such as... Figure 1A As shown, the image processing method includes:

[0090] S1. Extract features from the input image at the first scale to generate multiple initial feature maps at the first scale.

[0091] In some embodiments, the input image may be a low-resolution image acquired by the edge device from other devices or local memory, and may contain various types of images, such as landscape images, portrait images, building images, etc. In some embodiments, the input image may be a single-channel grayscale image. Alternatively, the input image may also be an image of the color channels of a color image; for example, for a color image containing YUV color channels, the input image may be an image of the Y channel (i.e., the luminance channel). Exemplarily, the edge device may directly extract the luminance channel from the multi-channel image to be reconstructed. For example, the luminance channel may be a channel representing luminance contained in an image encoded in color spaces such as YUV, YCbCr, Lab, etc., such as the Y channel in YUV and YCbCr and the L channel in Lab. Of course, the edge device may also acquire the luminance channel of an image in other ways.

[0092] In some embodiments, the input image can be convolved using a convolutional layer to generate a first initial feature map. The first initial feature map can be a low-level feature map; in one example, the low-level feature map is the feature map obtained after convolving the input image through a single convolutional layer.

[0093] It should be noted that the input image, output image, and each feature map in this disclosure are all represented in matrix form, and the matrix includes multiple feature elements. The scale of each image can be regarded as the number of rows and columns of the feature elements in the matrix.

[0094] S2. Perform a first processing or a second processing on multiple first initial feature maps to generate an output image with a second scale, wherein the second scale is larger than the first scale.

[0095] In some embodiments, the second scale is obtained by scaling up the first scale according to the target scaling factor. For example, in a scenario of reconstructing a 4K image based on a 1080P image, the first scale can be 1920*1080, and the second scale can be 3840*2160.

[0096] Figure 1A This is a schematic diagram illustrating an optional approach to step S2 provided in some embodiments of this disclosure. Figure 1A This illustrates a scenario where a first process is performed on multiple initial feature maps. When performing the first process on multiple initial feature maps, the multiple initial feature maps are divided into multiple groups of initial feature maps, and each group of initial feature maps includes multiple initial feature maps. For example... Figure 1A As shown, step S2 may include:

[0097] S21a. For each first initial feature map group, the pixels of the multiple first initial feature maps in the first initial feature map group are rearranged to generate a first feature map corresponding to the first initial feature map group. The first feature map has a second scale, which is larger than the first scale. In some embodiments, the feature element values ​​in the first feature map are obtained by arranging and combining the feature element values ​​of the multiple first initial feature maps in the corresponding first initial feature map group. In some embodiments, the number of first initial feature maps included in the first initial feature map group depends on the scaling factor between the second scale and the first scale. For example, if the first scale is H*W and the second scale is sH*sW, the first initial feature map group includes s*s first initial feature maps.

[0098] S21b: Process the first feature maps corresponding to multiple first initial feature map groups using a preset first normalization rule to generate an output image with a second scale.

[0099] In step S21b, each element in the output image is generated based on the feature element values ​​of multiple first feature maps. For example, the first normalization rule could be: weighted summation of feature element values ​​at the same position in multiple first feature maps to obtain the element value at the corresponding position in the output image; that is, weighted summation of the feature elements in the i-th row and j-th column of each first feature map to obtain the element value in the i-th row and j-th column of the output image. Another example is averaging the feature element values ​​at the same position in multiple first feature maps to obtain the element value at the corresponding position in the output image; that is, averaging the feature elements in the i-th row and j-th column of each first feature map to obtain the element value in the i-th row and j-th column of the output image. Yet another example is using the maximum value among the feature elements at the same position in multiple first feature maps as the element value at the corresponding position in the output image; that is, the element value in the i-th row and j-th column of the output image is the maximum value among the feature elements in the i-th row and j-th column of each first feature map. i takes integers from 1 to M; j takes integers from 1 to N. M is the number of rows of the feature elements in the first feature map, and N is the number of columns of the feature elements in the first feature map.

[0100] Figure 1A This is a schematic diagram illustrating alternative methods for step S2 provided in other embodiments of this disclosure. Figure 1A This illustrates a scenario where a second process is applied to multiple first initial feature maps, such as... Figure 1A As shown, step S2 may include:

[0101] S22a. The multiple first initial feature maps are processed using a preset second normalization rule to generate multiple fourth feature maps with a first scale.

[0102] S22b: The output image is obtained by rearranging the pixels of multiple fourth feature maps.

[0103] In this embodiment, steps S1 to S2 do not require a complex neural network structure to be implemented; therefore, the image processing method can be executed using widely deployed edge devices. Furthermore, by adjusting the parameters of the neural network, the image processing speed can be improved while maintaining the quality of the output image. The parameters of the neural network may include: the size of the convolution kernel when convolving the input image, and the number of first initial feature maps.

[0104] Figure 2 This diagram illustrates an image processing model provided in some embodiments of the present disclosure. This image processing model is used to execute steps S1 and S21a-S21b of the image processing method described above. This image processing model can be implemented using a convolutional neural network. Figure 2As shown, the image processing model includes: a first feature extraction module 11, a first pixel rearrangement module 12, and a first processing module 13. The first feature extraction module 11 is used to extract features from the input image INP at a first scale to generate multiple first initial feature map sets, each first initial feature map set including multiple first initial feature maps. Figure 2 As shown, the first feature extraction module 11 includes C*s 2 The first feature channel, C*s 2 The first feature channel, such as Figure 2 As shown to Wherein, C*s 2 The first feature channels are divided into C first channel groups, and each first channel group includes s 2 There are multiple first feature channels. Here, s is the magnification factor of the second scale relative to the first scale; that is, when the first scale is H*W, the second scale is sH*sW. The multiple first feature channels of the first feature extraction module 11 are used to convolve the input image INP to generate C*s. 2 A first initial feature map; the first initial feature maps generated from the same first channel group constitute a first initial feature map group. In some embodiments, for each first feature channel, only a single convolution kernel may be used to convolve the input image. That is, Figure 2 In to Corresponding to C*s 2 There is a first convolution kernel with a size of k×k and a convolution stride of, for example, 1×1.

[0105] The values ​​of C and k can be selected according to actual needs.

[0106] The first pixel rearrangement module 12 is used to perform the above step S21a, that is, for each first initial feature map group, the first pixel rearrangement module 12 rearranges the pixels of the multiple first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group. The multiple first feature maps generated by the first pixel rearrangement module 12 based on the multiple first initial feature map groups are respectively as follows: Figure 2 v1~v C As shown. In some embodiments, the feature elements of each first initial feature map are arranged in P rows and Q columns, and the first feature map includes a P row and Q column block, where P and Q are both positive integers greater than 1. Step S21a may specifically include: selecting multiple first initial feature maps v1 to v2. C The feature elements in the p-th row and q-th column of each element are arranged into a patch in the p-th row and q-th column of the first feature map.

[0107] Figure 3This is a schematic diagram of the pixel rearrangement process provided in some embodiments of this disclosure, such as... Figure 3 As shown, the four elements a1, a2, a3, and a4 in the first row and first column of the first initial feature maps f1, f2, f3, and f4 are used to form the first patch in the first feature map. These four elements are sequentially arranged into the four element positions contained in the first patch. Similarly, the elements in the i-th row and j-th column of each first initial feature map in the first initial feature map group are used to form patches at the corresponding positions in the first feature map. These elements are sequentially arranged into the element positions contained in the patch.

[0108] It should be noted that, Figure 3 The pixel rearrangement process is only illustrated exemplarily, and the pixel rearrangement process described above can also be performed in other ways as needed.

[0109] In some embodiments, the first processing module 13 is used to perform step S21b, wherein step S21b may specifically include: forming multiple feature element groups based on multiple feature elements at the same position in multiple first feature maps, and generating an output image OUTP based on the maximum element value in each feature element group.

[0110] For example, each first feature map includes X rows and Y columns of feature elements, where X and Y are both integers greater than 1; the feature elements in the x-th row and y-th column of each first feature map are grouped into a feature element group, and the maximum value of the feature element in the feature element group is used as the value of the feature element in the x-th row and y-th column of the output image OUTP. x and y are both positive integers, and 0 < x ≤ X, 0 < y ≤ Y. For example, the first row and first column feature elements of each first feature map form the first feature element group, and the maximum element value in this feature element group is used as the value of the first row and first column feature element in the output image OUTP; the first row and second column feature elements of each first feature map form the second feature element group, and the maximum element value in this feature element group is used as the value of the first row and second column feature element in the output image OUTP; the first row and third column feature elements of each first feature map form the third feature element group, and the maximum element value in this feature element group is used as the value of the first row and third column feature element in the output image OUTP; and so on, the X row and Y column feature elements of each first feature map form the X*Yth feature element group, and the maximum element value in this feature element group is used as the value of the X row and Y column feature element in the output image OUTP.

[0111] Figure 4 This is a schematic diagram of an image processing model provided in some other embodiments of the present disclosure. This image processing model is used to perform steps S1, S21a to S21b in the above-described image processing method. Figure 4As shown, this image processing model can be implemented using a convolutional neural network. The image processing model includes: a second feature extraction module 21, a second pixel rearrangement module 22, and a second processing module 23.

[0112] In some embodiments, the second feature extraction module 21 is used to extract features from the input image INP at the first scale to generate a plurality of first initial feature map groups, a plurality of second initial feature map groups and a plurality of third initial feature map groups. Each first initial feature map group includes a plurality of first initial feature maps, each second initial feature map group includes a plurality of second initial feature maps, and each third initial feature map group includes a plurality of third initial feature maps.

[0113] The second feature extraction module 21 may include: C*s 2 The first feature channel, C*s 2 The first feature channel, such as Figure 4 As shown to Wherein, C*s 2 The first feature channels are divided into C first channel groups, and each first channel group includes s 2 There are multiple first feature channels. Here, s is the magnification factor of the second scale relative to the first scale; that is, when the first scale is H*W, the second scale is sH*sW. These multiple first feature channels are used to convolve the input image separately to generate C*s. 2 A first initial feature map; the first initial feature maps generated from the same first channel group constitute a first initial feature map group. In some embodiments, for each first feature channel, only a single convolution kernel may be used to convolve the input image. That is, Figure 2 In to Corresponding to C*s 2 There is a first convolution kernel with a size of k×k and a convolution stride of, for example, 1×1.

[0114] In addition, the second feature extraction module 21 may also include: C*s 2 The second feature channel, and C*s 2 The third feature channel. C*s 2 The second feature channel, such as Figure 4 As shown to C*s 2 A third feature channel, such as Figure 4 As shown to Wherein, C*s 2 The second feature channels are divided into C second channel groups, and each second channel group includes s 2Multiple second feature channels are used to convolve the input image INP separately to generate C*s. 2 A second initial feature map; second initial feature maps generated from the same second channel group constitute a second initial feature map group. C*s 2 The three third feature channels are divided into C third channel groups, and each third channel group includes s 2 Multiple third feature channels are used to convolve the input image INP separately to generate C*s. 2 A third initial feature map; third initial feature maps generated from the same third channel group constitute a third initial feature map group. In some embodiments, for each second feature channel, only a single convolution kernel may be used to convolve the input image. That is, Figure 4 In to Corresponding to C*s 2 A second convolutional kernel, with a size of k×k, and a stride of, for example, 1×1. For each third feature channel, the input image can be convolved using only a single kernel. That is, Figure 4 In to Corresponding to C*s 2 There is a third convolution kernel, the size of which is k×k, and the convolution stride is, for example, 1×1.

[0115] The second pixel rearrangement module 22 is used to rearrange the pixels of multiple first initial feature maps in each first initial feature map group, to rearrange the pixels of second initial feature maps in each second initial feature map group, and to rearrange the pixels of third initial feature maps in each third initial feature map group. Specifically, for any first initial feature map group, the second pixel rearrangement module 22 can rearrange the pixels of multiple first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group; for any second initial feature map group, the second pixel rearrangement module 22 can rearrange the pixels of multiple second initial feature maps in the second initial feature map group to generate a second feature map corresponding to the second initial feature map group; for any third initial feature map group, the second pixel rearrangement module 22 can rearrange the pixels of multiple third initial feature maps in the third initial feature map group to generate a third feature map corresponding to the third initial feature map group. The first feature map, second feature map, and third feature map all have a second scale. The first feature map generated by the second pixel rearrangement module 22 is shown below. Figure 4 v1~v C As shown, the generated second feature map is as follows Figure 4 q1~q C As shown, the generated third feature map is as follows Figure 4k1~k C As shown.

[0116] The process of pixel rearranging of multiple first initial feature maps in the first initial feature map group is described above and will not be repeated here. The process of pixel rearranging of multiple second initial feature maps in the second initial feature map group specifically includes: selecting the feature element in the i-th row and j-th column of each of the multiple second initial feature maps, and arranging them into a patch in the i-th row and j-th column of the second feature map; i is a positive integer from 1 to M, and j is a positive integer from 1 to N. The process of pixel rearranging of multiple third initial feature maps in the third initial feature map group specifically includes: selecting the feature element in the i-th row and j-th column of each of the multiple third initial feature maps, and arranging them into a patch in the i-th row and j-th column of the third feature map. The principles of pixel rearranging of multiple second initial feature maps in the second initial feature map group and multiple third initial feature maps in the third initial feature map group can be found above. Figure 3 The description will not be repeated here.

[0117] The second processing module 23 may specifically include a mask image generation submodule 231 and an output submodule 232, wherein the mask image generation submodule 231 may be configured to: based on multiple second feature images q1~q C and multiple third feature maps k1~k C Generate multiple mask images m1 to m C Each mask map corresponds to a first feature map, and different mask maps correspond to different first feature maps. Each mask map represents the feature weight of the corresponding first feature map. The feature weight can be used to indicate the importance of a feature to image processing. The larger the weight, the higher the importance of the feature. Optionally, the feature weight can be associated with the sensitivity of human vision to the feature in the image. The larger the weight, the higher the visual sensitivity of the feature, and thus it is more enhanced during image processing. In some embodiments, the mask map can be pixel-level. Each mask map has the same second scale as the first feature map and contains multiple feature weights. In one example, each element in the mask map represents the feature weight of the feature element at the corresponding position in its corresponding first feature map. For example, the scale of the first feature map and its corresponding mask map are both sH*sW, that is, both the first feature map and the mask map include sH*sW feature elements, and the element in the c-th row and d-th column of the mask map is the feature weight of the feature element in the c-th row and d-th column of the first feature map, where c is an integer greater than 0 and less than or equal to sH, and d is an integer greater than 0 and less than or equal to d.

[0118] In some embodiments, the first feature map is a value feature map used in the self-attention mechanism, the second feature map is a query feature map used in the self-attention mechanism, and the third feature map is a key feature map used in the self-attention mechanism. There is a one-to-one correspondence between the second and third feature maps, and each second feature map and its corresponding third feature map form a feature map pair. The specific process of the mask map generation submodule 231 in generating the mask map may include: for any feature map pair, multiplying the second and third feature maps in the pair to generate a second intermediate feature map corresponding to that feature map pair. Then, a normalization function (e.g., the SOFTMAX function) is used to normalize multiple second intermediate feature maps to generate multiple mask maps.

[0119] For example, multiplying the second feature map q1 with the third feature map k1 yields a second intermediate feature map corresponding to the second feature map q1; then multiplying the second feature map q1 with the third feature map k1 yields a second intermediate feature map. C With the third feature map k C Perform a multiplication operation to obtain the second feature map q. C The corresponding second intermediate feature map. Specifically, the multiplication operation between the second and third feature maps can be performed by multiplying each feature element in the second feature map by the corresponding feature element in the third feature map. That is, multiplying the feature element in the i-th row and j-th column of the second feature map by the feature element in the i-th row and j-th column of the third feature map, and using this product as the value of the feature element in the i-th row and j-th column of the second intermediate feature map.

[0120] Figure 5 This is a schematic diagram illustrating the normalization operation of multiple second intermediate feature maps provided in some embodiments of this disclosure. For the sake of simplicity, Figure 5 Only four 2x2 second intermediate feature maps m1 are shown. ’ m2 ’ m3 ’ m4 ’ .like Figure 5 As shown, feature element groups can be formed based on multiple feature elements at corresponding positions in multiple second intermediate feature maps. For example, a feature element group can be formed using a feature element at the p-th row and q-th column in the second intermediate feature map (such as...). Figure 5 The four feature elements m1 (located in the upper left corner as shown) ’ (p, q) = a1, m2 ’ (p, q) = a2, m3 ’ (p, q) = a3, m4 ’(p, q) = a4 forms a feature element group (a1, a2, a3, a4) corresponding to the p-th row and q-th column of the image position, where p = 1, 2, 3, 4; q = 1, 2, 3, 4. Subsequently, normalization can be performed on the feature elements in the feature element group to obtain a normalized feature element group (b1, b2, b3, b4). For example, it can be done as follows... Figure 5 As shown, the feature element group (a1, a2, a3, a4) is input as a 4-dimensional vector into the SOFTMAX module. The SOFTMAX function is used to normalize each vector element to generate a normalized feature element group. Each element in the feature element group (b1, b2, b3, b4) is used as the element at the corresponding position (i.e., row p, column q) in the mask images m1, m2, m3, m4. For example, the four feature elements at position p, column q in the mask images m1, m2, m3, m4 are respectively set as normalized feature elements b1, b2, b3, and b4.

[0121] Please continue reading. Figure 4 The output submodule 232 is configured to output based on multiple mask images m1 to m2. C and multiple first feature maps v1~v C Multiple first intermediate feature maps with a second scale are generated; then, the output image OUTP is synthesized based on the multiple first intermediate feature maps.

[0122] In some embodiments, based on multiple mask images m1 to m C and multiple first feature maps v1~v C The specific process of generating multiple first intermediate feature maps with a second scale includes: weighting the corresponding first feature map based on the feature weights represented by each mask map to generate multiple first intermediate feature maps. That is, each mask map is multiplied by its corresponding first feature map. For example, as... Figure 4 As shown, the first feature map v1 is multiplied by the corresponding mask map m1, and the first feature map v... C With the corresponding mask image m C Multiplication. Here, multiplication can refer to multiplying each element of the first feature map with the corresponding element at the corresponding position in the corresponding mask map. In other words, it is to weight the corresponding elements at the same position in the first feature map based on the feature weights in the mask map.

[0123] In some embodiments, synthesizing an output image based on multiple first intermediate feature maps can specifically involve directly adding the feature elements at corresponding positions in the multiple first intermediate feature maps to obtain the output image.

[0124] exist Figure 4In the image processing model shown, by assigning different weights to different first feature maps, the feature elements of important channels are enhanced, thereby improving the quality of the output image.

[0125] Figure 6 This is a schematic diagram of an image processing model provided in some embodiments of the present disclosure, which is used to perform steps S1, S22a to S22b in the above-described image processing method. For example... Figure 6 As shown, this image processing model can be implemented using a convolutional neural network. The image processing model includes: a third feature extraction module 31, a third processing module 33, and a third pixel rearrangement module 32.

[0126] In some embodiments, the third feature extraction module 31 is used to extract features from the input image INP at the first scale to generate multiple first initial feature maps, multiple fifth initial feature maps, and multiple sixth initial feature maps. The number of first initial feature maps is d. V The number of the fifth initial feature maps is s. 2 *d K The number of the sixth initial feature maps is d. K *d V s is the magnification factor of the second scale relative to the first scale, d V and d K All are integers greater than 1.

[0127] The third feature extraction module 31 may include: d V The first feature channel, d V The first feature channel, such as Figure 6 V1 to V shown dv In some embodiments, for each first feature channel, the input image INP can be convolved using only a single convolution kernel. That is, Figure 6 V1 to V dv Corresponding to d V There is a first convolution kernel with a size of k×k and a convolution stride of, for example, 1×1.

[0128] In addition, the third feature extraction module 31 may also include: s 2 *d K The fifth feature channel and d K *d V The sixth feature channel, s 2 *d K The fifth feature channel, such as Figure 6 Q1 to shown d K *d V The sixth feature channel, such as Figure 6K1 to K dk* . s 2 *d K The fifth feature channel is used to convolve the input image INP to generate s 2 *d K The fifth initial feature map (e.g.) Figure 6 q in 1,1 to ). d K *d V The sixth feature channel is used to convolve the input image INP to generate d K *d V The sixth initial feature map (e.g.) Figure 6 k in 1,1 to In some embodiments, the input image can be convolved using only a single convolution kernel for each fifth feature channel. That is, Figure 6 Q1 to Corresponding to s 2 *d K A fifth convolutional kernel, with a size of k×k, and a stride of, for example, 1×1. For each sixth feature channel, the input image can be convolved using only a single kernel. That is, Figure 6 The sixth feature channel K1 to K dk* Corresponding to d K *d V There is a sixth convolution kernel, the size of which is k×k, and the convolution stride is, for example, 1×1.

[0129] Among them, the first initial feature map is the value feature map used in the self-attention mechanism, the fifth initial feature map is the query feature map used in the self-attention mechanism, and the sixth initial feature map is the key feature map used in the self-attention mechanism.

[0130] The third processing module 33 is used to process multiple first initial feature maps using a preset second normalization rule to generate multiple fourth feature maps with a first scale. The multiple fourth feature maps are as follows: Figure 6 The F1 to shown

[0131] The third processing module 33 includes a first processing submodule 331 and a second processing submodule 332. The first processing submodule 331 can be configured to generate multiple weighted feature map groups based on multiple fifth initial feature maps and multiple sixth initial feature maps. Each weighted feature map group corresponds one-to-one with a fourth feature map, and each weighted feature map group includes d... VThere are three weighted feature maps. The fifth initial feature map, the sixth initial feature map, and the weighted feature map all have the first scale.

[0132] In some embodiments, the magnification ratio of the second scale relative to the first scale is s, that is, when the first scale is H*W, the second scale is sH*sW. The specific process of the first processing submodule 331 generating multiple weighted feature map groups may include: performing a multiplication operation on a first matrix Mx1 composed of multiple fifth initial feature maps and a second matrix Mx2 composed of multiple sixth initial feature maps to generate a third matrix. The first matrix Mx1 includes elements arranged in s... 2 line d K The first matrix element of the column, each first matrix element is a fifth initial feature map; the second matrix Mx2 includes elements arranged in d... K line d V The second matrix element of the column, each second matrix element is a sixth initial feature map; the third matrix includes elements arranged in s... 2 line d V Multiple third matrix elements in a column, each of which is an initial weight feature map; third matrix elements in the same row form an initial weight feature map group; d K s and s are both integers greater than 1. Then, a normalization function (such as the SOFTMAX function) is used to normalize the multiple initial weight feature maps in the third matrix to generate multiple weight feature maps. The weight feature maps correspond one-to-one with the initial weight feature maps. The weight feature maps corresponding to multiple initial weight feature maps in the same initial weight feature map group form a weight feature map group.

[0133] Specifically, the multiplication operation between the first matrix Mx1 and the second matrix Mx2 is as follows: multiply the p-th row of the first matrix Mx1 by the d... K The elements of the first matrix and the d-th column of the second matrix Mx2 K Multiplying each element of the second matrix one by one, we get d. K The first product element will be d KThe first product elements are superimposed to obtain the element of the third matrix in the p-th row and q-th column. Here, the first matrix element is a fifth initial feature map, the second matrix element is a sixth initial feature map, and the first product element is also a feature map (denoted as the seventh initial feature map). Multiplying the first matrix element by the second matrix element is equivalent to multiplying the feature elements at the same position in the fifth and sixth initial feature maps to obtain the feature elements at the same position in the seventh initial feature map. For example, multiplying the feature element in the first row and first column of the fifth initial feature map by the feature element in the first row and first column of the sixth initial feature map yields the value of the feature element in the first row and first column of the seventh initial feature map; multiplying the feature element in the first row and second column of the fifth initial feature map by the feature element in the first row and second column of the sixth initial feature map yields the value of the feature element in the first row and second column of the seventh initial feature map, and so on. Furthermore, superimposing multiple seventh initial feature maps involves adding the feature elements at the same position in multiple seventh initial feature maps.

[0134] In some embodiments, the process of normalizing multiple initial weight feature maps is similar to... Figure 5 The normalization operations shown are the same, so they will not be repeated here.

[0135] The second processing submodule 332 can be configured to, for any weighted feature map group, generate a fourth feature map corresponding to the weighted feature map group based on the weighted feature map group and multiple first initial feature maps. As mentioned above, the third matrix includes arrays arranged in s 2 line d V The number of third matrix elements in the column, i.e., the number of weighted feature maps, is s. 2 Each weighted feature map group includes d V Given a weighted feature map, after processing by the second processing submodule 332, s can be generated. 2 The fourth feature map, such as Figure 6 F1 to As shown. Each fourth feature map has the aforementioned first scale.

[0136] Within the same weighted feature map group, each weighted feature map corresponds to a first initial feature map, and different weighted feature maps correspond to different first initial feature maps. For any weighted feature map group, the second processing submodule 332 can generate the fourth feature map corresponding to that weighted feature map through the following process: multiplying each weighted feature map in the weighted feature map group with its corresponding first initial feature map to generate multiple third intermediate feature maps; and superimposing the multiple third intermediate feature maps to generate the fourth feature map corresponding to the weighted feature map group.

[0137] Among them, the weighted feature map, the first initial feature map, and the fourth feature map have the same scale, for example, the scale of all three is H*W.

[0138] Specifically, multiplying the weighted feature map with the first initial feature map to generate a third intermediate feature map corresponding to the weighted feature map can include: multiplying the feature elements at the same position in the weighted feature map with the feature elements at the same position in the first initial feature map to obtain the values ​​of the feature elements at the same position in the third intermediate feature map. For example, multiplying the feature element in the first row and first column of the weighted feature map with the feature element in the first row and first column of the first initial feature map to obtain the value of the feature element in the first row and first column of the third intermediate feature map; multiplying the feature element in the first row and second column of the weighted feature map with the feature element in the first row and second column of the first initial feature map to obtain the value of the feature element in the first row and second column of the third intermediate feature map; and so on, multiplying the feature element in the last row and last column of the weighted feature map with the feature element in the last row and last column of the first initial feature map to obtain the value of the feature element in the last row and last column of the third intermediate feature map.

[0139] Superimposing multiple third intermediate feature maps can specifically include: adding feature elements at the same position in multiple third intermediate feature maps to obtain the value of the feature element at the same position in the fourth feature map. For example, adding feature elements in the first row and first column of multiple third intermediate feature maps to obtain the value of the feature element in the first row and first column of the fourth feature map; adding feature elements in the first row and second column of multiple third intermediate feature maps to obtain the value of the feature element in the first row and second column of the fourth feature map; and so on, adding feature elements in the last row and last column of multiple third intermediate feature maps to obtain the value of the feature element in the last row and last column of the fourth feature map.

[0140] The third pixel rearrangement module 32 is used to perform the above step S22b, that is, to rearrange the pixels of multiple fourth feature maps to obtain the output image.

[0141] In some embodiments, the feature elements in the fourth feature map are arranged in H rows and K columns, where H and K are both positive integers greater than 1. The output image includes an H-row, K-column image block. The process of rearranging multiple fourth feature maps by the third rearrangement processing module 32 may specifically include: selecting the feature element in the h-th row and k-th column of each fourth feature map and arranging it into the h-th row and k-th column image block in the output image; h is a positive integer from 1 to H, and k is a positive integer from 1 to K. The process of rearranging multiple fourth feature maps to obtain the output image is similar to... Figure 3 The process of rearranging multiple initial feature maps into a single initial feature map is similar and will not be described in detail here.

[0142] In this embodiment of the disclosure, Figure 2 , Figure 4 and Figure 6 The image processing models in this document are all trained image processing models. Figure 7 A flowchart of the training steps of an image processing model provided for some embodiments of this disclosure, such as... Figure 7 As shown, the training process of the image processing model includes:

[0143] S201. Input the degraded image generated based on the reference image into the image processing model to obtain the training output image.

[0144] For example, the reference image and the degraded image may share the same scene, meaning their content is identical. Furthermore, the reference image may have a second scale, while the degraded image may have a first scale. The ratio of the second scale to the first scale can be the target magnification ratio, which can be selected based on actual needs. Degraded images can be generated through various methods, such as downsampling, pooling, and filtering.

[0145] S202. Based on the training output image and the reference image, obtain the loss value of the image processing model according to the predetermined loss function.

[0146] S203. Adjust the parameters of the image processing model based on the loss value of the image processing model.

[0147] In some embodiments, the loss function Loss may include the mean squared error between the reference image and the training output image. Specifically, it is shown in the following formula:

[0148] Loss = E[(xy)] 2 ]

[0149] Where x is the training output image, y is the reference image, and E[] represents the calculation of matrix energy. For example, E[] can be used to calculate the maximum or average value of the elements in the matrix enclosed in "[]".

[0150] During the training of an image processing model, the training objective is to minimize the loss value. For example, during training, the parameters of the image processing model are continuously adjusted so that the training output image output by the model after parameter adjustment continuously approaches the reference image, thereby continuously reducing the loss value. Optionally, training can be considered complete when the loss value decreases to a predetermined threshold range or essentially stops changing.

[0151] It should be noted that the loss function provided in this embodiment is exemplary, and the embodiments of this disclosure include, but are not limited to, this. For example, the loss function may also be an L1 regularization term of the reference image and the training output image.

[0152] Figure 8 The processing time versus output quality curves for various image processing methods are shown. Figure 8 The horizontal axis represents output quality, expressed as the peak signal-to-noise ratio (PSNR) between the actual output image and the target output image, in dB; the vertical axis represents processing speed, in FHD / s. Various image processing methods are included: BICUBIC, ESPCN, FSRCNN, EdgeSR, MuxMax, and MuxTransformer. MuxMax utilizes... Figure 2 The MuxTransformer method is a method for image processing using models in the model. Figure 4 The method of image processing using the model in the model. Figure 9 A schematic diagram of the image processing model for executing the EdgeSR method, as shown below. Figure 9 As shown, the image processing model executing the EdgeSR method includes: a first transposed convolution module 410, a second transposed convolution module 420, a normalization module 430, and a weighted summation module 440. The first transposed convolution module 410 may include C feature channels F1 to Fc (corresponding to C convolution kernels). The C feature channels F1 to Fc are used to apply transposed convolution to the input image INP respectively to generate C feature maps f1 to fC. For each feature channel, only a single convolution kernel can be used to perform transposed convolution processing on the input image to obtain the corresponding feature map, that is, Figure 9 In the diagram, F1 to Fc correspond to individual convolution kernels. For example, the stride of the transposed convolution can be set to s, and the size of the convolution kernel can be set to sk'×sk', where s represents the target magnification ratio of the output image OUTP compared to the input image INP, and k' represents a positive integer.

[0153] The second transposed convolution module 420 may include C feature channels M1 to Mc, which are used to apply transposed convolution to the input image INP to generate C feature maps m01 to m0C. M1 to Mc each correspond to a single convolution kernel. The normalization module 430 performs normalization operations on the feature maps m01 to m0C to generate the multiple mask maps m1 to mC. The multiple mask maps m1 to mC correspond one-to-one with the feature maps f1 to fC. The weighted summation module 440 weights m1 to mC in a one-to-one correspondence with the feature maps f1 to fC, and synthesizes the weighted feature maps to generate the output image OUTP.

[0154] in, Figure 8 The various image processing methods involved are executed on the same hardware platform (e.g., NVIDIA Jetson AGX Xavier). For each image processing method, the quality and speed can be adjusted by modifying the parameters of the corresponding image processing model (e.g., kernel size, number of feature channels). Figure 8 It can be seen that the ESPCN and FSRCNN methods produce high-quality output images, but their processing speed is relatively slow. However, the image processing methods provided in this embodiment, including the eSR method, can significantly improve the image processing speed while ensuring that the output image quality is not too low by adjusting the parameters of the image processing model.

[0155] This disclosure also provides an image processing apparatus. Figure 10 This is a schematic block diagram of an image processing apparatus provided for some embodiments of the present disclosure, which can be deployed on an edge device. Figure 10 As shown, the image processing apparatus includes an image acquisition module 301 and an image processing module 302. The image acquisition module is configured to acquire an input image at a first scale. For example, the image acquisition module 301 may include a memory storing the input image and momentum terms at multiple levels; alternatively, the image acquisition module 301 may also include one or more cameras to acquire the input image. The image processing module is configured to perform steps S1 to S2 in the image processing method described above. In some embodiments, the image acquisition module 301 and the image processing module 302 may be implemented as hardware, software, firmware, or any feasible combination thereof.

[0156] Figure 11 The diagram illustrates an image processing apparatus provided for other embodiments of this disclosure, which can be deployed on an edge device. Figure 11As shown, the image processing apparatus includes a memory 201 and a processor 202. For example, the memory 201 is used to non-temporarily store computer-readable instructions, and the processor 202 is used to execute the computer-readable instructions, which, when executed by the processor, perform the image processing method provided in any embodiment of this disclosure.

[0157] The memory 201 and processor 202 can communicate directly or indirectly with each other. For example, in some examples, the image processing apparatus may also include a system bus, through which the memory 201 and processor 202 can communicate, such as the processor 202 accessing the memory via the system bus. In other examples, components such as the memory 201 and processor 202 can communicate via a network connection. The network may include a wireless network, a wired network, and / or any combination of wireless and wired networks. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things (IoT) based on the Internet and / or telecommunications networks, and / or any combination of the above networks. Wired networks may use methods such as twisted-pair cables, coaxial cables, or fiber optic transmission for communication, while wireless networks may use methods such as 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee, or WiFi. This disclosure does not limit the type and function of the network.

[0158] For example, processor 202 can control other components in the image processing device to perform desired functions. The processor can be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), tensor processor (TPU), or graphics processing unit (GPU). The CPU can be based on x86 or ARM architectures, etc. The GPU can be integrated directly onto the motherboard or built into the motherboard's northbridge chip. The GPU can also be built into the CPU.

[0159] For example, memory 201 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0160] For example, one or more computer instructions may be stored on memory 201, and processor 202 may execute computer instructions to perform various functions. Various application programs and various data, such as input images, output images, reference images, degraded images, and various data used and / or generated by application programs, may also be stored in the computer-readable storage medium.

[0161] For example, some computer instructions stored in memory 201 can be executed by a processor to perform one or more steps in the image processing method described above.

[0162] The image processing apparatus may also include an input interface that allows external devices to communicate with it. For example, the input interface may be used to receive instructions from an external computer device, a user, or other source. The image processing apparatus may also include an output interface that connects it to one or more external devices. For example, the image processing apparatus may display images through the output interface.

[0163] For a detailed explanation of the processing procedure of the image processing method, please refer to the relevant description in the embodiments of the above image processing method.

[0164] This disclosure also provides a non-transient computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image processing method described above. For example, the computer-readable storage medium may include a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0165] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. An image processing method for edge devices, wherein, The method comprises: performing feature extraction on an input image of a first scale to generate a plurality of first initial feature maps of the first scale; performing first processing or second processing on the plurality of first initial feature maps to generate an output image of a second scale, the second scale being larger than the first scale; wherein the first processing comprises: dividing the plurality of first initial feature maps into a plurality of first initial feature map groups, each of the first initial feature map groups comprising a plurality of first initial feature maps, for each of the first initial feature map groups, performing pixel rearrangement on the plurality of first initial feature maps in the first initial feature map group to generate a first feature map corresponding to the first initial feature map group, the first feature map after pixel rearrangement having the second scale; processing the first feature maps corresponding to the plurality of first initial feature map groups using a preset first normalization rule to generate the output image; the second processing comprises: processing the plurality of first initial feature maps using a preset second normalization rule to generate a plurality of fourth feature maps of the first scale; performing pixel rearrangement on the plurality of fourth feature maps to obtain the output image.

2. The image processing method of claim 1, wherein, When the image processing method comprises the first processing on the plurality of first initial feature maps, processing the first feature maps corresponding to the plurality of first initial feature map groups using a preset first normalization rule to generate the output image specifically comprises: forming a plurality of feature element groups based on a plurality of feature elements at corresponding positions in the plurality of first feature maps; generating the output image based on a maximum element value in each feature element group.

3. The image processing method of claim 1, wherein, When the image processing method comprises the first processing on the plurality of first initial feature maps, the first processing further comprises: performing convolution processing on the input image to generate a plurality of second initial feature map groups and a plurality of third initial feature map groups, each of the second initial feature map groups comprising a plurality of second initial feature maps, each of the third initial feature map groups comprising a plurality of third initial feature maps, each of the second initial feature maps and each of the third initial feature maps having the first scale; for any one of the second initial feature map groups, performing pixel rearrangement on the plurality of second initial feature maps in the second initial feature map group to generate the second feature map corresponding to the second initial feature map group; for any one of the third initial feature map groups, performing pixel rearrangement on the plurality of third initial feature maps in the third initial feature map group to generate the third feature map corresponding to the third initial feature map group; processing the first feature maps corresponding to the plurality of first initial feature map groups using a preset first normalization rule to generate the output image specifically comprises: generating a plurality of mask maps based on the plurality of second feature maps and the plurality of third feature maps; generating a plurality of first intermediate feature maps of the second scale based on the plurality of mask maps and the plurality of first feature maps; synthesizing the output image based on the plurality of first intermediate feature maps.

4. The image processing method of claim 3, wherein, The second feature maps correspond to the third feature maps one by one, and each second feature map group and its corresponding third feature map group form a feature map pair; The second feature maps correspond to the third feature maps one by one, and each second feature map group and its corresponding third feature map group form a feature map pair; For any one of the feature map pairs, the second feature map and the third feature map in the feature map pair are subjected to a multiplication operation to generate a second intermediate feature map corresponding to the feature map pair; The second intermediate feature maps are subjected to a normalization operation by using a normalization function to generate a plurality of mask maps.

5. The image processing method of claim 3, wherein, The mask maps correspond to the first feature maps one by one, and each mask map represents the feature weight of the corresponding first feature map; Based on the plurality of mask maps and the plurality of first feature maps, a plurality of first intermediate feature maps with a second scale are generated, specifically including: Based on the feature weight represented by each mask map, the corresponding first feature map is weighted to generate a plurality of first intermediate feature maps.

6. The image processing method of claim 3, wherein, Each feature element in each of the second initial feature maps and each of the third initial feature maps is arranged in M rows and N columns, and M and N are both positive integers greater than 1; the second feature maps and the third feature maps each include M rows and N columns of patches; The plurality of second initial feature maps in the second initial feature map group are subjected to pixel rearrangement to generate the second feature maps corresponding to the second initial feature map group, specifically including: selecting the feature element in the i-th row and the j-th column of each of the plurality of second initial feature maps, and arranging it into the i-th row and the j-th column patch in the second feature map; i is a positive integer in 1 to M, and j is a positive integer in 1 to N; The plurality of third initial feature maps in the third initial feature map group are subjected to pixel rearrangement to generate the third feature maps corresponding to the third initial feature map group, specifically including: selecting the feature element in the i-th row and the j-th column of each of the plurality of third initial feature maps, and arranging it into the i-th row and the j-th column patch in the third feature map.

7. The image processing method of claim 1, wherein, Each feature element in each first initial feature map is arranged in P rows and Q columns, and P and Q are both positive integers greater than 1; the first feature map includes P rows and Q columns of patches; The plurality of first initial feature maps in the first initial feature map group are subjected to pixel rearrangement to generate the first feature maps corresponding to the first initial feature map group, specifically including: The plurality of first initial feature maps in the first initial feature map group are subjected to pixel rearrangement to generate the first feature maps corresponding to the first initial feature map group, specifically including:

8. The image processing method of claim 1, wherein, When the image processing method comprises a second processing on a plurality of the first initial feature maps, the number of the first initial feature maps is d V , d V is an integer greater than 1; The plurality of first initial feature maps in the first initial feature map group are subjected to pixel rearrangement to generate the first feature maps corresponding to the first initial feature map group, specifically including: The plurality of first initial feature maps in the first initial feature map group are subjected to pixel rearrangement to generate the first feature maps corresponding to the first initial feature map group, specifically including: Based on the plurality of the fifth initial feature maps and the plurality of the sixth initial feature maps, a plurality of weight feature map groups are generated, the weight feature map groups correspond one-to-one to the fourth feature maps, and each of the weight feature map groups includes d V weight feature maps. The second scale is enlarged by a ratio s relative to the first scale, 9. The image processing method of claim 8, wherein, ​ The plurality of weight feature map groups are generated based on the plurality of fifth initial feature maps and the plurality of sixth initial feature maps, and specifically include: performing multiplication operation on a first matrix composed of a plurality of the fifth initial feature maps and a second matrix composed of a plurality of the sixth initial feature maps to generate a third matrix; wherein the first matrix comprises first matrix elements arranged in s 2 rows and d K columns, each of the first matrix elements being one of the fifth initial feature maps; the second matrix comprises second matrix elements arranged in d K rows and d V columns, each of the second matrix elements being one of the sixth initial feature maps; the third matrix comprises a plurality of third matrix elements, each of the third matrix elements being one of the initial weight feature maps; d K , s are integers greater than 1. The plurality of initial weight feature maps in the third matrix are normalized by using a normalization function to generate a plurality of weight feature maps, wherein the weight feature maps correspond to the initial weight feature maps one by one, and a weight feature map group is composed of weight feature maps corresponding to a plurality of initial weight feature maps located in the same row in the third matrix.

10. The image processing method of claim 8, wherein, In the same weight feature map group, each weight feature map corresponds to a first initial feature map, and different weight feature maps correspond to different first initial feature maps. The fourth feature map corresponding to the weight feature map group is generated based on the weight feature map group and the plurality of first initial feature maps, and specifically includes: Each weight feature map in the weight feature map group is multiplied by the first initial feature map corresponding thereto to generate a plurality of third intermediate feature maps. The plurality of third intermediate feature maps are superimposed to generate the fourth feature map corresponding to the weight feature map group.

11. The image processing method of claim 1, wherein, When the image processing method includes a second processing on the plurality of first initial feature maps, the feature elements in the fourth feature map are arranged in H rows and K columns, and H and K are both positive integers greater than 1; the output image includes H rows and K columns of image blocks; The output image is obtained by rearranging the pixels of the plurality of fourth feature maps, and specifically includes: Selecting the feature element in the hth row and the kth column of each fourth feature map and arranging it into the hth row and the kth column of the output image; h is a positive integer in 1 to H, and k is a positive integer in 1 to K.

12. The image processing method according to any one of claims 1 to 11, wherein, The trained image processing model is used to perform the steps of the image processing method.

13. The image processing method of claim 12, wherein, The image processing model is trained by the following steps: A degraded image generated based on a reference image is input into the image processing model to generate a training output image; According to the training output image and the reference image, a loss value of the image processing model is determined based on a predetermined loss function; The parameters of the image processing model are adjusted according to the loss value of the image processing model.

14. An image processing apparatus, comprising: It includes: An image acquisition module configured to acquire an input image of a first scale; An image processing module configured to perform feature extraction on the input image to generate a plurality of first initial feature maps of the first scale; A first processing or a second processing is performed on the plurality of first initial feature maps to generate an output image of a second scale, and the second scale is greater than the first scale; The first processing includes: The plurality of first initial feature maps are divided into a plurality of first initial feature map groups, each first initial feature map group includes a plurality of first initial feature maps, and for each first initial feature map group, the plurality of first initial feature maps in the first initial feature map group are rearranged to generate a first feature map corresponding to the first initial feature map group, and the first feature map after pixel rearrangement has the second scale; The first feature maps corresponding to the plurality of first initial feature map groups are processed using a preset first normalization rule to generate the output image; The second processing includes: The plurality of first initial feature maps are processed using a preset second normalization rule to generate a plurality of fourth feature maps having a first scale; The output image is obtained by performing pixel rearrangement on the plurality of fourth feature maps.

15. An image processing apparatus, comprising: Comprise: A memory and a processor, the memory has a computer program stored thereon, wherein the computer program is executed by the processor to implement the image processing method in any one of claims 1 to 13.

16. A non-transitory computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the image processing method in any one of claims 1 to 13.

Images