Image processing method, device, storage medium and program product

Abstract

The present application relates to the field of image processing technology, and discloses an image processing method, device, storage medium and program product. The image processing method includes: when adjusting the resolution of an image through a neural network model, if the input resolutions that can be received by the N downsampling layers in the neural network model are L 0 , L 1 , L 2 …L k …L N‑1 , when the electronic device preprocesses an image with a resolution of X, it can preprocess the image resolution from X to an input resolution L close to X. k‑1 (k ≥ 1). Will have resolution L k‑1 The image data is input into the corresponding kth downsampling layer, and the downsampling layer k, downsampling layer k+1…downsampling layer N, and other decoder modules are loaded and run in sequence, so as to obtain a target image with a target resolution. In this way, while adjusting the image resolution to the target resolution, it can also save computing resources and storage resources when the electronic device runs the neural network model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and program product. Background Art

[0002] With the development of deep learning technology, super-resolution processing (abbreviated as "super-resolution processing") technology has been widely used in image processing, computer vision and other fields. Super-resolution processing technology can be used to reconstruct low-resolution images with insufficient clarity into high-definition high-resolution images, thereby improving the quality and clarity of the image.

[0003] When electronic devices run super resolution networks (SRN) to perform super resolution processing on images, they usually need to load and run all the neural network layers in the super resolution network. For example, taking the classic super resolution network UNet model as an example, Figure 1 FIG. 4 shows a schematic diagram of a scene in which a low-resolution image A is subjected to super-resolution processing. Figure 1 As shown, the electronic device can first preprocess the low-resolution image A. Then, the image data obtained after preprocessing that meets the input conditions of the super-resolution network is input into the super-resolution network, and then all downsampling layers (e.g., N downsampling layers) in the super-resolution network are loaded and run, and image features such as color and texture are further extracted through the N downsampling layers. Next, the electronic device loads and runs all upsampling layers (e.g., M upsampling layers) in the super-resolution network, and reconstructs the acquired image features into a high-resolution image A' through the M upsampling layers.

[0004] However, if you want to load all the neural network layers of the super-resolution network (e.g. Figure 1 In order to perform super-resolution processing on the image, it is necessary to occupy more computing resources and storage resources of the electronic device to realize data operation and storage of each neural network layer. Moreover, in actual engineering applications, it is usually impossible to load and run all the neural network layers due to the limited computing power or memory of the electronic device, thus failing to realize super-resolution processing of the image. Summary of the invention

[0005] To solve the above problems, the present application provides an image processing method, device, storage medium and program product.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the method includes: obtaining a first image to be processed, the first image having a first resolution; adjusting the first resolution of the first image to a first input resolution, wherein the first input resolution is: the processing resolution of the kth downsampling layer among the N downsampling layers of a neural network model that processes the first image, 1<k≤N, k and N are both integers, and the absolute value of a first difference between the first input resolution and the first resolution is less than the absolute value of a second difference between the second input resolution and the first resolution, and the second input resolution is the processing resolution of the first downsampling layer of the N downsampling layers; inputting the first image into the kth downsampling layer, and running the neural network model to obtain a second image with a second resolution.

[0007] In the present application, the first image may be the image to be processed that needs to be adjusted in resolution as mentioned in the present application; the first resolution may be the initial resolution X of the image to be processed as mentioned in the present application; the first input resolution may be the input resolution L close to the initial resolution X as mentioned in the present application. k-1 ; The second image may be the target image with target resolution mentioned in the present application; the second resolution may be the target resolution of the target image obtained by the neural network model mentioned in the present application; the kth downsampling layer may be the kth downsampling layer when the neural network model is started to run mentioned in the present application; the first absolute value of the difference may be the absolute value of the difference between the first input resolution and the first resolution mentioned in the present application; the second absolute value of the difference may be the absolute value of the difference between the second input resolution and the first resolution mentioned in the present application; the second input resolution may be the processing resolution of the other downsampling layers among the N downsampling layers except the kth downsampling layer.

[0008] In some implementations, the second input resolution may be the processing resolution of the first downsampling layer. That is, when the first resolution differs greatly from the processing resolution of the first downsampling layer, and the first resolution differs little from the processing resolution of any other intermediate downsampling layer, the electronic device may adjust the resolution of the first image to the processing resolution of any intermediate downsampling layer, so that the electronic device does not need to run the neural network model from the first downsampling layer. In addition, the second input resolution may also be the processing resolution of any downsampling layer except the kth downsampling layer among the N downsampling layers. That is, the electronic device may select the first input resolution that is the smallest difference from the first resolution from the N input resolutions corresponding to the N downsampling layers, and adjust the first resolution of the first image to the first input resolution, so that the electronic device can run the least downsampling layer.

[0009] In some implementations, when the electronic device adjusts the resolution of an image through a neural network model, if the input resolutions that can be received by N (N≥1) downsampling layers in the neural network model are L 0 , L 1 , L 2 …L k-1 , L k …L N-1 , when the electronic device preprocesses an image with a resolution of X, it can preprocess the image resolution from X to the first input resolution L that is closest to X. k-1 (k ≥ 1). Will have the first input resolution L k-1 The image data is input into the corresponding k-th downsampling layer, and the downsampling layer k, downsampling layer k+1…downsampling layer N, and other decoder parts and other modules are loaded and run in sequence to obtain a target image with a target resolution.

[0010] In some implementations, compared to directly preprocessing the image resolution from X to the input resolution L that the entire neural network model can accept 0 (i.e., the processing resolution of the first downsampling layer) and then load and run the neural network model. The above preprocesses the image resolution from X to L k-1 The image processing method for post-loading and running the neural network model may not need to execute the k-1 downsampling layers before the downsampling layer k. In this way, through the above image processing method, while adjusting the image resolution to the target resolution, it can also save computing resources and storage resources when the electronic device runs the neural network model, thereby improving the operation rate.

[0011] In a possible implementation of the first aspect, the neural network model includes a super-resolution network, and the second resolution is greater than the first resolution.

[0012] In some implementations, the above image processing method can be applied to any computer vision task that adjusts the image resolution. Moreover, the applicable neural network model related to adjusting the image resolution is not limited to the super-resolution network of the encoder-decoder structure, but can also be applied to the neural network model including the downsampling layer and any other modules.

[0013] In this way, in any electronic device with sufficient or limited resources, when running any neural network model including a downsampling layer and any other modules, the neural network model can be run through the image processing method provided by the present application to obtain a target image, so that the target image has a target resolution, and at the same time, the computing resources and storage resources when the electronic device runs the neural network model can be saved, thereby improving the computing speed.

[0014] In a possible implementation of the first aspect above, the above-mentioned first input resolution can be determined in the following manner: determining the ratio between the processing resolution of each downsampling layer in the N downsampling layers and the processing resolution of the first downsampling layer, obtaining and storing the input scaling coefficient of each downsampling layer, and determining the ratio between the first resolution and the processing resolution of the first downsampling layer to obtain the image scaling coefficient; based on the input scaling coefficient and the image scaling coefficient of each downsampling layer, selecting a first input resolution from the processing resolutions of each downsampling layer, the absolute value of the difference with the first resolution being less than the absolute value of the second difference.

[0015] In some implementations, when determining a first input resolution L that is close to a first resolution X of the first image, k-1 When , we can first calculate the input resolution L that each downsampling layer can receive i (i ≥ 0) and the input resolution L of the entire super-resolution network 0 (i.e., the processing resolution of the first downsampling layer) i , and then calculate the image resolution X and the input resolution L of the super-resolution network 0 The image scaling factor S X Finally, the electronic device may select a first input resolution having a smaller absolute value of difference with the first resolution from the input resolutions that can be received by each downsampling layer based on the input scaling factors and the image scaling factor.

[0016] In some embodiments, the electronic device may also pre-store the calculated input scaling coefficients of each downsampling layer. When receiving the first image to be processed, the electronic device may directly read the stored input scaling coefficients for super-resolution processing, thereby improving processing efficiency and saving processing time. Alternatively, the electronic device may not store the input scaling coefficients, and when receiving the first image to be processed, directly calculate the input scaling coefficients for super-resolution processing. This application does not limit this.

[0017] In this way, a first input resolution close to the first resolution can be determined based on each input scaling factor and the image scaling factor, so that the first image can be subsequently input into the kth downsampling layer corresponding to the first input resolution, thereby reducing resource consumption of the electronic device.

[0018] In a possible implementation of the first aspect above, the above-mentioned selecting, from the processing resolutions of each downsampling layer based on the input scaling factor and the image scaling factor of each downsampling layer, a first input resolution whose absolute value of difference with the first resolution is less than an absolute value of a second difference, includes: corresponding to the image scaling factor being greater than or equal to 1, determining the first input resolution as the processing resolution of the first downsampling layer; corresponding to the image scaling factor being less than 1, sorting the input scaling factors and the image scaling factors corresponding to each downsampling layer, and selecting the processing resolution corresponding to any one of the input scaling factors adjacent to the image scaling factor as the first input resolution.

[0019] In some implementations, since the input resolution that each downsampling layer can receive is also the output resolution of the previous sampling layer, and each downsampling layer of the super-resolution network is a process of reducing the image resolution, the input resolution that each downsampling layer can receive is successively reduced. Therefore, the input scaling factor S is determined by the ratio of the processing resolution of each downsampling layer to the processing resolution of the first downsampling layer. i Among them, only S 0 is 1, and the other input scaling factors S i are all less than 1. Therefore, when the image scaling factor S X When it is greater than or equal to 1, it can be directly determined that the X The adjacent input scaling factor is S 0 , the image resolution can be directly preprocessed from X to the input resolution L of the super-resolution network 0 .

[0020] In some implementations, the image scaling factor S X When it is less than 1, the input scaling factor (S 0 , S 1 , S 2 …S N-1 ) and the image scaling factor S corresponding to the first image X Sort by, for example, from largest to smallest, it is S 0 >S 1 >…>S k-1 >S X >S k >…>S N-1 , select the image scaling factor S X Adjacent input scaling factors (e.g., S k-1 , S k ), the input resolution corresponding to any adjacent input scaling factor (e.g., S k-1 The corresponding L k-1) as the first input resolution close to the first resolution X of the first image. In this way, the first input resolution can be selected from the processing resolutions of each downsampling layer by simple sorting, which can reduce the resource consumption of the electronic device and improve the calculation efficiency.

[0021] In a possible implementation of the first aspect above, the super-resolution network includes N channel transformation modules, each of the N channel transformation modules corresponds one-to-one to each of the N downsampling layers, and the channel transformation module is used to adjust the channel feature dimension of the first image to the processing channel feature dimension of the downsampling layer corresponding to the channel transformation module.

[0022] In some implementations, for the image input data that can be received by each downsampling layer, in addition to the image resolution needing to satisfy the input resolution L that can be received by each downsampling layer i (Width W i ×Height H i ), the number of channels of the image (also called "channel feature dimension") also needs to satisfy the number of input channels that each downsampling layer can receive C i Therefore, a channel transformation module (for example, a neural network processing module such as 1×1 convolution) can be added before each downsampling layer of the super-resolution network to pre-process the image resolution from the first resolution X to a similar first input resolution L. k-1 Afterwards, the channel feature dimension of the image is adjusted to the first channel feature dimension C that can be received by the kth downsampling layer through the channel transformation module. k-1 Then adjust the completed image data (i.e., with the first input resolution L k-1 , with the first channel feature dimension C k-1 In this way, the image data of the image to be processed can meet the input conditions of the kth downsampling layer, so that the image data of the image to be processed can be input into the kth downsampling layer, and the super-resolution network can be run to obtain a high-resolution image.

[0023] In a possible implementation of the first aspect above, the first image is input into the kth downsampling layer, and the neural network model is run to obtain a second image with a second resolution, including: inputting the first image into a channel transformation module corresponding to the kth downsampling layer to obtain a first image whose channel feature dimension is a first channel feature dimension, and the first channel feature dimension is a processing channel feature dimension of the kth downsampling layer; inputting the first image with the first input resolution and the first channel feature dimension into the kth downsampling layer of the super-resolution network, and running the super-resolution network to obtain the second image with the second resolution.

[0024] The first channel feature dimension may be the channel feature dimension C that can be received by the kth downsampling layer mentioned in this application. k-1 .

[0025] In some implementations, the electronic device adjusts the resolution of the image to be processed to the first input resolution L k-1 After that, the electronic device can also adjust other image data (such as channel feature dimensions) of the image to be processed to the input data that can be received by the kth downsampling layer. Finally, the electronic device can input the image data to be processed with the first input resolution and satisfying other input conditions of the kth downsampling layer into the kth downsampling layer, so that the electronic device sequentially loads and runs downsampling layer k, downsampling layer k+1…downsampling layer N, and other neural network layers to obtain the final target image.

[0026] In some implementations, when the processing channel feature dimension of the kth downsampling layer of the super-resolution network is different from the channel feature dimension of the image to be processed, the image resolution is preprocessed from the first resolution X to a first input resolution L similar to the first resolution X. k-1 Afterwards, the channel feature dimension of the image is adjusted to the first channel feature dimension C that can be received by the kth downsampling layer through the channel transformation module. k-1 Then adjust the completed image data (i.e., with the first input resolution L k-1 , with the first channel feature dimension C k-1 In this way, the image data of the image to be processed can meet the input conditions of the kth downsampling layer, so that the image data of the image to be processed can be input into the kth downsampling layer, and the super-resolution network can be run to obtain a high-resolution image.

[0027] In some implementations, the channel transformation module can be a 1×1 convolution. Among them, the 1×1 convolution can increase or decrease the channel feature dimension of the image without changing the image resolution. Corresponding to different downsampling layers, the 1×1 convolution connected to the downsampling layer can be determined based on the channel feature dimension that each downsampling layer can receive. For example, if the channel feature dimension that the downsampling layer can receive is y, the connected 1×1 convolution is 1×1×y, which means that after the image is processed by the 1×1 convolution, the width and height of the image remain unchanged, and the channel feature dimension of the image becomes y. In addition, any other neural network layer that can adjust the channel feature dimension of the image can be used as a channel transformation module, and this application does not limit this.

[0028] In some implementations, a downsampling layer may include multiple neural network layers such as pooling layers and convolution layers to achieve functions such as reducing the resolution of the downsampling layer and increasing the channel feature dimension. Therefore, the number of parameters and the amount of calculation of a channel transformation module (such as 1×1 convolution) are much lower than the number of parameters and the amount of calculation of a downsampling layer. In this way, based on the above scheme, when running a super-resolution network, the electronic device can only load and run one channel transformation module and part of the downsampling layer. Compared with loading and running all downsampling layers, the image processing method provided by the present application can reduce memory usage and improve processing efficiency.

[0029] In a possible implementation of the first aspect above, the super-resolution network also includes M upsampling layers, the first image is input into the kth downsampling layer, and the neural network model is run to obtain a second image with a second resolution, including: inputting the first image into the kth downsampling layer of the super-resolution network, running N-k+1 downsampling layers to obtain a third image with a third resolution, and the third resolution is less than the first input resolution; inputting the third image into the 1st upsampling layer of the super-resolution network, running M upsampling layers to obtain a second image with a second resolution, and the second resolution is greater than the first input resolution.

[0030] In the present application, the third image may be the image data obtained after processing by N-k+1 downsampling layers mentioned in the present application; the third resolution may be the resolution of the image data obtained after processing by N-k+1 downsampling layers mentioned in the present application.

[0031] In some implementations, the electronic device may input image data having a first input resolution and a first channel feature dimension into the kth downsampling layer, and sequentially run each neural network layer after the kth downsampling layer (including the kth downsampling layer) to obtain a target image having a target resolution.

[0032] In some implementations, since k≥1 and k is an integer, when k is 1, the electronic device needs to run the entire super-resolution network to perform super-resolution processing on the image to be processed to obtain a high-resolution image. When k>1, the electronic device only needs to load and run some modules in the super-resolution network to perform super-resolution processing on the image to be processed to obtain a high-resolution image. In this way, while obtaining the target image with the target resolution, it can also reduce memory usage and improve processing efficiency. In addition, since the above scheme only needs to run some modules in the super-resolution network, for electronic devices with limited computing resources and limited storage resources, the super-resolution network can still be run normally to super-resolution the low-resolution image to obtain a high-resolution image.

[0033] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store instructions executed by one or more processors of the electronic device, and the processor being one of the one or more processors of the electronic device, and being used to execute the image processing method mentioned in the present application.

[0034] In a third aspect, the present application provides a readable storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the image processing method mentioned in the present application.

[0035] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method mentioned in the present application.

[0036] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 According to some embodiments of the present application, a schematic diagram of a scenario in which a super-resolution processing technology is implemented by running a super-resolution network is shown;

[0038] Figure 2 According to some embodiments of the present application, a schematic diagram of a super-resolution process is shown;

[0039] Figure 3 According to some embodiments of the present application, a schematic diagram of a super-resolution network structure is shown;

[0040] Figure 4 According to some embodiments of the present application, a flow chart of an image processing method is shown;

[0041] Figure 5 According to some embodiments of the present application, a schematic diagram of the structure of a super-resolution network after adding a channel transformation module is shown;

[0042] Figure 6 According to some embodiments of the present application, a schematic diagram of a process of super-resolution processing of a low-resolution image is shown when there are multiple encoder modules in the super-resolution network;

[0043] Figure 7 According to some embodiments of the present application, a specific flow chart of super-resolution processing of an image is shown;

[0044] Figure 8 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0045] The illustrative embodiments of the present application include, but are not limited to, an image processing method, an apparatus, a storage medium, and a program product.

[0046] The following is a brief description of the background of the image processing method provided in the embodiment of the present application.

[0047] As mentioned above, super-resolution processing technology can be used to reconstruct low-resolution images with insufficient clarity into high-definition high-resolution images, thereby improving the quality and clarity of the image.

[0048] Among them, when super-resolution processing is performed on a low-resolution image, the low-resolution image can be pre-processed first, so that the image resolution is scaled from the initial resolution to the input resolution that the super-resolution network can receive, and then the pre-processed image data is input into the super-resolution network for super-resolution processing to obtain the final high-resolution image.

[0049] Specifically, Figure 2 As shown, after receiving a low-resolution image with a resolution of X (201), the electronic device first needs to determine whether the resolution X is equal to the input resolution L that the super-resolution network can receive. 0 (202). If yes, it means that the electronic device has obtained a resolution of L 0 If not, the low-resolution image needs to be preprocessed so that the image is scaled from resolution X to resolution L. 0 (203), thus obtaining a resolution of L 0 Next, the electronic device may 0 The image data with a resolution of L is input into the super-resolution network. 0 The image data is input into the encoder part of the super-resolution network for processing (205) to extract high-dimensional features of the image such as texture and color; then the extracted high-dimensional features of the image are input into the decoder part of the super-resolution network for processing (206) to amplify the image feature resolution, thereby obtaining a high-resolution image with a resolution of H (207).

[0050] The encoder part of the super-resolution network mentioned in step 205 may include multiple downsampling layers, for example, the encoder part may include the above Figure 1 The N downsampling layers shown in , the resolution of the image to be processed is reduced in turn through each downsampling layer. The decoder part of the super-resolution network mentioned in the above step 206 may include multiple upsampling layers, for example, may include the above Figure 1The M upsampling layers shown in , through multiple upsampling layers, the image resolution is then enlarged to the final target resolution in turn. In addition, as mentioned above, when the electronic device runs the super-resolution network to perform super-resolution processing on the image, it is usually necessary to load all the down-sampling layers in the encoder part and load all the up-sampling layers in the decoder part.

[0051] The following example takes the encoder part including only three downsampling layers and the decoder part including only four upsampling layers as an example. Figure 3 The schematic diagram of the super-resolution network structure shown in the figure briefly describes the process of loading and running each neural network layer of the super-resolution network on the electronic device.

[0052] As mentioned above Figure 2 As described in step 204 and step 205 in the above, the electronic device determines that the image resolution is the input resolution L that the super-resolution network can accept. 0 After that, the resolution can be L 0 The image data is input into the super-resolution network for super-resolution processing. Specifically, Figure 3 As shown, the electronic device can 0 The image data of is input into the downsampling layer 1. The downsampling layer 1 may be the first downsampling layer, which is used to receive the image data with a resolution of L 0 The downsampling layer 1 has a resolution of L 0 The image data is subjected to feature extraction, resolution reduction and other operations, so that the output resolution is L 1 The image data is used as the input data of downsampling layer 2. The resolution of the input of downsampling layer 2 is L 1 The image data is subjected to feature extraction and resolution reduction again, so that the output resolution is L 2 The image data is used as the input data of the downsampling layer 3. Then, the downsampling layer 3 again processes the input with a resolution of L 2 The image data is subjected to feature extraction, resolution reduction and other operations, so that the output resolution is L 3 The image data is used as the input data of the decoder.

[0053] Similarly, if Figure 3 As shown, the upsampling layer 1 in the decoder receives the resolution L sent by the downsampling layer 3. 3 After the image data is obtained, operations such as resolution enlargement and feature fusion will be performed to output a resolution of L 4 The image data is used as the input data of upsampling layer 2. The upsampling layer 2 has a resolution L 4 The image data is magnified and the output resolution is L 5 The image data is used as the input data of upsampling layer 3. The resolution of the input of upsampling layer 3 is L5 The image data is magnified and the output resolution is L 6 The image data is used as the input data of upsampling layer 4. The resolution of the input of upsampling layer 4 is L 6 The image data is processed to output high-resolution image data with a resolution of H.

[0054] In summary, when super-resolution processing is performed on an image with a resolution of X, the image resolution is first pre-processed from X to the input resolution L of the super-resolution network. 0 , and then the preprocessed resolution is L 0 The image data is input into the super-resolution network, and the image data is processed in sequence through each sampling layer in the super-resolution network. However, at present, electronic devices usually need to start from the first downsampling layer of the super-resolution network, load and run N downsampling layers in sequence, and then load and run the decoder part and other modules to obtain the final high-resolution image. The above image processing method will cause electronic devices to consume a lot of computing resources and storage resources.

[0055] In order to solve the above problems, the present application provides an image processing method. In this method, when adjusting the resolution of an image through a neural network model, if the input resolutions that can be received by N (N≥1) downsampling layers in the neural network model are L 0 , L 1 , L 2 …L k-1 , L k …L N-1 , when the electronic device preprocesses an image with a resolution of X, it can preprocess the image resolution from X to an input resolution L close to X. k-1 (k ≥ 1). Will have input resolution L k-1 The image data is input into the corresponding kth downsampling layer, and the downsampling layer k, downsampling layer k+1…downsampling layer N, and other decoder modules are loaded and run in sequence to obtain the target image with the target resolution. It can be understood that compared with directly preprocessing the image resolution from X to the input resolution L that the entire neural network model can accept 0 (i.e., the processing resolution of the first downsampling layer) and then load and run the neural network model. The image resolution is preprocessed from X to L as mentioned in this application. k-1 By post-loading and running the neural network model, it is not necessary to execute the k-1 downsampling layers before the downsampling layer k. In this way, while adjusting the image resolution to the target resolution, it is also possible to save computing resources and storage resources when the electronic device runs the neural network model, thereby improving the operation rate.

[0056] It can be understood that the image processing method provided in this application can be applied to any computer vision task of adjusting image resolution. In addition, the applicable neural network model related to adjusting image resolution is not limited to the above Figure 3 The super-resolution network of the encoder-decoder structure shown can also be applied to a neural network model including a downsampling layer and any other modules. For ease of description, the image processing method provided by the present application is described below by taking the neural network model as a super-resolution network that processes an image from a low resolution to a high resolution, and the super-resolution network is a classic super-resolution network of the encoder-decoder structure as an example.

[0057] In some embodiments, when determining an input resolution L close to the image resolution X, k-1 When , we can first calculate the input resolution L that each downsampling layer can receive i (i ≥ 0) and the input resolution L of the entire super-resolution network 0 The input scaling factor S between i , and then calculate the image resolution X and the input resolution L of the super-resolution network 0 The image scaling factor S between X Finally, the input scaling factor (S 0 , S 1 , S 2 …S N-1 ) and the image scaling factor S corresponding to the image resolution X Sort by, for example, from largest to smallest, it is S 0 >S 1 >…> S k-1 >S X >S k >…>S N-1 , select the image scaling factor S X Adjacent input scaling factors (e.g., S k-1 , S k ), the input resolution corresponding to any adjacent input scaling factor (e.g., S k-1 The corresponding L k-1 ) as an input resolution close to the image resolution X.

[0058] Among them, the input resolution L that each downsampling layer can receive i The input resolution L of the entire super-resolution network 0 Input scaling factor S between i , can be obtained by inputting the resolution L i With the super-resolution network input resolution L 0 The ratio of image resolution X to the input resolution L of the super-resolution network is determined.0 The image scaling factor S between X , or through the image resolution X and the super-resolution network input resolution L 0 For example, the input scaling factor S corresponding to the input resolution of each downsampling layer can be calculated by referring to formula (1): i , refer to formula (ii) to calculate the image scaling factor S corresponding to the image resolution X .

[0059] (one)

[0060] (two)

[0061] In the above formula (1), S i The input resolution L of the i+1th downsampling layer can be i The input resolution L of the entire super-resolution network 0 Input scaling factor between i It can be the input resolution that the i+1th downsampling layer can accept, or it can be the output resolution of the i-th downsampling layer; L 0 It can be the input resolution of the entire super-resolution network and also the input resolution of the first downsampling layer; i ≥ 0 and S 0 is 1.

[0062] In the above formula (II), S X The image resolution X and the input resolution L of the entire super-resolution network can be 0 The image scaling factor between ; X can be the initial resolution of the image to be super-resolution processed; L 0 It can be the input resolution of the entire super-resolution network and also the input resolution of the first downsampling layer.

[0063] In this way, the electronic device can pre-calculate and store the input resolution L that each downsampling layer of the super-resolution network can receive through the above formula (I) i The input resolution L of the entire super-resolution network 0 Input scaling factor S between i Then, when the electronic device receives an image with a resolution of X that needs to be super-resolved, it first calculates the image resolution X and the input resolution L of the entire super-resolution network using the above formula (II). 0 The image scaling factor S between X Next, the electronic device can input the scaling factor (S 0 , S 1 , S 2 …S N-1 ) and the image scaling factor S XSort by, for example, from largest to smallest, it is S 0 >S 1 >…>S k-1 >S X >S k >…>S N-1 , select the image scaling factor S X Adjacent input scaling factors (e.g., S k-1 , S k ), the input resolution corresponding to any adjacent input scaling factor (e.g., S k-1 The corresponding L k-1 ) as the input resolution close to the image resolution X. Finally, the electronic device will preprocess the image resolution from X to the input resolution L close to X. k-1 , and the resolution is L k-1 The image data is input into the kth downsampling layer, and the downsampling layer k, downsampling layer k+1…downsampling layer N, and other decoder parts are loaded and run in sequence to obtain the final high-resolution image.

[0064] It can be understood that since the input resolution that each downsampling layer can receive is also the output resolution of the previous sampling layer, and each downsampling layer of the super-resolution network is a process of reducing the image resolution, the input resolution that each downsampling layer can receive is reduced in sequence. Figure 3 As shown in Figure 1, since downsampling layer 1, downsampling layer 2, and downsampling layer 3 will successively reduce the resolution of the image, Figure 3 The resolution shown is L 0 , resolution L 1 , resolution L 2 It can be seen that the input resolution L of each downsampling layer calculated based on the above formula (I) is i The input resolution L of the entire super-resolution network 0 Input scaling factor S between i Among them, only S 0 is 1, and the other input scaling factors S i Both are less than 1.

[0065] Therefore, the electronic device calculates the image resolution X and the input resolution L of the entire super-resolution network. 0 The image scaling factor S X When it is greater than or equal to 1, it can be directly determined that the X The adjacent input scaling factor is S 0 Therefore, the image resolution can be directly preprocessed from X to the input resolution L of the super-resolution network without sorting. 0 If the image resolution X is equal to the input resolution L of the entire super-resolution network 0The image scaling factor S X When it is less than 1, it is necessary to select the image scaling factor S through the sorting method mentioned above. X Adjacent input scaling factor S k-1 , scaling the adjacent inputs by a factor of S k-1 The corresponding input resolution L k-1 As an input resolution close to the image resolution X, so as to preprocess the image resolution from X to a similar input resolution L k-1 In this way, the electronic device does not need to execute k-1 downsampling layers before downsampling layer k, thereby saving computing resources and storage resources when the electronic device runs the neural network model and improving the operation rate.

[0066] In some embodiments, the image processing method provided by the present application can be applied to any electronic device with sufficient resources such as computing resources and storage resources, and can also be applied to any electronic device with limited resources such as limited computing resources or limited storage resources. Specifically, the applicable electronic device can be any terminal device such as a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a wearable device, a laptop computer, a vehicle-mounted smart terminal, etc. The electronic device can also be a physical server or a cloud device, such as an X86 server, an ARM server, etc., or a virtual machine (VM) based on a general physical server combined with network function virtualization (NFV) technology. The virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. The present application does not impose any restrictions on the specific type of the electronic device.

[0067] The following is based on Figure 4 The flowchart shown in FIG. 1 introduces the implementation process of the image processing method provided in the embodiment of the present application. The image processing method can be applied to electronic devices, such as any electronic devices such as the mobile phones and computers mentioned above. Figure 4 Specifically, the method comprises the following steps:

[0068] S401: Acquire a first image to be processed, wherein the first image has a first resolution.

[0069] In the present application, the first image may be the image to be processed that needs to be adjusted in resolution as mentioned in the present application; the first resolution may be the initial resolution X of the image to be processed as mentioned in the present application.

[0070] In some embodiments, the electronic device may receive an image to be processed, pre-process the image, and then input the image into a neural network model to obtain a target image with a target resolution.

[0071] S402: Adjust a first resolution of the first image to a first input resolution, wherein the first input resolution is: a processing resolution of a kth downsampling layer among N downsampling layers of a neural network model that processes the first image.

[0072] In the present application, the first input resolution may be the input resolution L mentioned above, which is close to the initial resolution X. k-1 , 1≤k≤N, k and N are both integers.

[0073] In some embodiments, the absolute value of the first difference between the first input resolution and the first resolution may be less than the absolute value of the second difference between the second input resolution and the first resolution. The second input resolution may be the processing resolution of the first downsampling layer. That is, when the first resolution differs greatly from the processing resolution of the first downsampling layer, and the first resolution differs little from the processing resolution of any other intermediate downsampling layer, the electronic device may adjust the resolution of the first image to the processing resolution of an intermediate downsampling layer, so that the electronic device does not need to run the neural network model from the first downsampling layer. In addition, the second input resolution may also be the processing resolution of any downsampling layer except the kth downsampling layer among the N downsampling layers. That is, the electronic device may select the first input resolution that differs the least from the first resolution from the N processing resolutions corresponding to the N downsampling layers, and adjust the first resolution of the first image to the first input resolution, so that the electronic device can run the least downsampling layer.

[0074] In some embodiments, the electronic device can pre-calculate and store the input resolution L that each downsampling layer in the neural network model can receive using the above formula (I). i The input resolution L that the first downsampling layer can accept 0 The ratio between them, that is, the input scaling factor S corresponding to each downsampling layer mentioned above is obtained i After the electronic device receives the image to be processed with the first resolution X through the above step S401, the electronic device can calculate the first resolution X and the input resolution L that can be received by the first downsampling layer through the above formula (II): 0 The ratio between them is the image scaling factor S mentioned above. XNext, the electronic device can select a first input resolution with a smaller absolute value of difference from the first resolution from the input resolutions that can be received by each downsampling layer based on each input scaling factor and the image scaling factor. For example, if the image scaling factor is greater than or equal to 1, the processing resolution of the first downsampling layer can be directly used as the first input resolution. If the image scaling factor is less than 1, the electronic device can obtain each input scaling factor S i and the image scaling factor S X Sort from high to low or from low to high, and then select the image scaling factor S by taking the value upward (selecting a larger input scaling factor) or taking the value downward (selecting a smaller input scaling factor). X Any adjacent input scaling factor S k-1 , enter the scaling factor S k-1 The corresponding processing resolution L k-1 That is, the first input resolution that is slightly different from the first resolution. In this way, the electronic device can determine the first input resolution that is slightly different from the first resolution.

[0075] In some embodiments, the electronic device may preprocess the image to be processed by a variety of image scaling methods such as nearest neighbor interpolation and bilinear difference, so that the resolution of the image to be processed is adjusted from the first resolution to the first input resolution. For example, if the first resolution is greater than the first input resolution, the electronic device may delete one or more pixels at a certain distance in the image to be processed, so that the number of pixels in the horizontal and vertical directions of the image to be processed is reduced to the number of pixels corresponding to the first input resolution, and the resolution of the image to be processed can be adjusted from the first resolution to the first input resolution. For another example, if the first resolution is less than the first input resolution, the electronic device may insert one or more pixels at a certain distance, and the inserted pixels may be directly copied from the pixels at adjacent positions, so that the number of pixels in the horizontal and vertical directions of the image to be processed is increased to the number of pixels corresponding to the first input resolution, and the resolution of the image to be processed can be adjusted from the first resolution to the first input resolution. It should be understood that in the process of preprocessing the resolution of the image to be processed from the first resolution to the first input resolution, the present application does not limit the specific implementation of the preprocessing method.

[0076] S403: Input the first image into the kth downsampling layer, and run the neural network model to obtain a second image with a second resolution.

[0077] In the present application, the second image may be a target image with a target resolution; the second resolution may be the target resolution of the target image obtained by the neural network model.

[0078] In some embodiments, the image processing method provided by the present application can be applied to any computer vision task that adjusts the image resolution. In addition, the applicable neural network model related to adjusting the image resolution is not limited to the super-resolution network of the encoder-decoder structure, but can also be applied to the neural network model including the downsampling layer and any other modules. Among them, when the neural network model is a super-resolution network, the second resolution of the second image obtained by the super-resolution network will be greater than the first resolution of the first image.

[0079] In some embodiments, the electronic device adjusts the resolution of the image to be processed to the first input resolution L in the above step S402. k-1 After that, the electronic device can also adjust other image data (such as the number of channels) of the image to be processed to the input data that can be received by the kth downsampling layer. Finally, the electronic device can input the image data to be processed with the first input resolution and satisfying the input condition of the kth downsampling layer into the kth downsampling layer, so that the electronic device sequentially loads and runs downsampling layer k, downsampling layer k+1...downsampling layer N, and M upsampling layers, thereby obtaining the final target image.

[0080] In this way, through the above-mentioned image processing method, while obtaining the target image, the electronic device does not need to execute the k-1 downsampling layers before the downsampling layer k, thereby saving computing resources and storage resources when the electronic device runs the neural network model, reducing memory usage time and increasing the computing speed.

[0081] In other embodiments, for the image input data that can be received by each downsampling layer, in addition to the image resolution that needs to meet the input resolution L that can be received by each downsampling layer, i (Width W i ×Height H i ), the number of image channels (also called “channel feature dimension”) also needs to satisfy the input channel feature dimension C that each downsampling layer can receive. i Among them, the channel feature dimension can be used to represent the number of feature maps that make up the image. For example, the channel feature dimension corresponding to image A is P, which can mean that image A is formed by stacking and combining P feature maps, and each feature map contains part of the information of image A, such as edge information, texture information, color information, etc.

[0082] Therefore, the image resolution is preprocessed from X to the closest first input resolution L k-1 After that, when the resolution is L k-1 Before the image data with a resolution of L is input to the kth downsampling layer, k-1 The channel feature dimension of the image is adjusted to the channel feature dimension C that the kth downsampling layer can acceptk-1 Finally, the adjusted image data (resolution L k-1 , the channel feature dimension is C k-1 The image data) is input into the kth downsampling layer and processed by the neural network model.

[0083] Among them, in the super-resolution network model, since the channel feature dimension of the image without adjusting the channel feature dimension is usually 3, and each downsampling layer of the super-resolution network will increase the channel feature dimension of the image. Therefore, N channel transformation modules (for example, neural network processing modules such as 1×1 convolution) can be added to the super-resolution network, and each channel transformation module corresponds to each downsampling layer one by one, which can be used to adjust the channel feature dimension of the image to the channel feature dimension that the corresponding downsampling layer can receive. For example, a channel transformation module can be connected in front of each downsampling layer of the super-resolution network, and the image resolution can be preprocessed from X to a similar input resolution L. k-1 (i.e., the processing resolution of the kth downsampling layer), the image to be processed needs to be input into the channel transformation module connected to the kth downsampling layer, and the channel feature dimension of the image to be processed is adjusted to the channel feature dimension C that the kth downsampling layer can accept through the channel transformation module. k-1 Then the adjusted image data (i.e., the resolution is L k-1 , the channel feature dimension is C k-1 The image data) is input into the kth downsampling layer for processing, so that the super-resolution network can be run to obtain the target image with the target resolution.

[0084] Specifically, taking the channel transformation module as a 1×1 convolution as an example, Figure 5 The schematic diagram of the structure of the super-resolution network after adding the channel transformation module is shown. Figure 5 In the super-resolution network shown in FIG. 1 , each downsampling layer can be described as an encoder module. For example, downsampling layer 1 can be described as encoder 1, and downsampling layer 2 can be described as encoder 2. Similarly, the encoder part in the super-resolution network can be divided into N encoder modules based on the N downsampling layers in the encoder part. For example, the above Figure 3 The encoder part in the current super-resolution network is shown to be divided into three encoder modules. It should be understood that for ease of illustration, Figure 5 Only 4 encoder modules, 4 upsampling layers in the decoder part, and 4 channel conversion modules corresponding to the 4 encoder modules are shown. In fact, more or fewer encoder modules, upsampling layers, and channel conversion modules corresponding to the encoder modules may be included, and this application does not limit this.

[0085] Among them, Figure 5In the example, encoder 1, encoder 2, encoder 3, and encoder 4 may correspond to a 1×1 convolution respectively, which is used to increase the channel feature dimension of the image to a dimension that can be received by the corresponding encoder module.

[0086] For example, refer to Figure 5 As shown, the encoder 1 can receive image data in the format of 3×H×W (i.e., channel feature dimension×height×width) as input data. Therefore, when the electronic device determines the image resolution X1 (height H X1 ×Width W X1 ) is close to the input resolution H×W of encoder 1, the electronic device can first preprocess the resolution of the image to the input resolution of encoder 1. Then, through the 1×1 convolution connected to encoder 1, the channel feature dimension of the preprocessed image is adjusted to the input channel feature dimension 3 of encoder 1. Finally, the electronic device can input the adjusted image data (that is, the image data with a channel feature dimension and resolution of 3×H×W) into encoder 1, and sequentially pass encoders 1 to 4 to perform operations such as resolution reduction and channel feature dimension increase. For example, the channel feature dimension and resolution of the image can be adjusted from 3×H×W to C×H / 2×W / 2 (C>3), 2C×H / 4×W / 4, 4C×H / 8×W / 8, and 8C×H / 16×W / 16 through encoders 1 to 4. Encoder 4 then sends the output data in the format of 8C×H / 16×W / 16 to the decoder part, and the image data is sequentially increased in resolution and reduced in channel feature dimension through each upsampling layer in the decoder. For example, the channel feature dimension and resolution of an image can be adjusted from 8C×H / 16×W / 16 to 4C×H / 4×W / 4, 2C×H / 2×W / 2, C×H×W, and 3×2H×2W through the four upsampling layers in the decoder part. X1 ×W X1 The low-resolution image is super-resolved to obtain a high-resolution image with a resolution of 2H×2W.

[0087] In other embodiments, Figure 5 In the super-resolution network shown, the electronic device can also start running the super-resolution network from encoder 2, encoder 3 or encoder 4, thereby reducing the load of running some encoders and saving running resources.

[0088] For example, the encoder 2 may receive image data in the format of C×H / 2×W / 2 (i.e., channel feature dimension×height×width) as input data. If the image resolution X2 (height H X2 ×Width W X2) is close to the input resolution H / 2×W / 2 of encoder 2. The electronic device can first preprocess the resolution of the image to a resolution of H / 2×W / 2. Then, through a 1×1 convolution connected to encoder 2, the channel feature dimension of the preprocessed image is upgraded to the input channel feature dimension C of encoder 2. Finally, the electronic device can input the image data with channel feature dimension and resolution C×H / 2×W / 2 into encoder 2, and then super-resolution is performed through encoder 2, encoder 3, encoder 4, and the decoder part in turn to obtain the final high-resolution image. In this way, the electronic device can convert the resolution of H to 1 without loading and running encoder 1. X2 ×W X2 The low-resolution image is super-processed to a high-resolution image with a resolution of 2H×2W.

[0089] Similarly, for another example, the encoder 3 may receive image data in the format of 2C×H / 4×W / 4 (i.e., channel feature dimension×height×width) as input data. If the image resolution X3 (height H X2 ×Width W X2 ) is close to the input resolution H / 4×W / 4 of encoder 3. The electronic device can first preprocess the resolution of the image to a resolution of H / 4×W / 4. Then, through a 1×1 convolution connected to encoder 3, the channel feature dimension of the preprocessed image is upgraded to the input channel feature dimension 2C of encoder 3. Finally, the electronic device can input the channel feature dimension and the image data with a resolution of 2C×H / 4×W / 4 into encoder 2, and then perform super-resolution processing through encoder 3, encoder 4, and the decoder part in turn to obtain the final high-resolution image. In this way, the electronic device can convert the resolution of H to 2 without running encoder 1 and encoder 2. X3 ×W X3 The low-resolution image is super-processed to a high-resolution image with a resolution of 2H×2W.

[0090] Among them, 1×1 convolution is a convolution operation in the neural network model, which can increase or decrease the channel feature dimension of the image without changing the image resolution. Corresponding to different encoder modules, the 1×1 convolution connected to the encoder module can be determined based on the channel feature dimension that the encoder module can receive. For example, if the channel feature dimension that the encoder module can receive is y, the connected 1×1 convolution is 1×1×y, which means that after the image is processed by the 1×1 convolution, the width and height of the image remain unchanged, and the channel feature dimension of the image becomes y. In addition, any other neural network layer that can adjust the channel feature dimension of the image can be used as a channel transformation module, and this application does not limit this.

[0091] It can be understood that a downsampling layer can include multiple neural network layers such as pooling layers and convolution layers, so as to realize the functions of reducing the resolution of the downsampling layer and increasing the channel feature dimension. Therefore, the number of parameters and the amount of calculation of a channel transformation module (such as 1×1 convolution) are much lower than the number of parameters and the amount of calculation of a downsampling layer. Therefore, based on the image processing method provided by the present application, if the image resolution X is close to the input resolution of a downsampling layer in the middle of the super-resolution network, the electronic device can only load and run one channel transformation module and part of the downsampling layer when running the super-resolution network. Compared with loading and running all downsampling layers, the image processing method provided by the present application can reduce memory usage and improve processing efficiency.

[0092] The following can be based on Figure 6 The flowchart shown briefly describes the process of super-resolution of an image in combination with the image processing method provided in this application.

[0093] Specifically, Figure 6 As shown, after receiving a low-resolution image (601) with a resolution of X, the electronic device can first calculate the image scaling factor S by the above formula (II): X , and the input scaling factor S corresponding to each encoder module in the pre-stored super-resolution network i Select a similar input scaling factor S from k-1 (602). Next, the electronic device needs to determine the image scaling factor S X Is it equal to the input scaling factor S? k-1 (603). If yes, it means that the image resolution X is equal to the input scaling factor S k-1 The corresponding input resolution L k-1 If not, the image resolution X needs to be scaled (604). That is, when there are multiple input resolutions and corresponding input scaling factors (605), the electronic device needs to scale the image resolution from X to the input scaling factor S. k-1 The corresponding input resolution L k-1 (6051). For example, encoder 1 corresponds to input resolution L 0 and input scaling factor S 0 , encoder 2 corresponds to input resolution L 1 and input scaling factor S 1 , encoder k corresponds to input resolution L k-1 and input scaling factor S k-1 , encoder N corresponds to input resolution L N-1 and input scaling factor S N-1 , the electronic device needs to select the image scaling factor S X Approximate input scaling factor S k-1, and then scale the image resolution X to the input scaling factor S k-1 The corresponding input resolution L k-1 .

[0094] Next, the electronic device inputs the image data to the input resolution L k-1 Before the corresponding encoder k, the image to be processed needs to be input into the channel transformation module connected to the encoder k, and the channel feature dimension of the image to be processed is adjusted to the input channel feature dimension C that the encoder k can receive through the channel transformation module. k-1 (606). Finally, the electronic device can process the processed image data (i.e., the resolution is L k-1 , the channel feature dimension is C k-1 The image data of the image is input into the super-resolution network for processing (607) to obtain a high-resolution image with a resolution of H (608). When the electronic device runs the super-resolution network, it does not need to load the k-1 encoder modules (e.g., encoder 1, encoder 2...encoder k-1) before running encoder k, and only needs to load the encoder modules (e.g., encoder k, encoder k+1...encoder N) after running encoder k (including encoder k) and the decoder part, thereby saving loading resources of the electronic device.

[0095] In this way, through the above method, when super-resolution processing is performed on an image, the computing resources and storage resources of the electronic device when running the super-resolution network can be saved, the memory occupied during operation can be reduced, and the processing efficiency can be improved.

[0096] Combine the following Figure 7 The flowchart shown in FIG. 1 is a detailed description of the specific implementation process of the image processing method provided by the present application. The image processing method can be applied to electronic devices, such as any electronic devices such as the mobile phones and computers mentioned above. Figure 7 Specifically, the method comprises the following steps:

[0097] S701: Divide the encoder part of the super-resolution network into N encoder modules.

[0098] In an embodiment of the present application, the electronic device can divide the encoder part in the super-resolution network into N encoder modules based on the N downsampling layers in the encoder part, and add a channel conversion module in the super-resolution network. Figure 5 As shown, each encoder module can be connected with a 1×1 convolution to adjust the channel feature dimension of the image to be processed.

[0099] In some embodiments, since the initial channel feature dimension of the image to be processed is 3, and the input channel feature dimension of the first encoder module in the super-resolution network is usually also 3, the first encoder module may not be connected to the channel transformation module. That is, among the N encoder modules, only N-1 channel transformation modules may be added.

[0100] In some embodiments, the divided N encoder modules can be recorded as encoder 1, encoder 2, ... encoder N (N ≥ 1) in sequence. The corresponding input resolutions can be recorded as input L 0 , L 1 , L 2 …L k-1 , L k …L N-1 (k ≥ 0), the corresponding output resolution can be recorded as L 1 , L 2 …L k , L k+1 …L N That is, the output resolution of the previous encoder module is the input resolution of the next encoder module.

[0101] S702: Calculate the input resolution L of each encoder module i With the super-resolution network input resolution L 0 Input scaling factor between .

[0102] In the embodiment of the present application, after obtaining N encoder modules and corresponding input resolutions through the above step S701, the electronic device can calculate the input resolutions L of each encoder module through the above formula (1). i With the super-resolution network input resolution L 0 Input scaling factor S between i .

[0103] In some embodiments, the electronic device may pre-store the calculated input scaling coefficients of each encoder module. When receiving a low-resolution image to be processed, the electronic device may directly read the stored input scaling coefficients for super-resolution processing, thereby improving processing efficiency and saving processing time. Alternatively, the electronic device may not store the input scaling coefficients, and when receiving a low-resolution image to be processed, directly calculate the input scaling coefficients for super-resolution processing using the above formula (I). This application does not limit this.

[0104] S703: For the resolution X of the image to be processed, calculate the image resolution X and the super-resolution network input resolution L 0 The image scaling factor between .

[0105] In the embodiment of the present application, when the electronic device receives the low-resolution image to be processed, the resolution X and the super-resolution network input resolution L can be calculated by the above formula (II). 0 The image scaling factor S between X .

[0106] In some embodiments, if the electronic device does not pre-store the input scaling coefficients corresponding to each encoder module, the electronic device may simultaneously execute step S702 and step S703 to obtain each input scaling coefficient and the image scaling coefficient; or, the electronic device may first execute step S702 to obtain each input scaling coefficient, and then execute step S703 to obtain the image scaling coefficient; or, the electronic device may first execute step S703 to obtain the image scaling coefficient, and then execute step S702 to obtain each input scaling coefficient. It should be understood that the present application does not limit the execution order of step S702 and step S703.

[0107] S704: Compare the image scaling factor with each input scaling factor, and select the input scaling factor S that is close to the image scaling factor. k-1 .

[0108] In the embodiment of the present application, after obtaining each input scaling factor and the image scaling factor, the electronic device can select an input scaling factor S that is close to the image scaling factor from each input scaling factor. k-1 .

[0109] In some embodiments, since the input resolution of each encoder module is also the output resolution of the previous encoder module, and each encoder module of the super-resolution network is a process of reducing the image resolution, the input resolution of each encoder module is reduced in sequence. Therefore, the electronic device calculates the input resolution L of each encoder module by the above formula (I). i The input resolution L of the entire super-resolution network 0 Input scaling factor S between i Among them, only S 0 is 1, and the other input scaling factors S i Both are less than 1.

[0110] Therefore, the electronic device selects the input scaling factor S k-1 When the image scaling factor S X If it is greater than or equal to 1, the similar input scaling factor S can be directly determined. k-1 For S 0 If the image scaling factor S X If it is less than 1, it is necessary to adjust the input scaling factor (S 0 , S 1 , S 2 …S N-1) and the image scaling factor are sorted from large to small or from small to large, and then a similar input scaling factor S is selected by taking the value upward (selecting a larger input scaling factor) or taking the value downward (selecting a smaller input scaling factor). k-1 For example, after sorting from largest to smallest, it is S 0 >S 1 >…>S k-1 >S X >S k >…>S N-1 , select the image scaling factor S X Adjacent input scaling factor S k-1 or S k .

[0111] In some embodiments, the electronic device selects the input zoom factor S k-1 When the image is scaled by the factor S X With each input scaling factor S i Subtract, and select the input scaling factor corresponding to the minimum difference as the selected similar input scaling factor S k-1 . This application selects a similar input scaling factor S k-1 The specific implementation method is not limited.

[0112] S705: Based on the selected input scaling factor S k-1 , the resolution of the image to be processed is scaled and the channel feature dimension is transformed.

[0113] In the embodiment of the present application, the electronic device selects an input scaling factor S that is close to the image scaling factor through the above step S704. k-1 After that, the corresponding input resolution L k-1 As an input resolution close to the image resolution X.

[0114] In some embodiments, the electronic device may not perform the above steps S701 to S704. The electronic device may also directly compare the image resolution X with each input resolution L i Subtract the input resolution corresponding to the minimum difference and select it as the selected input resolution close to the image resolution X. This application does not limit this.

[0115] In some embodiments, the electronic device selects an input resolution L close to the image resolution X. k-1 After that, the electronic device first needs to preprocess the image resolution from X to the input resolution L k-1 Among them, the electronic device can pre-process the image to be processed by various image scaling methods such as nearest neighbor interpolation and bilinear difference, so that the resolution of the image to be processed is pre-processed from X to the input resolution Lk-1 For details, please refer to the above Figure 4 The process is described in step S402 and will not be described again here.

[0116] In some embodiments, the electronic device also needs to determine whether the initial channel feature dimension of the image is the same as the input channel feature dimension of the encoder k. If they are the same, the resolution can be directly set to the input resolution L k-1 The image data is input into the encoder k. If different, the electronic device also needs to input the image data into the channel conversion module (such as the above Figure 5 In the 1×1 convolution shown in , the channel feature dimension of the image is adjusted through the channel transformation module, and then the image data is input into the encoder k.

[0117] In some implementations, the electronic device may preprocess the image resolution first, and then adjust the channel feature dimension of the image through the channel transformation module; or the electronic device may first adjust the channel feature dimension of the image through the channel transformation module, and then preprocess the image resolution. This application does not limit this.

[0118] S706: Perform super-resolution processing on the image to be processed by running the entire super-resolution network or running some modules of the super-resolution network to obtain a high-resolution image.

[0119] In the embodiment of the present application, after the image resolution and channel feature dimension of the image to be processed are processed through the above step S705, the processed image data (i.e., the resolution is L k-1 , the channel feature dimension is C k-1 The image data of the kth encoder module is input to the kth encoder module, and the super-resolution network is run to obtain a high-resolution image. For example, if the super-resolution network includes N downsampling layers and M upsampling layers, the processed image data can be input to the kth downsampling layer, and then the N-k+1 downsampling layers after the kth downsampling layer (including the kth downsampling layer) are run in sequence to obtain a resolution lower than the processing resolution L of the kth downsampling layer. k-1 Then, the intermediate image data is input into the first upsampling layer, and the M upsampling layers after the first upsampling layer (including the first downsampling layer) are run in sequence to obtain a target image with a target resolution greater than the processing resolution L of the kth downsampling layer. k-1 .

[0120] In some embodiments, since k≥1 and k is an integer, when k is 1, the electronic device needs to run the entire super-resolution network to perform super-resolution processing on the image to be processed to obtain a high-resolution image. When k>1, the electronic device only needs to load and run some modules in the super-resolution network to perform super-resolution processing on the image to be processed to obtain a high-resolution image.

[0121] Thus, based on the image processing method provided by the present application, if the image resolution X is close to the input resolution of a certain encoder module in the middle of the super-resolution network, the electronic device can only load and run one channel conversion module and part of the encoder module when running the super-resolution network. Compared with loading and running all encoder modules, the image processing method provided by the present application can reduce memory usage and improve processing efficiency.

[0122] An embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0123] An embodiment of the present application provides a computer program product, including: computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0124] An embodiment of the present application also provides an electronic device, which includes: a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the image processing method mentioned in the present application.

[0125] The image processing method of the present application can be executed by an electronic device, wherein the electronic device can be any electronic device such as a server, a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a laptop computer, etc. The form of the electronic device is not specifically limited in the embodiments of the present application.

[0126] like Figure 8 As shown, taking electronic device 100 as an example, a hardware structure diagram of an electronic device according to an embodiment of the present application is exemplified.

[0127] like Figure 8As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, a magnetic sensor 180B, an acceleration sensor 180C, an infrared sensor 180D, a temperature sensor 180E, a touch sensor 180F, an ambient light sensor 180G, a light sensor 180H, etc.

[0128] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0129] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), a controller, a video codec, a digital signal processor (DSP), etc. Different processing units may be independent devices or integrated into one or more processors.

[0130] In an embodiment of the present application, the processor 110 may be used to execute the image processing method mentioned in the present application. For example, the processor 110 may calculate the input scaling factor corresponding to the input resolution of each encoder module, and then calculate the image scaling factor corresponding to the image resolution. Finally, an input scaling factor close to the image scaling factor is selected from multiple input scaling factors, and the image resolution is adjusted to the input resolution corresponding to the close input scaling factor.

[0131] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0132] The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0133] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0134] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the screen 194, the camera 193, and the wireless communication module 160. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0135] The wireless communication function of the electronic device 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, and a modulation and demodulation processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, antenna 1 can be used in combination with a tuning switch.

[0136] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0137] The screen 194 is used to display images, videos, etc. The screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), etc. In some embodiments, the electronic device 100 may include 1 or N screens 194, where N is a positive integer greater than 1.

[0138] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0139] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0140] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0141] The gyro sensor 180A may be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180A.

[0142] The magnetic sensor 180B includes a Hall sensor and can be used to detect the strength of the surrounding magnetic field.

[0143] The acceleration sensor 180C can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0144] The infrared sensor 180D is used to detect, receive and analyze infrared radiation signals.

[0145] The temperature sensor 180E is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180E to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180E exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180E to reduce power consumption and implement thermal protection.

[0146] The touch sensor 180F is also called a "touch control device". The touch sensor 180F can be set on the screen 194, and the touch sensor 180F and the screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180F is used to detect touch operations acting on or near it.

[0147] The ambient light sensor 180G is used to sense the brightness of ambient light.

[0148] The light sensor 180H can be used to detect light intensity. It has high sensitivity and fast response speed. It can be used in various lighting environments and provide real-time feedback on light conditions.

[0149] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0150] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0151] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0152] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0153] The embodiments disclosed in the present application may be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0154] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0155] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0156] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0157] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0158] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "include one" do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0159] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: Acquire a first image to be processed, wherein the first image has a first resolution; adjusting a first resolution of the first image to a first input resolution, The first input resolution is: the processing resolution of the kth downsampling layer among the N downsampling layers of the neural network model that processes the first image, 1<k≤N, k and N are both integers, and An absolute value of a first difference between the first input resolution and the first resolution is smaller than an absolute value of a second difference between a second input resolution and the first resolution, and the second input resolution is a processing resolution of a first downsampling layer of the N downsampling layers, wherein: The first input resolution is determined in a manner including: based on a ratio between a processing resolution of each of the N downsampling layers and a processing resolution of the first downsampling layer, and a ratio between the first resolution and the processing resolution of the first downsampling layer, selecting a first input resolution from the processing resolutions of the downsampling layers, the absolute value of the difference between the first resolution and the processing resolution of the first downsampling layer being smaller than the absolute value of the second difference; The first image is input into the kth downsampling layer, and the neural network model is run to obtain a second image with a second resolution.

2. The method according to claim 1, characterized in that: The neural network model includes a super-resolution network, and the second resolution is greater than the first resolution.

3. The method according to claim 2, characterized in that The selecting, based on a ratio between a processing resolution of each of the N downsampling layers and a processing resolution of the first downsampling layer, and a ratio between the first resolution and the processing resolution of the first downsampling layer, a first input resolution whose absolute value of difference with the first resolution is smaller than the absolute value of the second difference from the processing resolutions of the downsampling layers, comprises: determining a ratio between a processing resolution of each of the N downsampling layers and a processing resolution of the first downsampling layer, obtaining and storing an input scaling factor of each downsampling layer, and, Determine a ratio between the first resolution and a processing resolution of the first downsampling layer to obtain an image scaling factor; Based on the input scaling factor of each downsampling layer and the image scaling factor, a first input resolution whose absolute value of difference with the first resolution is smaller than the absolute value of the second difference is selected from the processing resolutions of the downsampling layers.

4. The method according to claim 3, characterized in that The selecting, based on the input scaling factor of each downsampling layer and the image scaling factor, a first input resolution whose absolute value of difference with the first resolution is smaller than the absolute value of the second difference from the processing resolutions of each downsampling layer comprises: Corresponding to the image scaling factor being greater than or equal to 1, determining the first input resolution as a processing resolution of the first downsampling layer; Corresponding to the image scaling factor being less than 1, the input scaling factors and the image scaling factors of the downsampling layers are sorted, and a processing resolution corresponding to any input scaling factor adjacent to the image scaling factor is selected as the first input resolution.

5. The method according to claim 2, characterized in that: The super-resolution network includes N channel transformation modules, each of the N channel transformation modules corresponds to each of the N downsampling layers, and, The channel transformation module is used to adjust the channel feature dimension of the first image to the processing channel feature dimension of the downsampling layer corresponding to the channel transformation module.

6. The method according to claim 5, characterized in that Inputting the first image into the kth downsampling layer, and running the neural network model to obtain a second image with a second resolution, comprising: Inputting the first image into a channel transformation module corresponding to the k-th downsampling layer to obtain a first image whose channel feature dimension is a first channel feature dimension, where the first channel feature dimension is a processing channel feature dimension of the k-th downsampling layer; A first image having a first input resolution and a first channel feature dimension is input into the kth downsampling layer of the super-resolution network, and the super-resolution network is run to obtain a second image having a second resolution.

7. The method according to any one of claims 2 to 6, characterized in that The super-resolution network further includes M up-sampling layers, and the first image is input into the kth down-sampling layer, and the neural network model is run to obtain a second image with a second resolution, including: Inputting the first image into the kth downsampling layer of the super-resolution network, and running N-k+1 downsampling layers to obtain a third image with a third resolution, wherein the third resolution is smaller than the first input resolution; The third image is input into the first upsampling layer of the super-resolution network, and the M upsampling layers are run to obtain a second image with the second resolution, where the second resolution is greater than the first input resolution.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the image processing method described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the image processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that include: A computer instruction, when the computer instruction is executed on an electronic device, causes the electronic device to execute the image processing method according to any one of claims 1 to 7.

Images