Image processing method, electronic device, and readable storage medium

Abstract

The application relates to the technical field of image processing, and discloses an image processing method, an electronic device and a readable storage medium. In the method, a low-resolution image is input into an image processing model, and the image processing model is used for image super-resolution processing on the low-resolution image. In the process of image super-resolution processing on the low-resolution image by the image processing model, a first hidden variable can be obtained by encoding the low-resolution image through a first encoding layer, a second hidden variable can be obtained by performing reverse diffusion processing on the first hidden variable through a diffusion network layer, N third hidden variables can be obtained by performing convolution on the second hidden variable through N convolution kernels in an interleaving network layer, the N third hidden variables are spliced into a fourth hidden variable, and a high-resolution image can be obtained by decoding the fourth hidden variable through a decoding layer. In this way, the low-resolution image can be reconstructed into multiple high-resolution images with different resolutions by adjusting the value of N.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a readable storage medium. Background Technology

[0002] Image super-resolution models can reconstruct a low-resolution (LR) image (an image with fewer pixels) into a larger high-resolution (HR) image (an image with more pixels) to increase the image's sharpness and detail.

[0003] Currently, well-trained image super-resolution models only support reconstructing a low-resolution image of one resolution into a high-resolution image of the same resolution, but do not support reconstructing a low-resolution image of one resolution into multiple high-resolution images of different resolutions. Therefore, it is difficult to meet users' needs to reconstruct a single low-resolution image into multiple high-resolution images of different resolutions. Summary of the Invention

[0004] Some embodiments of this application provide an image processing method, an electronic device, and a storage medium. The following describes this application from multiple aspects, and the embodiments and beneficial effects of the following aspects can be referred to each other.

[0005] In a first aspect, this application provides an image processing method for an electronic device, the method comprising: acquiring a first image at a first resolution; performing a first encoding process on the first image to obtain a first latent variable, the first latent variable representing image features of the first image; performing a diffusion process on the first latent variable to obtain a second latent variable, the diffusion process being used to remove noise from the first image; performing a scaling process on the values ​​in the second latent variable to obtain a plurality of third latent variables, and concatenating the plurality of third latent variables into a fourth latent variable, the size of the fourth latent variable being larger than the size of the first latent variable; and decoding the fourth latent variable to obtain a second image at a second resolution, wherein the second resolution is larger than the first resolution.

[0006] In this method, the second resolution of the second image is determined by the size of the fourth latent variable, which in turn is determined by the size and number of the third latent variable. The size of the third latent variable is determined by the size of the first latent variable of the second latent variable, which in turn is determined by the first resolution of the first image. When the first resolution is fixed, the size of the third latent variable is also fixed. In this case, the second resolution of the second image is determined solely by the number of the third latent variable, meaning that the second resolution can be adjusted by changing the number of the third latent variable. Thus, this method can reconstruct a low-resolution image of a certain resolution into multiple high-resolution images of different resolutions.

[0007] In some implementations, scaling the values ​​in the second latent variable yields multiple third latent variables, including: convolving the second latent variable with multiple 1D convolution kernels with different parameters to obtain multiple third latent variables.

[0008] In some implementations, concatenating multiple third latent variables into a fourth latent variable includes: alternating the row data of multiple third latent variables in the column direction, and alternating the column data of multiple third latent variables in the row direction to obtain the fourth latent variable.

[0009] For example, a second latent variable of size 512×512 can be convolved by two 1×1 2D convolution kernels to obtain two third latent variables of size 512×512. Then, the two third latent variables of size 512×512 are concatenated to form a fourth latent variable of size 1024×1024.

[0010] It is understandable that the fourth latent variable obtained after interleaving and splicing includes more image features than the second latent variable. This is equivalent to adding more image details to the high-resolution image obtained after decoding the fourth latent variable, or it is equivalent to interpolating pixels around a certain pixel in the low-resolution image to be processed, such as pixel A. The pixel value of the interpolated pixel is not taken from the pixel value of pixel A or the original pixel value of the pixels around pixel A, thus adding more image information.

[0011] In some embodiments, the method further includes: performing a second encoding process on the first image to obtain a query vector, wherein the query vector is used to characterize the semantic content of the first image; and performing a diffusion process on the first latent variable to obtain a second latent variable, including: performing a backdiffusion process on the first latent variable based on the query vector to obtain the first latent variable.

[0012] In some implementations, the second encoding process of the first image to obtain the query vector includes: dividing the first image into multiple regions, performing linear mapping processing on the pixels of the multiple regions to obtain multiple feature vectors, and concatenating the multiple feature vectors into the query vector.

[0013] In some implementations, the first latent variable is obtained by backdiffusion processing based on the query vector, including: weighting the query vector to obtain an attention score; and performing backdiffusion processing on the first latent variable based on the attention score to obtain the first latent variable.

[0014] In some implementations, an image processing model is deployed on the electronic device, the image processing model including an encoding layer, a diffusion network layer, an interleaving network layer, and a decoding layer.

[0015] The first image is subjected to a first encoding process to obtain a first latent variable, including:

[0016] The first latent variable is obtained by performing a first encoding process on the first image through an encoding layer;

[0017] The second latent variable is obtained by performing diffusion processing on the first latent variable, including:

[0018] The second hidden variable is obtained by diffusion processing of the first hidden variable through a diffusion network layer;

[0019] The values ​​in the second latent variable are scaled to obtain multiple third latent variables, and these multiple third latent variables are concatenated to form a fourth latent variable, including:

[0020] Multiple third hidden variables are obtained by scaling the values ​​in the second hidden variable through a diffusion network layer, and then the multiple third hidden variables are concatenated into a fourth hidden variable.

[0021] Decoding the fourth latent variable yields a second image at a second resolution, including:

[0022] The second image is obtained by decoding the fourth hidden variable through the decoding layer.

[0023] In some implementations, the interleaved network layer includes M one-dimensional convolutional kernels with different parameters, where M is a positive integer. The second hidden variable can be convolved by N one-dimensional convolutional kernels to obtain N third hidden variables, where N is a positive integer less than M.

[0024] In some implementations, the encoding layer includes a first encoder, and the diffusion network layer includes a diffusion model, which does not include an attention network layer.

[0025] A first encoder is used to perform a first encoding process on a first image to obtain a first latent variable; a diffusion model is used to perform a back-diffusion process on the first latent variable to obtain a second latent variable.

[0026] In some implementations, the encoding layer includes a first encoder and a second encoder, and the diffusion network layer includes a cross-fused multi-head attention diffusion model, wherein the cross-fused multi-head attention diffusion model includes an attention network layer, wherein...

[0027] A first encoder is used to perform a first encoding process on a first image to obtain a first latent variable;

[0028] The second encoder is used to perform a second encoding process on the first image to obtain a query vector, wherein the query vector is used to represent the semantic content of the first image;

[0029] A cross-fusion multi-head attention diffusion model is used to perform back-diffusion processing on the first latent variable based on the query vector to obtain the first latent variable.

[0030] In some implementations, the query vector can be weighted by an attention network layer to obtain an attention score, and the first latent variable can be backdiffused based on the attention score to obtain the first latent variable.

[0031] Secondly, embodiments of this application provide an electronic device, including a memory for storing instructions executable by one or more processors of the electronic device; and a processor, which, when executing the instructions in the memory, causes the electronic device to perform the method described in any embodiment of the first aspect of this application. The beneficial effects achievable in the second aspect can be referred to the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.

[0032] Thirdly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any embodiment of the first aspect. The beneficial effects achievable through this third aspect can be referenced to the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.

[0033] Fourthly, embodiments of this application provide a computer program product including computer program code. When the computer program code is run on a computer, it causes the computer to implement the method described in any embodiment of the first aspect. The beneficial effects achievable in this fourth aspect can be found in the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here. Attached Figure Description

[0034] Figure 1 Exemplary application scenarios of this application are shown;

[0035] Figure 2A A schematic diagram of the super-resolution model is shown.

[0036] Figure 2B A schematic diagram of the diffusion model is shown;

[0037] Figure 3 The diagram illustrates the principle of interleaving and splicing of interleaved network layers;

[0038] Figure 4 A schematic diagram of the image processing model is shown.

[0039] Figure 5 A schematic diagram is shown illustrating the reconstruction of a low-resolution image into a high-resolution image of different resolutions;

[0040] Figure 6A A schematic diagram of the structure of image processing model A is shown;

[0041] Figure 6B A schematic diagram of the structure of image processing model B is shown;

[0042] Figure 6C This diagram illustrates image super-resolution processing of low-resolution images using image processing model B.

[0043] Figure 7 A schematic diagram of the attention layer is shown;

[0044] Figure 8 Another structural diagram of image processing model B is shown;

[0045] Figure 9 A schematic flowchart of the image processing method provided in an embodiment of this application is shown;

[0046] Figure 10 A schematic diagram of the electronic device is shown. Detailed Implementation

[0047] This application provides an image processing model, method, electronic device, and storage medium.

[0048] For ease of description, this article uses the format "width × height" to describe image resolution, with the unit being pixels. For example, an image with a resolution of 512 × 64 has a width of 512 pixels and a height of 64 pixels.

[0049] Terminology explanation;

[0050] (1) Upsampling

[0051] Upsampling refers to increasing the resolution of an image. When upsampling an original image, the pixels are "interpolated," meaning new pixels are added between existing pixels to achieve a higher resolution. Upsampling methods include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, transposed convolution, and unpooling.

[0052] (2) Downsampling

[0053] Downsampling refers to reducing the resolution of an image. Downsampling removes some pixels from the original image, resulting in a lower-resolution image. Downsampling methods include, but are not limited to, average pooling, max pooling, and Gaussian blur post-sampling.

[0054] Figure 1 This is an exemplary application scenario for this application.

[0055] refer to Figure 1 The electronic device 100 is equipped with a trained super-resolution model, which is used to reconstruct a 512×512 image into a 1024×1024 image. The user can use the super-resolution model deployed on the electronic device 100 to reconstruct a 512×512 image M1 into a 1024×1024 image M2.

[0056] Figure 2A A schematic diagram of the super-resolution model is shown below. (The following section combines...) Figure 2A The principle of the super-resolution model is explained.

[0057] refer to Figure 2A The super-resolution model consists of an encoder and a decoder. The encoder encodes the input low-resolution image, downsampling it during the encoding process, and outputs the corresponding latent variables. These latent variables are high-dimensional vectors representing image features, containing vector representations of details, texture, color, and other characteristic information. The decoder decodes the latent variables output by the encoder, upsampling them during the decoding process, and outputs the high-resolution image.

[0058] In some embodiments, a 512×512 resolution sample image and a 1024×1024 resolution label image can be used as image pairs to train the super-resolution model. This allows the trained super-resolution model to reconstruct a 1024×1024 resolution image from a 512×512 image. The sample image and label image are identical. Specifically, a 512×512 resolution sample image can be input into the super-resolution model. The super-resolution model processes the sample image and outputs a predicted image. A loss function is then calculated based on the difference between the predicted image and the label image, and the model parameters are adjusted according to the loss function. Training ends when the loss function converges, the super-resolution model achieves the expected result, or the number of training iterations reaches a preset number.

[0059] It is understandable that during the training process of the super-resolution model using the aforementioned sample and label images, the super-resolution model continuously learns how to insert new pixel information into the input image to improve its resolution. Thus, after the super-resolution model is trained, it can reconstruct a high-quality, high-resolution image from a low-resolution image.

[0060] Furthermore, after the super-resolution model is trained, the downsampling rate of the encoder and the upsampling rate of the decoder are fixed. Therefore, the resolution improvement factor after super-resolution processing of the input image is also fixed. For example, if the encoder's downsampling rate is 8x and the decoder's upsampling rate is 32x, then the trained super-resolution model can improve the image resolution by a fixed factor of 4. (Refer to the above.) Figure 1 After the super-resolution model is trained, it can be deployed on the electronic device 100, so that the user can use the super-resolution model deployed on the electronic device 100 to reconstruct the image M1 with a resolution of 512×512 into an image M2 with a resolution of 1024×1024, and the image M2 is clearer.

[0061] However, in some cases, users may want to reconstruct image M1 into a higher resolution image (such as 720×720, 650×650, etc.). The super-resolution model, trained with sample images of 512×512 resolution and label images of 1024×1024 resolution, can only increase the resolution of the input image by a factor of 4. This means the super-resolution model can only reconstruct a 512×512 image into a 1024×1024 image, and cannot reconstruct a 512×512 image into other high-resolution images. Therefore, it cannot meet the user's need to reconstruct a single 512×512 image into multiple high-resolution images of different resolutions.

[0062] In some embodiments, a 512×512 resolution sample image and other high-resolution (e.g., 720×720, 650×650, etc.) label images can be used as training pairs to retrain the super-resolution model. The trained super-resolution model is then redeployed to the electronic device 100 to reconstruct the 512×512 image M1 into other high-resolution images using the retrained super-resolution model on the electronic device 100. However, this sampling method, which reconstructs a single 512×512 image M1 into multiple high-resolution images of different resolutions, requires multiple training iterations of the super-resolution model and multiple model deployments. This consumes significant computational resources and time, and is relatively complex.

[0063] In other embodiments, upsampling methods such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, transposed convolution, and unpooling can be used to upsample image M1 at various sampling rates, resulting in high-resolution images of various resolutions (e.g., 720×720, 650×650, etc.). This allows image M1 to be reconstructed into other high-resolution images to meet user needs. However, directly upsampling an image simply inserts existing pixel information into the original image without adding new pixel information. This may result in a lack of detail in the high-resolution image, thus reducing image quality.

[0064] Figure 2B A schematic diagram of the diffusion model is shown.

[0065] refer to Figure 2B The diffusion model includes an encoder, a diffusion network layer, and a decoder.

[0066] It can be understood that a diffusion model (or "generative model") is a model that generates a corresponding image based on a specific input (such as image hint text, image, semantics, representation, etc.), including but not limited to noise-based diffusion models, variational diffusion models, transformation-based diffusion models, stable diffusion models, etc.

[0067] The diffusion network layer is used to diffuse the input data, a process that includes forward diffusion and backward diffusion. Forward diffusion adds noise to the input data, while backward diffusion removes it. During the training phase of the diffusion model, it first adds noise to the input data using forward diffusion, and then predicts and removes the noise (the noise added during forward diffusion) using backward diffusion. During training, the model's parameters can be continuously adjusted to improve the accuracy of the backward diffusion's noise prediction. Finally, in the usage phase, the trained diffusion model can remove noise from the input data using only the backward diffusion process.

[0068] It is understandable that diffusion models can be applied to scenarios such as text-to-image, image-to-image, image inpainting, image transformation, image style transfer, and image super-resolution.

[0069] For example, in image super-resolution scenarios, refer to Figure 2B A low-resolution image can be input into a diffusion model. The encoder of the diffusion model encodes the low-resolution image, transforming it from the image space (the space where image data exists) to the latent space (also called the latent space, which is a low-dimensional, continuous space, usually with Gaussian distribution properties). This yields the first latent variable (a matrix representing the features of the low-resolution image, such as color, edges, and details). The diffusion network layer of the diffusion model then diffuses the first latent variable (backdiffusion) to obtain the second latent variable. This process removes noise from the low-resolution image and reconstructs it. The decoder of the diffusion model then decodes the second latent variable, transforming it from the latent space back to the image space to obtain the corresponding high-resolution image. Since the size of the second latent variable is fixed after the diffusion model is trained, and the resolution of the high-resolution image output by the diffusion model is determined by the size of the second latent variable, the resolution of the high-resolution image is also fixed. Therefore, the diffusion model can only reconstruct a high-resolution image of a single resolution from a low-resolution image.

[0070] In view of this, this application provides an image processing method. In this method, the second latent variable in the aforementioned diffusion model is expanded into multiple third latent variables of varying numbers, and these multiple third latent variables are interleaved and stitched together to obtain fourth latent variables corresponding to high-resolution images of different resolutions. Specifically, different numbers of third latent variables result in different resolutions of the high-resolution images obtained after interleaving and stitching; furthermore, a larger number of third latent variables results in a higher resolution of the high-resolution image.

[0071] It is understandable that the fourth latent variable obtained after interleaving and splicing includes more image features than the second latent variable. This is equivalent to adding more image details to the high-resolution image obtained after decoding the fourth latent variable, or it is equivalent to interpolating pixels around a certain pixel in the low-resolution image to be processed, such as pixel A. The pixel value of the interpolated pixel is not taken from the pixel value of pixel A or the original pixel value of the pixels around pixel A, thus adding more image information.

[0072] Specifically, the values ​​of the second latent variables in the above diffusion model can be expanded or reduced to obtain multiple different third latent variables, and then these multiple third latent variables can be concatenated into a fourth latent variable. In some embodiments, the expansion or reduction of the values ​​of the second latent variables can be achieved using a 1-dimensional convolution kernel (a convolution kernel with a size of 1×1).

[0073] In some embodiments, the first latent variable of the low-resolution image can be diffused through the diffusion network layer in the diffusion model to obtain the second latent variable. Then, the second latent variable can be convolved through multiple 1D convolution kernels with different parameters to obtain multiple third latent variables. These multiple third latent variables are then interleaved and spliced ​​into a fourth latent variable. Finally, the fourth latent variable is decoded by a decoder to obtain the corresponding high-resolution image.

[0074] The resolution of the high-resolution image is determined by the size of the fourth latent variable, which in turn is determined by the number of the third latent variables. Therefore, the resolution of the high-resolution image can be adjusted by changing the number of the third latent variables, thus allowing a low-resolution image to be reconstructed into multiple high-resolution images of different resolutions.

[0075] It can be understood that interleaving and splicing refers to alternating the row data of multiple third latent variables in the column direction and alternating the column data of multiple third latent variables in the row direction to obtain the fourth latent variable.

[0076] For example, suppose the size of the second latent variable corresponding to a 2×2 resolution image is 2×2 (2 rows and 2 columns). The data at each position in the second latent variable represents the color, texture, edge, and other features of the corresponding pixel location in the image. Figure 3As shown, the second latent variable can be convolved with four 1×1 2D convolution kernels to obtain four 2×2 third latent variables. These four third latent variables are then interleaved and concatenated (alternating rows and columns) to form a fourth latent variable with a size of 4×4. Decoding the fourth latent variable then yields a 4×4 resolution image.

[0077] This application provides an image processing model for performing the above-described image processing method.

[0078] Figure 4 A schematic diagram of the image processing model provided in an embodiment of this application is shown.

[0079] refer to Figure 4 The image processing model includes an encoding layer, a diffusion network layer, an interleaving network layer, and a decoding layer. The interleaving network layer includes M 1×1 2D convolutional kernels, and the values ​​of these M convolutional kernels are different.

[0080] In some embodiments, a low-resolution image of the first resolution can be input into a trained image processing model, and the trained image processing model can perform image super-resolution processing on the low-resolution image to obtain a high-resolution image of the second resolution, wherein the second resolution is greater than the first resolution.

[0081] In the image processing model's processing of a low-resolution image (first resolution), the low-resolution image is encoded using an encoding layer to obtain a first latent variable (a feature vector representing the image's texture, edges, color, and other features). Then, a diffusion network layer denoises the first latent variable to obtain a second latent variable. The denoised second latent variable possesses more detailed features than the first latent variable, which improves the image resolution. Next, N 1×1 2D convolutional kernels in an interleaved network layer are used to convolve the second latent variable, resulting in N third latent variables. These N third latent variables are then concatenated into a larger fourth latent variable, where N is a positive integer less than or equal to M. Finally, a decoding layer decodes the fourth latent variable to obtain the high-resolution image (second resolution).

[0082] It's understandable that the larger the size of the fourth latent variable, the more feature information it contains about the image pixels, resulting in a higher resolution image after decoding. Since the fourth latent variable is obtained by concatenating N third latent variables, the larger the value of N, the larger the size of the fourth latent variable, and correspondingly, the higher the resolution of the high-resolution image obtained after decoding. Therefore, when processing low-resolution images using an image processing model, the resolution of the high-resolution image output by the model can be adjusted by changing the value of N. In this way, a low-resolution image can be reconstructed into multiple high-resolution images of different resolutions using an image processing model.

[0083] In some embodiments, during the training phase of the image processing model, low-resolution images of a certain resolution (e.g., 512×512) can be used as sample images, and high-resolution images of various different resolutions (e.g., 1024×1024, 720×720, 650×650, etc.) can be used as label images to train the image processing model. Furthermore, during training, the value of parameter N in the interleaved network module is adjusted to correspond to the resolution of the label images. After training, the trained image processing model can input a low-resolution image with the same resolution as the sample images from the training phase into the trained image processing model. By adjusting the value of parameter N in the image model, the low-resolution image can be reconstructed into various high-resolution images of different resolutions.

[0084] For example, refer to Figure 5 The electronic device 100 is equipped with an image processing model trained using sample images with a resolution of 512×512 and various high-resolution images of different resolutions. Users can adjust the value of parameter N in the image processing model via the electronic device 100 to reconstruct the 512×512 image M1 input into the model into high-resolution images of different resolutions. For example, when the user adjusts the value of parameter N in the image processing model to n1 (e.g., n1 = 8), the electronic device 100 can reconstruct image M1 into an image M2 with a resolution of 1024×1024. When the user adjusts the value of parameter N in the image processing model to n2 (e.g., n2 = 4), the electronic device 100 can reconstruct image M1 into an image M3 with a resolution of 720×720. When the user adjusts the value of parameter N of the image processing model to n3 (for example, n3 is 2) through the electronic device 100, the electronic device 100 can reconstruct the image M1 into an image M3 with a resolution of 650×650 through the image processing model.

[0085] In some embodiments, the value of parameter N can be determined by the following formula (1) or formula (2).

[0086]

[0087] Among them, target w The width of the target image, target h The height of the target image, source w The width of the input image, source h The height of the input image is given by `height`, and `roundup()` is the function for rounding up.

[0088] It can be understood that the target image is the high-resolution image output by the image processing model. The input image is the low-resolution image input to the image processing model.

[0089] The internal structure of the image processing model will be described in detail below.

[0090] Figure 6A A schematic diagram of the structure of image processing model A is shown.

[0091] refer to Figure 6A The encoding layer of the image processing model includes a first encoder, which encodes the input low-resolution image to obtain a first latent variable.

[0092] The diffusion network layer of the image processing model includes a diffusion model without an attention mechanism, which is used to perform back-diffusion processing on the first latent variable output by the first encoder, that is, denoising processing, and output the second latent variable.

[0093] The decoding layer of the image processing model includes a first decoder, which decodes the fourth latent variable output by the interleaved network layer to output a high-resolution image.

[0094] It is understandable that the first encoder can be a variational autoencoder (VAE), a convolutional neural network encoder, a residual network encoder, a generative adversarial network encoder, or an encoder of U-Net (an image segmentation network), etc., used to encode images into latent variables.

[0095] It is understandable that the first decoder can be a convolutional neural network decoder, a residual network decoder, a generative adversarial network decoder, a U-Net encoder, etc., used to decode latent variables into an image decoder.

[0096] It should be noted that when the first encoder is a U-net encoder and the first decoder is a U-net decoder, the training of the first encoder and the second encoder can be carried out in U-net or in the image processing model A.

[0097] It should be noted that the first encoder has an upsampling function, and the first decoder has a downsampling function. This application does not limit the sampling rate of the first encoder and the first decoder. For ease of understanding, when describing the technical solution of this application below, it is assumed that the sampling rate of the first decoder and the first decoder is 1x, which will not be elaborated further below.

[0098] The interleaved network layer consists of M 1×1 2D convolutional kernels, such as conv_0, conv_1, conv_2, ..., conv_M. It can be understood that the interleaved network layer can use these M kernels to convolve the second hidden variable to obtain M third hidden variables, such as x_bar_0, x_bar_1, x_bar_2, ..., x_bar_M, and then concatenate these M third hidden variables to form a fourth hidden variable.

[0099] like Figure 6A As shown, in the image processing model's processing of the input low-resolution image, the first encoder of the encoding layer encodes the input low-resolution image and outputs a first latent variable. Then, the diffusion model of the diffusion network layer performs back-diffusion processing on the first latent variable to output a second latent variable. Next, the interleaving network layer convolves the second latent variable with multiple convolutional kernels to obtain multiple third latent variables, and then concatenates these multiple third latent variables into a fourth latent variable. Finally, the first decoder of the decoding layer decodes the fourth latent variable and outputs the corresponding high-resolution image.

[0100] It is understandable that the interleaved network layer processes the second hidden variable of the input through multiple convolution kernels to obtain multiple third hidden variables, and concatenates these multiple third hidden variables into a fourth hidden variable. This can avoid expanding the canvas (i.e., increasing the image size) when generating inaccurate neighborhoods, and preserve the generation effect of the model (such as detail richness, color consistency, texture naturalness, etc., ensuring that the expanded canvas is visually consistent with the original generated image) without introducing additional mesh artifacts and sampling degradation.

[0101] Mesh artifacts refer to unnatural grid-like patterns that occur in image processing due to pixelation or undersampling. For example, when you zoom in on an image, if nearest neighbor interpolation is used, you might see pixel blocks become apparent, forming a grid. This is because each pixel is copied to fill a larger area without considering the influence of surrounding pixels, causing the image to appear as if it is composed of squares.

[0102] Sampling degradation is a common problem during image enlargement or reduction, leading to a decrease in image quality. Interleaving stitching technology, by optimizing stitching strategies and algorithms, reduces image blurring, jagged edges, and other degradation phenomena caused by improper sampling, thereby maintaining high image clarity.

[0103] Figure 6B A schematic diagram of the structure of image processing model B is shown.

[0104] refer to Figure 6B The difference between image processing model B and image processing model A is that the encoding layer of image processing model B also includes a second encoder (also known as a "text encoder" or "language encoder"), and the diffusion model in the diffusion network layer of image processing model B is a diffusion model that incorporates a multi-head attention mechanism (hereinafter referred to as "cross-attention diffusion model").

[0105] The second encoder encodes the input image and outputs a corresponding query vector. The query vector is a feature vector representing the semantic content of the image (such as the type of person, animal, or scene in the image), used for tasks such as image retrieval, classification, and recognition. The query vector can be viewed as a query or indication of the image content, guiding the model to focus on which parts or features of the image.

[0106] In some embodiments, in order to obtain more detailed local detail information, a second encoder can be used to encode the low-resolution image and feed it into a cross-fusion multi-head attention diffusion model for detail querying, thereby improving the image super-resolution effect.

[0107] Specifically, when the second encoder encodes the input low-resolution image, it can first divide the low-resolution image into multiple pixel patches according to a preset number, that is, divide the entire image region of the input image into multiple local regions. Then, through a linear mapping layer (used to map the input image from image space to vector space), the pixels of each pixel patch are linearly mapped to obtain the corresponding feature vector. Each feature vector is then encoded into a uniform specific length. Finally, the feature vectors corresponding to the multiple local regions are concatenated into a global linear vector, which is the query vector.

[0108] It is understandable that the second encoder can be a text encoder in a contrastive language-image pre-training (CLIP) model, or it can be any other encoder used to encode an image into a query vector.

[0109] It can be understood that CLIP's text encoder breaks down the input sentence (a sentence used to describe the content of an image, such as "there is a puppy in the middle of the image") into multiple tokens, then maps each token to a corresponding feature vector through a linear mapping layer, and finally concatenates the feature vectors of multiple tokens into a linear vector, which serves as the query vector for the sentence.

[0110] It's understandable that image processing model B processes images, not sentences. Since the sentence input to the text encoder describes the image content, there's a mapping relationship between the sentence and the image. When CLIP's text encoder acts as the second encoder to encode the input image, it can divide the input image into multiple pixel blocks based on the number of tokens that the corresponding sentence would be split into. Then, these pixel blocks are mapped as pseudo-terms through a linear mapping layer to obtain corresponding feature vectors. These feature vectors are then concatenated to obtain the query vector corresponding to the input image.

[0111] like Figure 6BAs shown, during the processing of the input low-resolution image by image processing model B, the first encoder and the second encoder of the encoding layer of image processing model B encode the low-resolution image and output a first latent variable and a query vector, respectively. The first latent variable is input to the input layer of the cross-fusion multi-head attention diffusion model, and the query vector is input to the attention network layer of the cross-fusion multi-head attention diffusion model. Then, the cross-fusion multi-head attention diffusion model performs back-diffusion processing on the first latent variable based on the query vector to output a second latent variable. Next, the interleaved network layer convolves the second latent variable with multiple convolutional kernels to obtain multiple third latent variables, and then concatenates these multiple third latent variables into a fourth latent variable. Finally, the first decoder of the decoding layer decodes the fourth latent variable and outputs the corresponding high-resolution image.

[0112] It is understandable that the query vector output by the second encoder contains the content information of the low-resolution image. Inputting the query vector into the attention network layer of the cross-fusion multi-head attention diffusion model can help the cross-fusion multi-head attention diffusion model understand the content in the low-resolution image. This allows the cross-fusion multi-head attention diffusion model to more accurately denoise the first latent variable during the back-diffusion process of the first latent variable, which means it can more accurately reconstruct the content of the low-resolution image, thereby improving the super-resolution effect of the low-resolution image.

[0113] For example, refer to Figure 6C The electronic device 100 can input an image M1 with a resolution of 512×512 into the image processing model B, and perform super-resolution processing on the image M1 through the image processing model B to obtain an image M2 with a resolution of 1024×1024.

[0114] Specifically, during the super-resolution processing of image M1 by image processing model B, the first encoder encodes image M1 to obtain a first latent variable with a size of 512×512. The second encoder encodes image M1 to obtain a query vector. Then, the cross-fusion multi-head attention diffusion model performs back-diffusion processing on the first latent variable based on the query vector to obtain a second latent variable with a size of 512×512. Next, the interleaved network layer convolves the second latent variable with two convolutional kernels to obtain two third latent variables, and concatenates these two third latent variables to form a fourth latent variable with a size of 1024×1024. Finally, the first decoder decodes the fourth latent variable to obtain an image M2 with a resolution of 1024×1024.

[0115] Figure 7 This diagram illustrates how the attention network layer of the cross-fusion multi-head attention diffusion model processes query vectors.

[0116] refer to Figure 7 After the second encoder encodes the low-resolution image to obtain the query vector, this query vector can be input into multiple head modules of the attention network layer in the cross-fusion multi-head attention diffusion model. The query vector is then weighted to obtain an attention score, which is input into the next layer of the cross-fusion multi-head attention diffusion model (e.g., a backdiffusion network) for further processing. Then, in the backdiffusion network of the cross-fusion multi-head attention diffusion model, the first latent variable can be backdiffused based on this attention score to obtain the second latent variable.

[0117] In some embodiments, the attention network layer of the cross-fusion multi-head attention diffusion model can calculate the attention score of the query vector using the following formula (3).

[0118]

[0119] Where Attention(Q,K,V) is the attention score of the query vector, softmax() is the activation function, Q is the query matrix, K is the key matrix, V is the value matrix, and d is the dimension of the model, i.e. the dimension of Q, K, and V.

[0120] The following formulas (4), (5), and (6) show how Q, K, and V are calculated:

[0121] Q = W q ·I(4)

[0122] K = W k ·I(5)

[0123] V = W V ·I(6)

[0124] Where I is the query vector, W q W k and W V All of these are learnable matrices, and their values ​​are determined during the model training phase.

[0125] In some embodiments, reference Figure 8 When performing super-resolution processing on low-resolution images using image processing model B, noise can be added to the first latent variable to simulate the noise added during the forward diffusion process of image processing model B in the training phase, thereby improving the super-resolution effect of image processing model B and obtaining a high-resolution image with higher image quality.

[0126] Figure 9 A schematic flowchart of an image processing method provided in an embodiment of this application is shown.

[0127] It should be noted that the image processing method implemented in this application embodiment can be performed on electronic devices such as laptops, tablets, mobile phones, desktop computers, personal digital assistants (PDAs), and servers. The executing entities for each of the following processes are all the aforementioned electronic devices; therefore, the executing entities for each process will not be described again when introducing each process.

[0128] refer to Figure 9 The method includes the following steps:

[0129] S101: Obtain the first image at the first resolution.

[0130] In some embodiments, the electronic device may acquire a first image at a first resolution, such as those described above. Figure 1 The image M1 with a resolution of 512×512 shown is the image to be super-resolution processed.

[0131] S102: Encode the first image to obtain the first latent variable.

[0132] In some embodiments, the electronic device can encode the first image to obtain the first latent variable through the coding layer of the aforementioned image processing model.

[0133] Specifically, electronic devices can be accessed through... Figure 6A Image processing model A shown Figure 6B In the image processing model B shown, the first encoder of the coding layer encodes the first image to obtain the first latent variable. For example, the first encoder encodes the image M1 with a resolution of 512×512 to obtain the first latent variable with a size of 512×512.

[0134] S103: Perform backdiffusion on the first latent variable to obtain the second latent variable.

[0135] In some embodiments, the electronic device can perform backdiffusion processing on the first hidden variable of the diffusion network layer of the aforementioned image processing model to obtain the second hidden variable.

[0136] In some embodiments, the electronic device can... Figure 6A The diffusion model of the diffusion network layer in the image processing model A shown performs reverse diffusion processing on the first latent variable to obtain the second latent variable.

[0137] In other embodiments, the electronic device can... Figure 6BIn the image processing model B shown, the second encoder of the encoding layer encodes the first image to obtain a query vector. Then, the cross-fusion multi-head attention diffusion model of the image diffusion network layer performs back diffusion processing on the first latent variable based on the query vector to obtain the second latent variable.

[0138] In some embodiments, the second latent variable has the same size as the first latent variable.

[0139] S104: The second latent variable is scaled to obtain multiple third latent variables, and the multiple third latent variables are concatenated to obtain the fourth latent variable.

[0140] In some embodiments, the electronic device can perform convolution processing (scaling processing) on ​​the second latent variable using N 1×1 2D convolution kernels in the interleaved network layer of the aforementioned image processing model to obtain N third latent variables of the same size, and then interleave and splice these N third latent variables of the same size into a fourth latent variable of a larger size.

[0141] For example, an electronic device can perform convolution processing on a second latent variable of size 512×512 using two 1×1 2D convolution kernels in an interleaved network layer to obtain two third latent variables of size 512×512. Then, the two third latent variables of size 512×512 are concatenated into a fourth latent variable of size 1024×1024.

[0142] In some embodiments, the row data of the N third latent variables can be alternated in the column direction, and the column data of the N third latent variables can be alternated in the row direction to obtain the fourth latent variable. The specific concatenation method can be referred to the above. Figure 3 The splicing method shown will not be described in detail here.

[0143] S105: Decode the fourth latent variable to obtain a second image with a second resolution, wherein the second resolution is greater than the first resolution.

[0144] In some embodiments, the electronic device can decode the fourth latent variable through the first decoder of the decoding layer in the aforementioned image processing model to obtain a second image with a second resolution.

[0145] For example, the electronic device can decode the fourth latent variable with a size of 1024×1024 using the first encoder to obtain the above... Figure 1 The image M2 shown has a resolution of 1024×1024 (as an example of the first resolution) and is shown as an example of the second image.

[0146] In this embodiment, if it is desired to reconstruct a low-resolution image (such as the first image) into multiple images of different resolutions, the second latent variable of the low-resolution image can be convolved by adjusting the convolution kernels of different data to obtain different numbers of third latent variables, thereby obtaining fourth latent variables of different sizes, and finally obtaining high-resolution images of different resolution sizes. This satisfies the user's need to reconstruct a low-resolution image into multiple high-resolution images of different sizes.

[0147] Figure 10 A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0148] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0149] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). In some embodiments, processor 110 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, and subscriber identification module (SIM) interfaces.

[0150] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0151] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0152] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0153] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 100. The wireless communication module 160 can provide wireless communication solutions, including wireless local area networks (WLANs) (such as Wi-Fi), Bluetooth, Global Navigation Satellite System (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies, for use on the electronic device 100. In some embodiments, antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, enabling the electronic device 100 to communicate with networks and other devices via wireless communication technologies.

[0154] Electronic device 100 implements display functions through a GPU, screen 194, and application processor. The GPU is a microprocessor for image processing, connecting the screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0155] Screen 194 is used to display images, videos, etc. Screen 194 includes a display panel. In some embodiments, electronic device 100 may include one or N screens 194, where N is a positive integer greater than 1.

[0156] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, screen 194 and application processor.

[0157] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0158] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. Electronic device 100 can implement audio functions, such as music playback and recording, through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0159] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0160] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0161] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0162] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, read-only memory, magneto-optical disks, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0163] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0164] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0165] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image processing method for electronic devices, characterized in that, The method includes: Acquire a first image at a first resolution; The first image is subjected to a first encoding process to obtain a first latent variable, and the first latent variable represents the image features of the first image; The first latent variable is subjected to diffusion processing to obtain the second latent variable, and the diffusion processing is used to remove noise from the first image; The values ​​in the second latent variable are scaled to obtain multiple third latent variables, and the multiple third latent variables are concatenated to form a fourth latent variable, the size of the fourth latent variable being larger than the size of the first latent variable; Decoding the fourth latent variable yields a second image with a second resolution, wherein the second resolution is greater than the first resolution; The scaling process of the values ​​in the second latent variable yields multiple third latent variables, including: The second hidden variable is convolved with multiple convolution kernels to obtain the multiple third hidden variables; The step of concatenating the plurality of third latent variables into a fourth latent variable includes: The fourth hidden variable is obtained by alternating the row data of the plurality of third hidden variables in the column direction and alternating the column data of the plurality of third hidden variables in the row direction.

2. The method according to claim 1, characterized in that, The multiple convolution kernels include multiple 1D convolution kernels with different parameters.

3. The method according to claim 1, characterized in that, The method further includes: The first image is subjected to a second encoding process to obtain a query vector, wherein the query vector is used to characterize the semantic content of the first image; The second latent variable is obtained by performing a diffusion process on the first latent variable, including: The first latent variable is obtained by performing backdiffusion processing on the first latent variable based on the query vector.

4. The method according to claim 3, characterized in that, The step of performing a second encoding process on the first image to obtain a query vector includes: The first image is divided into multiple regions, and the pixels of the multiple regions are linearly mapped to obtain multiple feature vectors. The multiple feature vectors are then concatenated to form the query vector.

5. The method according to claim 3, characterized in that, The step of performing backdiffusion processing on the first latent variable based on the query vector to obtain the first latent variable includes: The query vector is weighted to obtain an attention score; The first latent variable is obtained by performing the backdiffusion process on the first latent variable based on the attention score.

6. The method according to any one of claims 1 to 5, characterized in that, The electronic device is equipped with an image processing model, which includes an encoding layer, a diffusion network layer, an interleaving network layer, and a decoding layer. The step of performing a first encoding process on the first image to obtain the first latent variable includes: The first latent variable is obtained by performing the first encoding process on the first image through the encoding layer; The process of diffusing the first latent variable to obtain the second latent variable includes: The second hidden variable is obtained by performing diffusion processing on the first hidden variable through the diffusion network layer; The scaling process of the values ​​in the second latent variable to obtain multiple third latent variables, and the concatenation of the multiple third latent variables into a fourth latent variable, includes: The values ​​in the second hidden variable are scaled through the diffusion network layer to obtain multiple third hidden variables, and the multiple third hidden variables are concatenated to form the fourth hidden variable. Decoding the fourth latent variable to obtain a second image at a second resolution includes: The second image is obtained by decoding the fourth hidden variable through the decoding layer.

7. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device; A processor, when executing the instructions in the memory, causes the electronic device to perform the image processing method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the image processing method according to any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, causes the computer to implement the image processing method according to any one of claims 1 to 6.

Images