Image processing method, electronic device, and computer-readable storage medium

Abstract

An image processing method, an electronic device and a computer readable storage medium are provided. In the method, the electronic device can perform super-resolution processing on low-resolution latent space features to obtain high-resolution latent space features. Furthermore, the electronic device can generate a high-resolution image based on the high-resolution latent space features. Since the data volume of the latent space features is often small when the electronic device performs super-resolution on the latent space features in the latent space domain, the electronic device can speed up the generation of high-resolution images in the case of hardware constraints, and provide high-resolution images to users more quickly.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal and communication, and particularly relates to an image processing method, an electronic device and a computer readable storage medium. BACKGROUND

[0002] Electronic devices often need to display images for users to watch. The images displayed by the electronic devices can be generated by the electronic devices or obtained by the electronic devices from a network. Sometimes, due to the limitation of hardware conditions or network conditions, the resolution of the images displayed by the electronic devices is low, so that it is difficult for users to clearly see the details of the images, and the experience of users when watching the images is affected. SUMMARY

[0003] The present application provides an image processing method, an electronic device and a computer readable storage medium. In the method, the electronic device can perform super-resolution on the latent space feature after obtaining the low-resolution latent space feature to obtain a high-resolution latent space feature. Then, the electronic device decodes based on the high-resolution latent space feature to obtain a high-resolution image. In this way, the electronic device can generate a high-resolution image more quickly.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an electronic device. The method comprises: the electronic device extracts latent space features of multiple scales from a first latent space feature, and splices the latent space features of the multiple scales to obtain a second latent space feature, the resolution of the second latent space feature being higher than the resolution of the first latent space feature; the electronic device splits the second latent space feature into multiple third latent space features, the resolution of the third latent space feature being lower than the resolution of the second latent space feature; the electronic device decodes the multiple third latent space features and the first latent space feature in parallel, and splices the decoded multiple third latent space features to obtain first image data; the electronic device obtains second image data by using the first image data and a fourth latent space feature, the fourth latent space feature being obtained by linearly stretching the decoded first latent space feature, and the resolution of the fourth latent space feature being the same as that of the first image data.

[0005] The electronic device extracts multiple scale latent space features from the first latent space feature, which can ensure the depth and width of feature extraction, thereby extracting more rich features from the first latent space feature, and ensuring that the details of the image will not be lost during subsequent decoding. Moreover, the electronic device obtains the second latent space feature through the multiple scale latent space features, which has smaller computational complexity than schemes that extract features using an attention mechanism, so that the electronic device can generate a high-resolution image more quickly. The first latent space feature can be a lower-resolution latent space feature, and the electronic device can increase the resolution of the latent space feature to obtain a second latent space resolution, and then the electronic device can decode the high-resolution second latent space feature to obtain a high-resolution image. Before decoding the high-resolution second latent space feature, the electronic device can split the second latent space feature into multiple third latent space features with smaller resolutions. Then the electronic device can decode the third latent space features in parallel, which can speed up the calculation. Finally, the electronic device adjusts the color of the first image data obtained by splicing and decoding the third latent space features using the initial first latent space feature to obtain second image data, thereby balancing the color difference of different regions in the first image data.

[0006] In combination with the first aspect, in some embodiments, one scale latent space feature of the multiple scale latent space features is obtained after the first latent space feature is fused once or multiple times with depth separable convolution operations, and the sizes of the convolution kernels in the multiple depth separable convolution operations are different from each other.

[0007] The multiple scale latent space features are obtained by performing depth separable convolution operations on the first latent space feature using multiple convolution kernels with different sizes, so that the electronic device can cover each spatial dimension when extracting image features in the first latent space feature, so that the image features in each spatial dimension interact with each other, thereby better extracting the image features in the first latent space feature and ensuring the quality of the generated image.

[0008] In combination with the first aspect, in some embodiments, after the electronic device obtains the second latent space feature, the method further includes: the electronic device performing multiple convolution operations on the second latent space feature respectively and fusing the results of the multiple convolution operations.

[0009] That is, the electronic device can also perform multiple convolution operations with different sizes on the second latent space feature, thereby ensuring the depth of image feature extraction.

[0010] In some embodiments, the electronic device obtains the second image data from the first image data and the fourth latent space feature, including: the electronic device linearly stretches the first latent space feature to obtain the fourth latent space feature; the electronic device determines the first color compensation value based on the fourth latent space feature; and the electronic device compensates the color of the first image data based on the first color compensation value to obtain the second image data.

[0011] In some embodiments, the electronic device can stretch the first latent space feature to the same resolution as the spliced latent space feature of the plurality of third latent space features (the resolution is equal to the resolution of the first image data). Then the electronic device can determine the first color compensation value based on the fourth latent space feature, and the first color compensation value can contain the pixel value that each pixel point in the first image data needs to compensate. The first image data can contain three color channels (such as R / G / B channels), and the first color compensation value can also correspond to contain three color channels. The electronic device can add the compensation value at the corresponding position in the first color compensation value to the pixel value of each pixel point in the first image data to finally obtain the second image data. Since the first image data is obtained by splicing and decoding a plurality of smaller third latent space features, the colors of the regions in the first image data can be uneven, and the boundaries between the regions can be obvious. The electronic device can eliminate the color difference between the regions in the first image data, so that the boundaries between the regions are more smoothly transitioned.

[0012] In some embodiments, the first latent space feature is a latent space feature used to describe an image generated based on a user input.

[0013] In some embodiments, the image generated based on the user input is determined based on first text input by the user, and the latent space feature of the image generated based on the user input is generated based on random noise and the first text.

[0014] In a scenario of generating an image based on a text, the electronic device can obtain a first text input by a user, the first text containing content of an image that the user intends to generate, for example, the first text is: "draw a flying puppy". The electronic device can generate random noise in a latent space, and then denoise the random noise multiple times based on the first text to obtain a first latent space feature, wherein the first latent space feature can contain feature values of the image that the user intends to generate. Further, the electronic device can further extract image features from the first latent space feature, and perform super-resolution and decoding on the first latent space feature, and finally obtain a high-resolution image. Compared with a method of directly obtaining a high-resolution random noise, and then denoising the high-resolution random noise based on the first text, and further decoding the denoised latent space feature to generate an image, the method has smaller computational complexity, and the electronic device can generate a high-resolution image based on the text input by the user more quickly, thereby improving the user experience.

[0015] In combination with the first aspect, in some embodiments, the image that the user intends to generate is determined based on the first text input by the user and a first image, and the latent space feature of the image that the user intends to generate is generated based on the first image to which random noise is added and the first text.

[0016] In a scenario of generating an image based on an image, the electronic device can obtain a first image and a first text input by a user, the first text can indicate content of editing the first image that the user intends to perform, for example, the first text can indicate eliminating a certain object in the first image, changing a certain object in the first image, adding a certain object in the first image, and the like. The electronic device can add random noise to the first image, and then encode the first image to which the random noise is added to obtain a corresponding latent space feature. Then the electronic device can gradually denoise the above latent space feature based on the first text. In the process of denoising the latent space feature, the electronic device can gradually change the image content indicated by the latent space feature based on the first text, and finally the electronic device can obtain a first latent space feature with a lower resolution. The electronic device can extract image features from the first latent space feature and perform decoding to finally obtain first image data. Compared with a scheme of directly generating a high-resolution latent space feature and then extracting image features and decoding, the method has smaller computational complexity, so that the electronic device can change the content of the first image based on the first text input by the user more quickly, thereby improving the user experience.

[0017] In combination with the first aspect, in some embodiments, the first latent space feature is any one of the following: a latent space feature of an image input by the user, a latent space feature of an image stored by the electronic device, and a latent space feature of an image obtained by the electronic device from a network.

[0018] That is, the electronic device can provide an entry for a user to improve the resolution of an image. The electronic device can obtain a low-resolution image and then perform super-resolution on the image to obtain a high-resolution image. Alternatively, the electronic device can also generate a higher-resolution image based on a lower-resolution image obtained from the network when the network condition is poor, improving the user's experience of viewing images on the network.

[0019] In combination with the first aspect, in some embodiments, after the electronic device obtains the second image data by using the first image data and the fourth latent space feature, the method further includes: displaying, by the electronic device, the first image based on the second image data, wherein the resolution of the first image is greater than the resolution of the image input by the user, the resolution of the first image is greater than the resolution of the image stored by the electronic device, and the resolution of the first image is greater than the resolution of the image obtained by the electronic device from the network.

[0020] That is, the electronic device can display a high-resolution image after super-resolution processing of a low-resolution image, thereby improving the user's viewing experience.

[0021] In combination with the first aspect, in some embodiments, the electronic device extracts multiple scales of latent space features from the first latent space feature using the multi-scale feature extraction model and concatenates the multiple scales of latent space features, the electronic device decodes the multiple third latent space features and the first latent space feature in parallel using the decoding model, and before the electronic device extracts the multiple scales of latent space features, the method further includes: training, by the electronic device, the parameters of the multi-scale feature extraction model through a latent space domain loss function; and training, by the electronic device, the parameters of the multi-scale feature extraction model and the decoding model through an image domain loss function, the latent space domain loss function is used to describe the degree of deviation between the latent space feature of the first sample image and the latent space feature of the second sample image, and the latent space feature of the second sample image is a high-resolution latent space feature obtained by inputting the down-sampled latent space feature of the first sample image to the multi-scale feature extraction model. The image domain loss function is used to describe the degree of deviation between the predicted image and the first sample image, and the predicted image is an image obtained by inputting the latent space feature of the second sample image to the decoding model.

[0022] That is to say, the electronic device can be trained from different angles, thereby improving the ability of the multi-scale feature extraction model and the decoding model to process latent space features. First, the parameters in the multi-scale feature extraction model are trained based on the latent space domain loss function, thereby improving the ability of the multi-scale feature extraction model to extract image features in the latent space. Then, the electronic device can train the parameters of the decoding model and the multi-scale extraction model based on the image domain loss function, so that the electronic device can further adjust the parameters of the multi-scale extraction model and the parameters of the decoding model from the perspective of image generation effect. So that the electronic device can generate clearer high-resolution images through the multi-scale extraction model and the decoding model.

[0023] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor coupled to the memory; the memory stores a computer program, and the processor executes the computer program to enable the electronic device to implement the method of any one of the first aspect.

[0024] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the method of any one of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer program product, and when the computer program product is executed by a processor, the method of any one of the first aspect will be implemented.

[0026] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a memory, wherein the memory is used to store a computer program or computer instructions, and the processor is used to execute the computer program or computer instructions stored in the memory, so that the chip executes the method of any one of the first aspect.

[0027] The solutions provided in the second aspect to the fifth aspect are used to implement or cooperate to implement the method provided in the first aspect, so the same or corresponding beneficial effects can be achieved as the corresponding method in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figures 1A-1B is a scene schematic diagram of some electronic devices provided by embodiments of the present application generating low-resolution images in a text-to-image scene;

[0029] Figure 2 is a structural schematic diagram of an electronic device 100 provided by an embodiment of the present application;

[0030] Figure 3is an architecture schematic diagram of an electronic device 100 provided by an embodiment of the present application to generate a high-resolution image;

[0031] Figure 4 is an architecture schematic diagram of a feature super-resolution module 302 provided by an embodiment of the present application;

[0032] Figure 5 is an architecture schematic diagram of a multi-branch feature extraction module 402 provided by an embodiment of the present application;

[0033] Figure 6 is an architecture schematic diagram of a multi-branch depth convolution layer provided by an embodiment of the present application;

[0034] Figure 7 is an architecture schematic diagram of a multi-branch convolution layer provided by an embodiment of the present application;

[0035] Figure 8 is an example diagram of re-parameterization of a plurality of depth separable convolution layers provided by an embodiment of the present application;

[0036] Figure 9 is a flow schematic diagram of an electronic device 100 decoding a high-resolution feature map to generate a high-resolution image provided by an embodiment of the present application;

[0037] Figure 10 is a flow schematic diagram of training a feature super-resolution module based on a latent space domain loss provided by an embodiment of the present application;

[0038] Figure 11 is a flow schematic diagram of training a feature super-resolution module and a decoder based on an image domain loss provided by an embodiment of the present application;

[0039] Figure 12 is a flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0040] Figure 13 is a structure schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0041] Figure 14 is a structure schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the application and the appended claims, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in the description of the embodiments of the application, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0043] Hereinafter, the terms "first", "second" are used only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0044] Since the present application relates to the application of neural networks, the following will introduce the related terms of neural networks that may be involved in the embodiments of the present application.

[0045] 1. Neural network

[0046] The neural network can be composed of neural units, and the neural unit can refer to an operation unit taking x s and intercept 1 as input. The output of the operation unit can refer to the formula: Wherein, s = 1, 2, …, n, n is a natural number greater than 1, W s is the weight of x s , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units described above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0047] 2. Convolutional neural network

[0048] A convolutional neural network (CNN) is a neural network with a convolutional structure. A CNN contains a feature extractor composed of convolutional layers and sub-sampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to a layer of neurons in a CNN that performs convolutional processing on an input signal. In a convolutional layer of a CNN, a neuron can be connected to only a portion of the neurons in the adjacent layer. A convolutional layer typically contains several feature maps, each of which can be composed of a number of rectangularly arranged neurons. The neurons of the same feature map share weights, where the shared weights are convolutional kernels. The shared weights can be understood as being independent of the way and position of extracting image information. The implicit principle is that the statistical information of a certain part of an image is the same as that of other parts. That is, the image information learned in a certain part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more the number of convolutional kernels, the more image information reflected by the convolution operation.

[0049] The convolutional kernels can be initialized in the form of a matrix of random size, and the convolutional kernels can obtain reasonable weights through learning in the training process of the CNN. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the CNN, while also reducing the risk of overfitting.

[0050] 3. Depthwise separable convolution

[0051] Depthwise separable convolution (DW Conv) is an operation in convolutional neural networks used to reduce the number of model parameters and computations, to improve the efficiency and performance of the model. Depthwise separable convolution can include depthwise convolution and pointwise convolution. Depthwise convolution refers to performing convolution operations on each input channel separately. For an input feature map, if there are k input channels, depthwise convolution will apply a convolution kernel to each channel separately, generating k output feature maps. In this way, depthwise convolution operation only uses k parameters (the size of the convolution kernel), which greatly reduces the number of parameters compared to traditional convolution. Pointwise convolution refers to applying a 1*1 convolution kernel to the entire feature map. Pointwise convolution is actually a fully connected operation, corresponding to performing a dot product operation between each position of the input feature map and the convolution kernel, thereby generating an output feature map. Pointwise convolution does not change the spatial dimensions of the feature map, but can change the number of channels of the feature map. Through pointwise convolution, different channels can be linearly combined, further extracting features and expanding the feature representation capability.

[0052] 4、Loss function

[0053] In the process of training a neural network, because the output of the neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network with the truly intended target value, and according to the difference between the two. (Of course, before the first update, there is usually an initialization process, that is, the parameters of each layer in the neural network are pre-configured), for example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value" in advance, which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the neural network becomes a process of trying to minimize this loss.

[0054] 5、Backpropagation algorithm

[0055] The convolutional neural network can adopt a back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output generates an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal neural network model parameters, such as weight matrix.

[0056] 6、Channel feature

[0057] The feature map can include multiple channels, and each channel can represent a feature. A feature can include global information of a certain pattern or attribute in an image. For example, some channels can include edge information in an image, and some channels can include texture, color, shape, and other information in an image. In an RGB image, three channels can include red, green, and blue information. Among them, the height and width of the feature of each channel in the feature map are consistent. For example, in a 64x64x4 feature map, 4 is the number of channels of the feature map, and the size of the feature of each channel in the feature map is 64x64.

[0058] The number of channels in the feature map can be determined by the number of convolution kernels. Each convolution kernel in the convolutional neural network will generate a new channel, and the goal of each convolution kernel is to extract a certain specific image feature, such as image edge, texture, color, etc.

[0059] 7、Spatial feature

[0060] The spatial feature refers to the information of each position in the feature map in space, which is related to the spatial dimension (i.e. height and width) of the feature map. The spatial feature can be used to represent the information of a local region in the image. For example, in a 64x64x4 feature map, 64x64 is the size of the feature map in the spatial dimension (i.e. the size of the spatial feature), and each element in the spatial dimension can include a corresponding feature value. For example, in a 64x64x4 feature map, there are 4 channels, and each channel includes 64x64 (a total of 4096) feature values. The size of the spatial feature is determined by the size of the convolution kernel and the step.

[0061] Electronic devices often need to display images for users to view. Some of the images can be generated by the electronic devices, such as photos taken by the electronic devices, images generated by the electronic devices based on voice or text instructions of the users (text-to-image), images edited by the electronic devices based on voice or text instructions of the users (image-to-image), and the like. Some of the images can also be obtained by the electronic devices from remote servers or other electronic devices through wired or wireless networks. Sometimes, the resolution of the images generated by the electronic devices is low due to the limitation of the hardware computing power and inference time delay, or the network quality between the electronic devices and the remote servers or other electronic devices is poor, and the remote servers or other electronic devices compress the images when transmitting the images, thereby reducing the resolution of the images, resulting in low resolution of the images received by the electronic devices. The resolution of the images can be represented by the number of pixel points in each row (horizontal direction) multiplied by the number of pixel points in each column (vertical direction), for example, the resolution of an image is 400 x 800, which means that the image has 400 pixel points in each row and 800 pixel points in each column.

[0062] The following exemplary introduces a scenario in which an electronic device generates a low-resolution image. Figures 1A-1B is a scenario diagram in which some electronic devices provided by embodiments of the present application generate low-resolution images in a text-to-image scenario.

[0063] As shown in Figure 1A , the electronic device 100 can display a smart assistant interface 101, which can be used for conversation interaction between the user and the electronic device 100. The electronic device 100 can provide one or more functions for the user based on the text or voice input by the user, and the one or more functions may, for example, include a function of obtaining information consultation, a chat function, a function of generating text, pictures, and videos based on conditions set by the user, and the like. The smart assistant interface 101 can include a conversation window 102, an editing box 103, and a send button 104.

[0064] The conversation window 102 is used to display the content of the conversation interaction between the user and the electronic device 100.

[0065] The editing box 103 is used to provide an editing entry for the conversation content for the user.

[0066] The send button 104 is used to submit the conversation content in the editing box 103, so that the electronic device 100 can respond through text, voice, images, and the like based on the conversation content input by the user.

[0067] As shown in Figure 1AAs shown, the electronic device 100 can receive the user's operation of inputting "draw a puppy sticking out its tongue" in the edit box 103, and then the electronic device 100 receives the user's operation of clicking the send button 104, and in response to the operation, the electronic device 100 can perform syntax analysis, syntax analysis, user intention analysis, etc. on "draw a puppy sticking out its tongue", and determine that the user wants an image of a puppy. The electronic device 100 can generate an image of a puppy sticking out its tongue through an image generation model. The above image generation model may, for example, be a Flux model, a stable diffusion model, an Omost model, etc., and the embodiments of the present application do not limit this.

[0068] As shown in Figure 1B , the electronic device 100 can display the dialog content 105 submitted by the user in the dialog window 102, and the dialog content 106 replied by the electronic device 100 to the dialog content 105. Among them, the dialog content 106 can contain the text reply "OK, created for you" of the electronic device 100, and the image 107 of the puppy sticking out its tongue generated by the electronic device 100. The image 107 can be saved to the gallery, so that the user can view it at any time.

[0069] Among them, the electronic device 100 can be to generate the above-mentioned puppy image using its own processor (such as the neural network processing unit NPU inside the electronic device 100), when the electronic device 100 itself processor computing power is small, or the electronic device 100 has a time delay limit for generating images, the electronic device 100 can only generate images with low resolution. As shown in Figure 1B , the resolution of the image 107 is low, and the image 107 may have problems such as unclear boundaries of image content, containing more jaggies, missing image details, etc., which may not meet the user's demand for the electronic device to generate high-quality images, affecting the user's experience.

[0070] To solve the above problems, the embodiments of the present application provide an image processing method, an electronic device and a computer readable storage medium. In the method, the electronic device performs super-resolution on the low-resolution image features from the feature level, reducing the amount of calculation during super-resolution. Then the electronic device divides the image features after super-resolution into multiple regions, and generates sub-images by decoding the image features of the multiple regions in parallel. Finally, the electronic device splices the sub-images into a complete high-resolution image. In this way, the electronic device can generate high-resolution images faster under the condition of limited hardware, generation time delay or network time delay, thereby improving the user's experience.

[0071] For ease of description and better understanding, the embodiments of the present application take Figure 1A , Figure 1BThe image processing method is introduced by taking the scenario of generating an image from text (also referred to as text-to-image) as an example. The image processing method can be applied to other scenarios. For example, the image processing method can be applied to improving the resolution of a generated image when editing an image based on text (for example, replacing one or more targets in the image with other targets, changing the position of one or more targets in the image, eliminating one or more targets in the image, etc.), or can be applied to other scenarios of generating an image. Alternatively, the image processing method can be applied to improving the resolution of an image obtained by an electronic device in a poor network environment. Alternatively, the image processing method can be applied to improving a low-resolution image input by an electronic device. The embodiments of the present application do not limit the scenarios in which the image processing method can be used.

[0072] An example electronic device 100 provided by an embodiment of the present application is introduced below.

[0073] Figure 2 FIG. 1 is a structural schematic diagram of an electronic device 100 provided by an embodiment of the present application.

[0074] The embodiments are specifically described below by taking an electronic device 100 as an example. It should be understood that the electronic device 100 can have more or fewer components than those shown in FIG. 1, can combine two or more components, or can have a different component configuration. Figure 2 The various components shown in FIG. 1 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits. Figure 2

[0075] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a display screen 130, a sensor module 180, and the like. The sensor module 180 can include a pressure sensor 180A, a touch sensor 180B, and the like.

[0076] ​The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0077] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0078] The memory in the processor 110 can also be configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0079] The electronic device 100 realizes the display function through the GPU, the display screen 130, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 130 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0080] The display screen 130 is used to display images, videos, etc. The display screen 130 includes a display panel.

[0081] The NPU is a neural-network (NN) computing processor, which can quickly process input information by drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition and other applications, such as image recognition, face recognition, voice recognition, text understanding, generating images based on user text or voice instructions, editing image content, etc.

[0082] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to extend the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external memory card.

[0083] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function (such as a face recognition function, a fingerprint recognition function, a mobile payment function, etc.), and the like. The data storage area can store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0084] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 130. When a touch operation acts on the display screen 130, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A.

[0085] The touch sensor 180B, also referred to as a "touch panel". The touch sensor 180B can be disposed on the display screen 130, and the touch sensor 180B and the display screen 130 form a touch screen, also referred to as a "touch panel". The touch sensor 180B is used to detect a touch operation acting on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the touch event type.

[0086] The method for the electronic device 100 to generate a high-resolution image provided by the embodiments of the present application is described below.

[0087] Figure 3 is an architecture schematic diagram of the electronic device 100 generating a high-resolution image provided by the embodiments of the present application.

[0088] As shown in Figure 3 , the electronic device 100 includes an image feature acquisition module 301, a feature super-resolution module 302, a feature splitting module 303, and a feature decoder 304.

[0089] The image feature acquisition module 301 is configured to acquire a low-resolution latent space feature. The image feature acquisition module 301 can include an encoder and a feature sampling module 305.

[0090] The encoder can receive an input of a generation condition and then output a low-resolution latent space feature (which can also be referred to as a feature map in the embodiments of the present application). The latent space refers to a low-dimensional space that cannot be directly observed in the fields of statistics, machine learning and deep learning. The latent space feature (i.e., the feature map) can be represented in the form of a tensor and reflects the potential features or attributes of an image, such as the color, edge, texture and the like of the image. The resolution size of the latent space feature refers to the size of the latent space feature in the spatial dimension. The resolution size of the latent space feature is related to the resolution size of the image corresponding to the latent space feature. The smaller the resolution of the latent space feature, the smaller the resolution of the image.

[0091] In some embodiments, for example in the image-to-image scenario, the generation condition can include an image condition and a text condition, where the image condition can be an image to be processed, and the text condition can be a text used to indicate the processing content when processing the above-mentioned image to be processed. For example, the text condition can indicate deleting or replacing one or more targets in the image to be processed, changing the position of one or more targets in the image to be processed, and the like. For example, the image condition can be an image containing a cat lying on a sofa, and the text condition can be "eliminate the cat on the image", or "replace the cat on the image with a dog", and the like. The electronic device 100 can add random noise to the image condition (i.e., the image to be processed), and convert the text condition into a text embedding vector through a text encoder. It can be understood that the text encoder converts high-dimensional language into low-dimensional machine language that the electronic device 100 can understand, where the text embedding vector can contain semantic information, syntactic information, and the like of the text condition. Wherein the above-mentioned random noise can be generated by an encoder. The above-mentioned text encoder can be, for example, a text encoder in a contrastive language-image pre-training (CLIP) model. The encoder can convert the image to be processed with random noise into a feature map in a latent space, and then the electronic device 100 can perform one or more downsampling and / or upsampling operations on the feature map based on the text embedding vector to predict the random noise in the feature map, and then eliminate the random noise in the feature map. Wherein, when predicting the random noise in the feature map based on the text embedding vector, the feature sampling module 305 will change the feature map towards the image content indicated by the generation condition, for example, change the feature value in the feature map indicating a cat to a feature value indicating a dog, so that the final decoded image changes the cat in the initial image to a dog. Finally, the feature sampling module 305 can obtain a low-resolution feature of the image to be generated indicated by the text condition.

[0092] In some embodiments, for example in the image-to-image scenario, the generation condition can include an image condition and a text condition, where the image condition can be an image to be processed, and the text condition can be a text used to indicate the processing content when processing the above-mentioned image to be processed. For example, the text condition can indicate deleting or replacing one or more targets in the image to be processed, changing the position of one or more targets in the image to be processed, and the like. For example, the image condition can be an image containing a cat lying on a sofa, and the text condition can be "eliminate the cat on the image", or "replace the cat on the image with a dog", and the like. The electronic device 100 can add random noise to the image condition (i.e., the image to be processed), and convert the text condition into a text embedding vector through a text encoder. It can be understood that the text encoder converts high-dimensional language into low-dimensional machine language that the electronic device 100 can understand, where the text embedding vector can contain semantic information, syntactic information, and the like of the text condition. Wherein the above-mentioned random noise can be generated by an encoder. The above-mentioned text encoder can be, for example, a text encoder in a contrastive language-image pre-training (CLIP) model. The encoder can convert the image to be processed with random noise into a feature map in a latent space, and then the electronic device 100 can perform one or more downsampling and / or upsampling operations on the feature map based on the text embedding vector to predict the random noise in the feature map, and then eliminate the random noise in the feature map. Wherein, when predicting the random noise in the feature map based on the text embedding vector, the feature sampling module 305 will change the feature map towards the image content indicated by the generation condition, for example, change the feature value in the feature map indicating a cat to a feature value indicating a dog, so that the final decoded image changes the cat in the initial image to a dog. Finally, the feature sampling module 305 can obtain a low-resolution feature of the image to be generated indicated by the text condition. Figures 1A-1BAs an example of the illustrated scenario, the text condition can be "draw a puppy sticking out its tongue". The electronic device 100 can then convert the text condition into a text embedding vector. The electronic device 100 can generate a random noise map through an encoder, where the pixel value of each pixel point in the random noise map is random, and the noise can be Gaussian noise. The electronic device 100 can convert the random noise map into a feature map in the latent space through the encoder, and then the electronic device 100 can gradually eliminate the random noise in the feature map through one or more downsampling and / or upsampling operations based on the text embedding vector by the feature sampling module, so that the feature value of the feature map changes towards the image content indicated by the text condition. Finally, the feature sampling module 305 can obtain a low-resolution feature of an image generated by the user's intention indicated by the text condition.

[0093] In some embodiments, the generated condition can only include an image condition. The image condition can be a low-resolution image to be improved in resolution. The electronic device 100 can convert the above-mentioned low-resolution image into a feature map in the latent space through the encoder, and then the electronic device 100 can extract the image features of the low-resolution image through the upsampling operation and / or downsampling of the feature sampling module, and further obtain a low-resolution feature map.

[0094] The downsampling operation can be used to reduce the resolution of the feature map. The feature sampling module can realize the downsampling of the feature map through a 3x3 convolution layer, a ReLU activation function and a max-pooling layer, thereby reducing the resolution of the feature map and increasing the depth of feature extraction. The upsampling operation is used to improve the resolution of the feature map, and the feature sampling module can realize the upsampling of the feature map through an upsampling convolution, thereby increasing the feature information of the feature map.

[0095] Optionally, the above-mentioned encoder can be an encoder in a variational autoencoder (VAE). The feature sampling module can include a U-shaped network (U-Net).

[0096] The low-resolution feature map generated by the image feature acquisition module 301 can be input to the feature super-resolution module 302 and the feature decoder 304 for further processing.

[0097] The feature super-resolution module 302 can be used to improve the resolution of the feature map, convert the low-resolution feature map into a high-resolution feature map, and then input the high-resolution feature map to the feature splitting module 303 for processing. The content of the feature super-resolution module can be referred to the introduction of the subsequent embodiments, which will not be expanded here. The feature super-resolution module can also be referred to as a multi-scale feature extraction model.

[0098] The feature splitting module 303 can receive a high-resolution feature map, split the high-resolution feature map into a plurality of sub-feature maps by region. The plurality of sub-feature maps can constitute a complete high-resolution feature map. Then, the feature splitting module can input the sub-feature maps obtained by splitting into the feature decoder 304.

[0099] The feature decoder 304 is used to restore the feature map to a visible image. Among them, the feature decoder 304 can decode the plurality of sub-feature maps obtained by splitting to obtain a plurality of sub-images. Then the feature decoder 304 can splice the plurality of sub-images through the image recombination module 306 to obtain a complete high-resolution image. Since the high-resolution image is spliced from a plurality of sub-images, there may be color differences between the sub-images, for example, the difference in pixel value at the junction between two sub-images may be large. The image recombination module can adjust the color of the sub-image of each region in the high-resolution image based on the low-resolution feature map, thereby eliminating the color difference between each region in the high-resolution image. Finally, the feature decoder 304 can output the high-resolution image, so that the electronic device 100 can display the high-resolution image for the user to view. The structure of the feature decoder will be introduced in the subsequent embodiments, which will not be expanded here. The feature decoder can also be referred to as a decoding model.

[0100] Exemplarily, the initial feature map obtained by the image feature acquisition module 301 can be 64x64x4, wherein 64x64 is the size of the feature map in the spatial dimension, and 4 is the number of channels of the feature map. The feature super-resolution module 302 can perform super-resolution on the feature map to obtain a feature map of 256x256x4. Among them, 256x256 is the size of the feature map in the spatial dimension. The feature splitting module 303 can split the feature map, for example, split the 256x256x4 feature map into 16 blocks, so that 16 64x64x4 feature maps are obtained. Further, the 16 64x64x4 feature maps are connected in parallel with the initial 64x64x4 feature map and input into the feature decoder 304, and the feature decoder 304 can decode the 16 64x64x4 feature maps to obtain 16 sub-images, and then splice the 16 sub-images into a complete high-resolution image. Further, the feature decoder can also adjust the color of the above complete image based on the initial 64x64x4 feature map, so that the color at the junction between each region and other regions can smoothly transition, avoid the color of a sub-image being too conspicuous in the entire complete image, and so on.

[0101] That is, the electronic device 100 generates low-resolution latent space features when generating latent space features, which is less computationally intensive than directly generating high-resolution latent space features, and the electronic device 100 generates latent space features faster. The electronic device 100 can super-resolve the low-resolution latent space features, and then decode based on the high-resolution latent space features to obtain the final image. The above method is less computationally intensive than the electronic device directly generating high-resolution latent space features and then generating an image based on the high-resolution latent space features, and the electronic device 100 can provide a high-resolution image to the user faster.

[0102] The embodiments of the present application are not limited to the architecture of the electronic device shown in Figure 3 The electronic device 100 can include more or fewer modules, or combine certain modules, or split certain modules when generating a high-resolution image than Figure 3 The embodiments of the present application are not limited to the architecture of the electronic device shown in

[0103] Here, first, the architecture of the feature super-resolution module 302 provided by the embodiments of the present application is introduced.

[0104] Figure 4 is a schematic diagram of an architecture of a feature super-resolution module 302 provided by the embodiments of the present application.

[0105] As shown in Figure 4 The feature super-resolution module 302 can receive a low-resolution feature map and then output a high-resolution feature map. The feature super-resolution module 302 can include a single-branch feature extraction module 401, Q multi-branch feature extraction modules 402, and a feature up-sampling module 403, where Q can be a positive integer, for example, Q can be 5, 6, 7, etc., and the embodiments of the present application do not limit the value of Q.

[0106] The single-branch feature extraction module 401 can include one or more convolution layers for extracting image features. For example, the single-branch feature extraction module 401 can include a convolution layer with a convolution kernel size of 3x3.

[0107] The multi-branch feature extraction module 402 can be used to further extract image features. The multi-branch feature extraction module 402 can include multiple parallel branches, and the multiple branches are used to extract image features from different spatial dimensions to ensure the complexity of feature extraction. The feature super-resolution module 302 can include multiple serial multi-branch feature extraction modules 402, for example, the feature super-resolution module 302 can include 7, 8, or 9 serial multi-branch feature extraction modules 402. The image features extracted by the multiple serial multi-branch feature extraction modules 402 can ensure the depth of feature extraction, thereby improving the fitting capability of the feature super-resolution module 302 when extracting image features, so as to ensure that the feature up-sampling module 403 will not lose the details of the image when up-sampling the image features. The architecture of the multi-branch feature extraction module 402 will be introduced later, and will not be expanded here.

[0108] The feature up-sampling module 403 can be used to improve the resolution of the feature map. The feature up-sampling module 403 can receive the feature map output by the multi-branch feature extraction module 402 and then output a high-resolution feature map. The feature map output by the multi-branch feature extraction module 402 can include multiple channels, and the feature up-sampling module 403 can convert the features in the feature map from the channel dimension to the spatial dimension, that is, reduce the number of channels of the feature map and increase the size of the feature map in the spatial dimension (that is, increase the size of the spatial feature), thereby improving the resolution of the feature map. Optionally, the feature up-sampling module 403 can include a pixel shuffle operation. The feature up-sampling module 403 can split and rearrange the features of multiple channels in the feature map to improve the resolution of the feature map in the spatial dimension. Not limited to the pixel shuffle operation, other up-sampling operations can also be used, which are not limited by the embodiments of the present application.

[0109] The feature super-resolution module 302 can include more or fewer software modules than those shown in FIG. 3, or combine certain software modules, or split certain software modules, which are not limited by the embodiments of the present application. Figure 4 The feature super-resolution module 302 can include more or fewer software modules than those shown in FIG. 3, or combine certain software modules, or split certain software modules, which are not limited by the embodiments of the present application.

[0110] The architecture of the multi-branch feature extraction module 402 provided by the embodiments of the present application will be introduced in detail below.

[0111] Figure 5 FIG. 4 is a schematic diagram of the architecture of the multi-branch feature extraction module 402 provided by the embodiments of the present application.

[0112] As shown in FIG. 4, the multi-branch feature extraction module 402 can include multiple parallel branches, and the multiple branches are used to extract image features from different spatial dimensions to ensure the complexity of feature extraction. Figure 5As shown, the multi-branch feature extraction module 402 can include a deep feature extraction module 501 and a normal feature extraction module 502. After receiving the feature map, the multi-branch feature extraction module 402 first processes the feature map through the deep feature extraction module 501. The deep feature extraction module 501 can extract features from the feature map based on a plurality of deep separable convolutional layers of multiple branches, and the feature extraction results of the plurality of deep separable convolutional layers are input to the normal feature extraction module 502 after being processed by splicing and the like. Then, the normal feature extraction module 502 can further extract image features through a plurality of convolutional layers of multiple branches, and then combine the feature extraction results of the multiple branches. Finally, the multi-branch feature extraction module 402 can input the feature extraction results to the feature upsampling module 403.

[0113] (1) Deep feature extraction module 501

[0114] The deep feature extraction module 501 can include a layer normalization module, a plurality of multi-branch deep convolutional layers of multiple branches, a channel dimension splicing processing module 503, a 1x1 convolutional layer module, a SiLU (sigmoid linear unit) activation layer, an element-wise multiplication module 504, and an element-wise addition module 505.

[0115] After the deep feature extraction module 501 receives the feature map, first, the layer normalization module is used to perform layer normalization (LayerNorm) processing on the feature map. Then, the deep feature extraction module 501 inputs the layer normalization processed feature map into multiple branches for separate processing. In addition, the deep feature extraction module 501 will also input the layer normalization processed feature map into the element-wise multiplication module 504. The multiple branches may, for example, include branch 1, branch 2, branch 3, branch 4, and the like. Among them, branch 1 can include a multi-branch deep convolutional layer (3). Branch 2 can include a max pooling layer (2), a multi-branch deep convolutional layer (5), an up-sampling layer (2). Branch 3 can include a max pooling layer (4), a multi-branch deep convolutional layer (7), an up-sampling layer (4). Branch 4 can include a max pooling layer (8), a multi-branch deep convolutional layer (9), an up-sampling layer (8), and the like. In order to facilitate description, the numbers in the parentheses after the above-mentioned max pooling layer, multi-branch deep convolutional layer and up-sampling layer are the respective input parameters. For example, the input parameter of the multi-branch deep convolutional layer (3) in branch 1 is 3. The input parameter of the max pooling layer (2) in branch 2 is 2, which means that the max pooling operation is performed with a step of 2, the input parameter of the multi-branch deep convolutional layer (5) is 5, and the input parameter of the up-sampling layer (2) is 2, which means that the resolution of the received feature map is increased by 2 times. The multi-branch deep convolutional layer will be introduced later, and will not be expanded here. Among them, w represents the index of the branch, and the parameters of the max pooling layer and the up-sampling layer in branch w can be (w-1) times of 2. The parameter of the multi-branch deep convolutional layer in branch w can be 2xw+1, where w is an integer greater than 1. The deep feature extraction module can contain more or fewer branches than shown, which is not limited by the embodiments of the present application. Among them, the number of branches can be related to the size of the feature map. The larger the feature map in the spatial dimension, the more the number of branches. Figure 5

[0116] Figure 6 is an architecture schematic diagram of the multi-branch deep convolutional layer provided by the embodiments of the present application. As shown in Figure 6 ​As shown, each branch in the multi-branch deep convolutional layer can include a depth separable convolutional layer and a batch normalization layer (BatchNorm), and the depth separable convolutional layer can be used for depth convolution and point-wise convolution on the feature map. The number of branches of the multi-branch deep convolutional layer and the size of the convolution kernel of the depth separable convolutional layer in the multi-branch deep convolutional layer can be related to the incoming parameters of the multi-branch deep convolutional layer. When the incoming parameters of the multi-branch deep convolutional layer are n, the multi-branch deep convolutional layer can include (n-1) / 2 branches. The size of the convolution kernel of the depth separable convolutional layer in each branch can increase sequentially, for example, the size of the convolution kernel of the depth separable convolutional layer in the pth branch can be [n-2×(p-1)]×[n-2×(p-1)], and p is a positive integer.

[0117] As shown in the multi-branch deep convolutional layer (3) in Branch 1, the incoming parameters n of the multi-branch deep convolutional layer are 3, so the multi-branch deep convolutional layer (3) only includes a single branch, that is, p only takes 1, and the size of the convolution kernel of the depth separable convolutional layer in this branch is 3×3. That is, in the multi-branch deep convolutional layer (3), the electronic device uses a 3×3 depth separable convolutional layer to process the feature map, and then performs batch normalization. Figure 5 As shown in the multi-branch deep convolutional layer (7) in Branch 3, the incoming parameters n of the multi-branch deep convolutional layer are 7, so the multi-branch deep convolutional layer can include 3 branches, and p can take 1, 2, and 3. In the branch where p takes 1, the convolution kernel of the depth separable convolutional layer is 7×7; in the branch where p takes 2, the size of the convolution kernel of the depth separable convolutional layer is 5×5; and in the branch where p takes 3, the size of the convolution kernel of the depth separable convolutional layer is 3×3. That is, the multi-branch deep convolutional layer (7) includes 3 branches, and the electronic device uses 3×3, 5×5, and 7×7 depth separable convolutional layers to process the feature map, respectively, and performs batch normalization processing. Finally, the electronic device 100 uses the element-wise addition module 601 to element-wise add the three feature maps obtained by processing the above three branches, to obtain a complete feature map.

[0118] Figure 5 The depth feature extraction module 501 can process the feature map through the above-mentioned multiple branches, and extract the features of the feature map from multiple spatial dimensions using convolution kernels of different sizes. In this way, it can not only ensure that the entire spatial dimension of the feature map is covered when extracting image features, allowing feature values at all positions in the spatial dimension of the feature map to interact, but also reduce the amount of calculation, so that the image features can be extracted faster under the condition that the hardware of the electronic device is limited.

[0119] The depth feature extraction module 501 can process the feature map through the above-mentioned multiple branches, and extract the features of the feature map from multiple spatial dimensions using convolution kernels of different sizes. In this way, it can not only ensure that the entire spatial dimension of the feature map is covered when extracting image features, allowing feature values at all positions in the spatial dimension of the feature map to interact, but also reduce the amount of calculation, so that the image features can be extracted faster under the condition that the hardware of the electronic device is limited.

[0120] ​Further, the deep feature extraction module 501 can concatenate the results of processing the feature map by the multiple branches in the channel dimension, and the processing result of each branch can correspond to a channel. Further, the deep feature extraction module 501 can input the feature map obtained after concatenation into a 1x1 size convolution layer and a SiLU activation layer, wherein the 1x1 convolution layer can realize dimension reduction of the feature map channel, and convert some features from the channel dimension to the spatial dimension, facilitating subsequent decoding.

[0121] Further, the SiLU activation layer can input the feature map into the element-wise multiplication module 504, and the element-wise multiplication module 504 can perform element-wise multiplication on the feature values at each position in the feature map output by the SiLU activation layer and the feature map output by the aforementioned layer normalization, thereby realizing fusion of the features, so that the final result at each position in the feature map retains the original features and also retains the features extracted by the multi-branch deep convolution layer.

[0122] The deep feature extraction module 501 can input the feature map processed by the element-wise multiplication module 504 and the initial feature map (i.e., the feature map received by the multi-branch feature extraction module 402 from the single-branch feature extraction module 401) into the element-wise addition module 505, and then the element-wise addition module 505 performs element-wise addition on the above two feature maps, thereby realizing fusion between the processed features and the initial features.

[0123] Finally, the deep feature extraction module 501 can pass the calculation result of the element-wise addition module 505 to the ordinary feature extraction module 502. Through the deep separable convolution operation of the multiple branches, the electronic device 100 can better extract image features in the spatial dimension.

[0124] (2) Ordinary feature extraction module 502

[0125] The ordinary feature extraction module 502 can include a layer normalization module, a multi-branch convolution layer, a SiLU activation layer, a 1x1 convolution layer module, and an element-wise addition module 506.

[0126] The ordinary feature extraction module 502 can first input the feature map input by the deep feature extraction module 501 into the layer normalization module for layer normalization processing. Then, the ordinary feature extraction module 502 can input the feature map processed by the layer normalization into the multi-branch convolution layer to extract image features.

[0127] Figure 7 is an architecture schematic diagram of a multi-branch convolution layer provided by an embodiment of the present application. As shown in Figure 7As shown, the multi-branch convolution layer can include 2 branches. Among them, branch 1 can include a 3x3 convolution layer and a batch normalization layer, and branch 2 can include a 1x1 convolution layer and a batch normalization layer. The ordinary feature extraction module 502 can process the feature maps respectively by using the above two branches, and then fuse the features of the two feature maps through the element-by-element addition module 701. In some embodiments, the multi-branch convolution layer can include more or fewer branches, for example, the multi-branch convolution layer can also include a branch of a 5x5 convolution layer and a batch normalization layer, a branch of a 7x7 convolution layer and a batch normalization layer, etc. Among them, the convolution kernel size of the convolution layer of each branch can be increased relative to the convolution kernel size of the convolution layer of the previous branch. The number of branches that the multi-branch convolution layer can include is not limited in the embodiments of the present application. The electronic device 100 can implement operations in the channel dimension and the spatial dimension through the multi-branch convolution layer, thereby better extracting image features.

[0128] In some embodiments, the ordinary feature extraction module 502 can include L multi-branch convolution layers in series, where L can be any positive integer, for example, L can be 2, 3, 4, etc., which can increase the depth of feature extraction. The number of multi-branch convolution layers in series is not limited in the embodiments of the present application.

[0129] The ordinary feature extraction module 502 can input the output result of the multi-branch convolution layer to the SiLU activation layer and the 1x1 convolution layer for processing, and then reduce the dimension of the channel of the feature map. Finally, the ordinary feature extraction module 502 can use the element-by-element addition module 506 to perform element-by-element addition on the feature map output by the 1x1 convolution layer and the feature map obtained from the deep feature extraction module 501 by the ordinary feature extraction module 502, thereby realizing feature fusion. Finally, the ordinary feature extraction module 502 can send the feature map obtained by the above feature fusion to the feature splitting module 303.

[0130] In some embodiments, the multiple branch depth separable convolution layers can be re-parameterized. Figure 8 is an example diagram provided by an embodiment of the present application for re-parameterizing multiple depth separable convolution layers. The depth separable convolution layer A has a 7x7 convolution kernel, the depth separable convolution layer B has a 5x5 convolution kernel, and the depth separable convolution layer C has a 3x3 convolution kernel. Among them, the convolution kernel includes a plurality of weights, for example, a11, a12, a13, etc. After the electronic device 100 adjusts the weights in the convolution kernel through multiple training of the above depth separable convolution layers, the electronic device 100 can re-parameterize the multiple depth separable convolution layers, that is, combine the multiple depth separable convolution layers into one convolution layer. Among them, the electronic device aligns the center elements of each depth separable convolution layer, and then realizes re-parameterization by element-by-element addition according to the position. For example,Figure 8 As shown, after the alignment of the depth separable convolution layer A, the depth separable convolution layer B and the depth separable convolution layer C, a44 of the depth separable convolution layer A, b33 of the depth separable convolution layer B and c22 of the depth separable convolution layer C correspond to the same position, and then the electronic device 100 can add each depth separable convolution layer element by element according to the position to obtain a total depth separable convolution layer D. The weight of the center position of the depth separable convolution layer D is obtained based on the above three depth separable convolution layers, and the weight is a44+b33+c22. The weight of the outermost layer of the depth separable convolution layer D is obtained based on the larger depth separable convolution layer A, such as a11, a12 and the like in the depth separable convolution layer D. The re-parameterized depth separable convolution layer D has the weights of the depth separable convolution layer A, the depth separable convolution layer B and the depth separable convolution layer C. The electronic device 100 can directly use the depth separable convolution layer D to realize feature extraction. In this way, the electronic device can ensure the effect of feature extraction, and there is no need to calculate the convolution results of the depth separable convolution layers of each branch respectively each time, thereby reducing the calculation amount. The electronic device 100 can also re-parameterize the depth separable convolution layers of more or less branches. The way in which the electronic device 100 re-parameterizes the depth separable convolution layers of more or less branches can refer to the way in which the electronic device 100 re-parameterizes the depth separable convolution layers of three branches as shown in the above Figure 8 The way in which the electronic device 100 re-parameterizes the depth separable convolution layers of three branches as shown in the above

[0131] In some embodiments, the electronic device 100 can also re-parameterize the weights of each convolution layer in the multi-branch depth convolution layer, or the electronic device 100 can also re-parameterize the weights of each batch-normalized convolution layer in the multi-branch depth convolution layer. The way in which the electronic device 100 re-parameterizes each convolution layer and the way in which the electronic device 100 re-parameterizes each batch-normalized convolution layer can refer to the introduction of the embodiments as shown in the above Figure 6 The way in which the electronic device 100 re-parameterizes the depth separable convolution layers of three branches as shown in the above Figure 8 The way in which the electronic device 100 re-parameterizes the depth separable convolution layers of three branches as shown in the above

[0132] The following introduces the method in which the electronic device 100 generates a high-resolution image based on the high-resolution feature map output by the feature super-resolution module 302, by combining the feature splitting module 303 and the feature decoder 304.

[0133] Figure 9 is a flowchart of the electronic device 100 decoding a high-resolution feature map to generate a high-resolution image according to an embodiment of the present application.

[0134] AsFigure 9 As shown, the high-resolution feature map output by the feature super-resolution module 302 is first input to the feature splitting module 303. The feature splitting module 303 can divide the high-resolution feature map into multiple regions, and then split the high-resolution feature map into multiple sub-feature maps based on the divided regions. The resolution of the sub-feature maps can be the same as the resolution of the low-resolution feature map obtained by the image feature acquisition module 301, which can facilitate subsequent splicing of the sub-feature maps and the low-resolution feature map obtained by the image feature acquisition module 301. For example, as shown in the figure, the size of the low-resolution feature map obtained by the image feature acquisition module 301 can be 64x64x4, and the size of the high-resolution feature map output by the feature super-resolution module 302 can be 256x256x4, where 4 is the number of channels. Then, the feature splitting module 303 can divide the high-resolution feature map into 16 regions of the same size, and then split the high-resolution feature map to obtain 16 sub-feature maps, so that the size of each sub-feature map obtained after splitting is 64x64x4. Figure 9

[0135] ​The electronic device 100 can splice the plurality of sub-feature maps (for example, 16 64x64x4 feature maps) obtained after splitting the high-resolution feature map with the low-resolution feature map (for example, one 64x64x4 feature map) obtained by the image feature acquisition module 301 in the batch dimension, and then input the spliced feature map (a total of 17 64x64x4 feature maps) into the feature decoder 304. In turn, the feature decoder 304 can perform decoding processing on the plurality of feature maps (i.e., 17 64x64x4 feature maps) in parallel through the convolutional decoder. The decoding processing is used to gradually restore the feature map from a lower layer, a hidden space tensor that is not easy for a human to observe, to a higher layer, an image containing specific image content, which is easy for a human to observe and understand. The decoding processing can include one or more upsampling operations and / or one or more downsampling operations. During the decoding processing, the size of the feature map in the spatial dimension and the number of channels change multiple times with the upsampling and downsampling operations. Then the feature decoder 304 can splice the higher layer sub-feature maps (for example, converted into 16 512x512x128 feature maps, where 128 is the number of channels) obtained after decoding processing of the plurality of sub-feature maps (i.e., 16 64x64x4 feature maps) according to their positions before splitting in the spatial dimension, to obtain a high-resolution feature map (for example, one 2048x2048x128 feature map). In turn, the feature decoder 304 can input the high-resolution feature map (i.e., the 2048x2048x128 feature map) into the output convolutional layer to obtain a high-resolution image (the size of the image can be 2048x2048x3, where 3 is the R / G / B three-channel). Since the high-resolution image (2048x2048x3) is obtained by decoding and splicing the plurality of sub-feature maps (i.e., the 16 64x64x4 feature maps) respectively, the colors between the corresponding regions of each sub-feature map and the colors of the boundaries of each region can be uneven. To solve this problem, the feature decoder 304 can perform linear stretching on the higher layer feature map (for example, one 512x512x128 feature map, where 128 is the number of channels) obtained by decoding (i.e., one 64x64x4 feature map) the low-resolution feature map obtained by the image feature acquisition module 301, so that the size of the linearly stretched feature map is the same as the size of the plurality of sub-feature maps after splicing. The linear stretching is used to stretch the size of the image, and the linear stretching can include nearest neighbor interpolation, bilinear interpolation, etc., which are not limited by the embodiments of the present application. In turn, the feature decoder 304 can input the linearly stretched feature map (for example, a 2048x2048x128 feature map) into the color alignment convolutional layer.The color alignment convolutional layer can receive the input of the linearly stretched feature map, and then output a tensor containing color adjustment values for each position in the high-resolution image. Finally, the feature decoder 304 can adjust the color of each region in the high-resolution image (e.g., a 2048x2048x3 image) output by the output convolutional layer based on the above-mentioned tensor containing color adjustment values (e.g., a tensor composed of 2048x2048x3 adjustment values), to obtain a high-resolution image with more balanced colors in each region. Wherein, the feature decoder 304 can adjust the color of the high-resolution image by element-wise addition of the above-mentioned tensor and the high-resolution image, which is not limited to element-wise addition, but can also be element-wise multiplication, etc. The present application does not limit this.

[0136] In some embodiments, the structure of the convolutional decoder can be the same as the first layer to the second-to-last layer neural network in the decoder of the variational autoencoder (VAE), and the structure of the output convolutional layer and the color alignment layer can be the same as the last layer neural network structure in the decoder structure of the VAE encoder.

[0137] Finally, the electronic device 100 can display the high-resolution image for the user to view.

[0138] The training process of the feature super-resolution module and the feature decoder is introduced below.

[0139] Wherein, the feature super-resolution module and the feature decoder include the following two steps during training:

[0140] I. Training the feature super-resolution module based on the latent space domain loss.

[0141] Figure 10 is a flowchart of training the feature super-resolution module based on the latent space domain loss provided by an embodiment of the present application.

[0142] As Figure 10As shown, the electronic device 100 can use a plurality of high-resolution sample images for training. Among them, the electronic device 100 can first down-sample the high-resolution sample images, reduce the resolution of the high-resolution sample images, and obtain low-resolution images. The above-mentioned down-sampling can be realized through the degradation process of the enhanced super-resolution generative adversarial network (Real-ESRGAN). The resolution reduction rate of the high-resolution sample images during training can be equal to the rate when the electronic device subsequently enlarges the image resolution through the feature super-resolution module and the feature decoder. Then, the electronic device 100 can use an encoder to encode the low-resolution image (which can be referred to as a second sample image here) to obtain a low-resolution latent space feature (which can also be referred to as a low-resolution feature map) corresponding to the low-resolution image. Further, the low-resolution latent space feature can be input to the feature super-resolution module, and the feature super-resolution module can perform super-resolution processing on the low-resolution latent space feature to obtain a model predicted latent space feature. The electronic device can also use an encoder to encode the high-resolution sample image to obtain a real latent space feature corresponding to the high-resolution sample image. The high-resolution sample image can also be referred to as a first sample image.

[0143] In some embodiments, the electronic device 100 can use an absolute error loss function (mean absolute error, MAE) to calculate the loss value of the model predicted latent space feature and the real latent space feature. The above-mentioned absolute error loss function can also be referred to as an L1 loss function. Among them, the L1 loss function refers to calculating the absolute value of the difference between the feature values at each position in the model predicted latent space feature and the real latent space feature, and then adding the absolute values of the differences between the feature values at all positions in the two feature maps, denoted as where N is the number of feature points in the feature map, T i is the feature value of the i-th feature point in the real latent space feature, T′ i is the feature value of the i-th feature point in the model predicted latent space feature, and the L1 loss function can also be referred to as a first loss function.

[0144] In some embodiments, the electronic device can also determine the value of the adversarial loss Lgan by setting a discriminator. The discriminator can receive the model predicted latent space features and the real latent space features, and then the discriminator can determine whether the feature maps it receives are the latent space features generated by the feature super-resolution module or the latent space features generated by the sample image. Further, the electronic device 100 can determine the probability a that the discriminator identifies the latent space features generated by the sample image successfully. On the other hand, the electronic device 100 can also determine the probability b that the model predicted latent space features generated by the feature super-resolution module are incorrectly determined by the discriminator as real latent space features. The electronic device can make the size of a and b approximately balanced during the training process. Then the electronic device 100 can determine the adversarial loss Lgan based on the two probabilities described above. Wherein, Lgan can be calculated by using cross-entropy loss, for example, Lgan can be loga+log(1-b). Not limited to this, Lgan can also be other values, and the embodiments of the present application do not limit this. In this way, the electronic device can improve the similarity between the model predicted latent space features generated by the feature super-resolution module 302 and the real latent space features, and increase the consistency of the feature distribution of the model predicted latent space features and its corresponding real latent space features. The above training method can also be called adversarial training based on, and Lgan can also be called a generative adversarial network GAN loss function. Lgan loss function can also be called a second loss function.

[0145] Finally, the electronic device 100 can determine the latent space domain loss function based on one or both of L1 and Lgan by using the back propagation algorithm, and then train the parameters of the feature super-resolution module by using the latent space domain loss function until the value of the latent space domain loss function converges. For example, the latent space domain loss function can be the sum of L1 and Lgan.

[0146] II. Training the feature super-resolution module and the decoder based on the image domain loss

[0147] Figure 11 is a flowchart of training the feature super-resolution module and the decoder based on the image domain loss provided by the embodiments of the present application.

[0148] As Figure 11As shown, the electronic device 100 can employ the encoder, the feature super-resolution module trained in the first step to process the low-resolution image to obtain the model predicted latent space feature, and then the electronic device 100 can split the model predicted latent space feature and input it into the feature decoder. The method of the electronic device 100 for splitting the model predicted latent space feature can refer to the method of the feature splitting module 303 splitting the feature map as described above, which will not be described here. The split feature map can be input into the feature decoder 304 to generate a high-resolution predicted image. Further, the electronic device 100 can determine the size of the loss value based on the high-resolution sample image and the high-resolution predicted image.

[0149] In some embodiments, the electronic device 100 can employ the mean square error (MSE) to calculate the loss value between the high-resolution sample image and the high-resolution predicted image. The mean square error can also be referred to as the L2 loss function. The L2 loss function refers to calculating the sum of the squares of the differences between the pixel values of each position of the high-resolution sample image and the high-resolution predicted image, denoted as where N is the number of pixel points, y i is the pixel value of the i-th pixel point in the high-resolution sample image, y′ i is the pixel value of the i-th pixel point in the high-resolution predicted image. In this way, the electronic device can better train the parameters of the feature super-resolution module and the decoder by comparing the deviation of the pixel values of each pixel point of the two images, and improve the boundary consistency between the high-resolution sample image and the high-resolution predicted image. The L2 loss function can also be referred to as the third loss function.

[0150] In some embodiments, the electronic device 100 can also calculate the perceptual loss (LPIPS) between the high-resolution sample image and the high-resolution predicted image. The LPIPS is used to measure the similarity between the high-resolution sample image and the high-resolution predicted image in visual perception. The electronic device 100 can extract the features of the high-resolution sample image and the high-resolution predicted image, and then calculate the Euclidean distance or Manhattan distance between the two features to determine the value of LPIPS. In this way, the electronic device can better train the parameters in the feature super-resolution module and the decoder from the perspective of visual perception, so that the high-resolution predicted image generated by the electronic device is closer to the high-resolution sample image in visual perception. The LPIPS loss can also be referred to as the fourth loss function.

[0151] In some embodiments, the electronic device 100 can also calculate a color loss value between the high-resolution sample image and the high-resolution, which is used to determine the color consistency between the high-resolution sample image and the high-resolution predicted image. Wherein the electronic device 100 can first blur the high-resolution sample image and the high-resolution predicted image, and then calculate the loss value of the two images using the L2 loss function (that is, calculate the sum of the square differences between the pixel values of each position in the blurred high-resolution sample image and the high-resolution predicted image), and finally obtain the color loss value, denoted as Where N is the number of pixels, b i is the pixel value of the i-th pixel point in the blurred high-resolution sample image, b i is the pixel value of the i-th pixel point in the blurred high-resolution predicted image. In this way, the electronic device can better train the parameters in the feature super-resolution module and the decoder from the perspective of color and color, so that the color of the high-resolution predicted image generated by the electronic device is closer to that of the high-resolution sample image. The color loss can also be referred to as the fifth loss function.

[0152] Finally, the electronic device 100 can determine the image domain loss function in the second step training based on one or more of the above L2 loss function, LPIPS loss and color loss colorloss through the back propagation algorithm, and then train the feature super-resolution module 302 and the feature decoder 304, until the image domain loss function converges. Optionally, the image domain loss function can be the sum of the L2 loss function, the LPIPS loss and the color loss colorloss. Optionally, when training the color alignment convolution layer in the feature decoder 304, the initial weight and the initial bias in the color alignment convolution layer can be initialized to 0, thereby improving the training efficiency.

[0153] Not limited to training the parameters of the feature super-resolution module and the feature decoder by the electronic device 100, the parameters of the feature super-resolution module and the feature decoder can also be trained on a server, and the present application embodiment does not limit this.

[0154] Figure 12 is a flowchart of an image processing method provided by the present application embodiment.

[0155] As Figure 12 shown, the image processing method can include but is not limited to the following steps:

[0156] S1201, the electronic device extracts a plurality of scale hidden space features from the first hidden space feature, and splices the plurality of scale hidden space features to obtain a second hidden space feature, the resolution of the second hidden space feature is higher than that of the first hidden space feature.

[0157] The first latent space feature can be, for example Figure 3 The low-resolution feature map obtained by the image feature acquisition module 301. The electronic device 100 can extract latent space features of multiple scales from the first latent space feature by using the feature super-resolution module 302. The electronic device 100 can extract latent space features of multiple scales by using the multi-branch feature extraction module 402 in the feature super-resolution module 302. The method of extracting latent space features of multiple scales by using the multi-branch feature extraction module 402 is described in the foregoing embodiments, which will not be described here.

[0158] The latent space feature of one scale in the latent space features of multiple scales can be obtained by the electronic device 100 by using one branch (for example, branch 1, branch 2, and the like) of the deep feature extraction module 501 in the multi-branch feature extraction module 402. The one branch can include one or more deep separable convolution operations, and the sizes of the convolution kernels in the multiple deep separable convolution operations are different from each other. The calculation in each branch can refer to the architecture diagram of the multi-branch deep convolution layer shown in FIG. 3B, which will not be described here. Figure 5 The latent space feature of one scale in the latent space features of multiple scales can be obtained by the electronic device 100 by using one branch (for example, branch 1, branch 2, and the like) of the deep feature extraction module 501 in the multi-branch feature extraction module 402. The one branch can include one or more deep separable convolution operations, and the sizes of the convolution kernels in the multiple deep separable convolution operations are different from each other. The calculation in each branch can refer to the architecture diagram of the multi-branch deep convolution layer shown in FIG. 3B, which will not be described here. Figure 6 The latent space feature of one scale in the latent space features of multiple scales can be obtained by the electronic device 100 by using one branch (for example, branch 1, branch 2, and the like) of the deep feature extraction module 501 in the multi-branch feature extraction module 402. The one branch can include one or more deep separable convolution operations, and the sizes of the convolution kernels in the multiple deep separable convolution operations are different from each other. The calculation in each branch can refer to the architecture diagram of the multi-branch deep convolution layer shown in FIG. 3B, which will not be described here. Figure 5 The electronic device can obtain the latent space feature of one scale in the latent space features of multiple scales by using the 3x3 deep separable convolution and the 5x5 deep separable convolution in the branch 2 shown in FIG. 3B. Then, the electronic device 100 can splice the latent space features of multiple scales by using the channel dimension splicing processing module 503.

[0159] In some embodiments, after the electronic device obtains the second latent space feature, the method further includes: the electronic device 100 performing multiple convolution operations on the second latent space feature respectively, and fusing the results of the multiple convolution operations. The electronic device 100 can perform multi-branch convolution on the second latent space feature by using the ordinary feature extraction module 502, and then fuse the results of the convolution by using the element-by-element addition module 506. Figure 5 The electronic device 100 can perform multi-branch convolution on the second latent space feature by using the ordinary feature extraction module 502, and then fuse the results of the convolution by using the element-by-element addition module 506.

[0160] In some embodiments, the first latent space feature is a latent space feature for describing an image generated by user intention. For example, in the text-to-image scenario, the image generated by user intention can be determined based on a first text input by the user. The electronic device 100 can identify the image generated by user intention in the first text, then generate random noise, and further generate the latent space feature based on the random noise and the first text. The method of generating the latent space feature by the electronic device 100 based on the text condition input by the user can refer to the description of the foregoing embodiments, which will not be described here. For example, the text condition can be “draw a cat”, “draw a ship on the sea”, etc. The text condition here can also be referred to as the first text. Alternatively, the image generated by user intention can be determined based on a second text input by the user and a first image input by the user, where the second text can be used to indicate the editing content of the first image. For example, the content of the first image is a ship on the sea, and the second text can be “replace the ship in the image with a fish”, “eliminate the ship in the image”, etc. Further, the electronic device 100 can generate the first latent space feature based on the random noise, the second image and the second text. Alternatively, in some embodiments, the electronic device can also perform super-resolution processing on an existing image, where the first latent space feature is any one of the following: a latent space feature of an image input by the user, a latent space feature of an image stored by the electronic device, and a latent space feature of an image obtained by the electronic device from the network. The above-mentioned image input by the user, the image stored by the electronic device, and the image obtained by the electronic device from the network can be a lower resolution image. Further, the electronic device 100 can extract the latent space feature in the above-mentioned image through the encoder, which is the first latent space feature.

[0161] S1202, the electronic device splits the second latent space feature into a plurality of third latent space features, and the resolution of the third latent space feature is lower than that of the second latent space feature.

[0162] The electronic device 100 can split the second latent space feature, which can be regarded as a high-resolution feature map as shown in the embodiment of Figure 9 , and obtain a plurality of sub-features, Figure 9 which are third latent space features. Figure 9

[0163] S1203, the electronic device decodes the plurality of third latent space features and the first latent space feature in parallel, and splices the decoded plurality of third latent space features to obtain the first image data.

[0164] Further, the electronic device 100 can adopt Figure 9 ​The feature decoder 304 shown decodes the third latent space feature and the first latent space feature in parallel. Further, the electronic device 100 can employ the image recombination module 306 to stitch the plurality of third latent space features, and input the stitched plurality of third latent space features into an output convolutional layer to obtain the first image data.

[0165] S1204, the electronic device obtains second image data by using the first image data and fourth latent space features, the fourth latent space features being obtained by linearly stretching the decoded first latent space features, and the fourth latent space features having the same resolution as the first image data.

[0166] The electronic device 100 can linearly stretch the decoded first latent space features to obtain the fourth latent space features. Further, the electronic device 100 can input the fourth latent space features into a color alignment convolutional layer to obtain first color compensation values, for example, a tensor containing compensation values of each position in the red / green / blue (R / G / B) channel in the first image data. Further, the electronic device 100 can adjust the color in the first image data based on the first color compensation values, for example, add the compensation values of each position in the first color compensation values to the pixel values of the corresponding positions in the first image data to obtain the second image data, which can contain pixel values (which can include R / G / B three channels) of a plurality of pixel points for composing an image.

[0167] In some embodiments, in the scenario where the electronic device performs super-resolution processing on an existing image, after the electronic device obtains the second image data by using the first image data and the fourth latent space features, the method further includes: the electronic device displays a first image on the display screen based on the second image data, wherein the resolution of the first image can be greater than the resolution of the image input by the user, the resolution of the first image is greater than the resolution of the image stored by the electronic device, and the resolution of the first image is greater than the resolution of the image obtained by the electronic device from the network.

[0168] The device for executing the above method provided by the embodiments of the present application is described below. As shown in Figure 13 , Figure 13 is a structural schematic diagram of an image processing device provided by the embodiments of the present application. The image processing device 1300 can be an electronic device or a chip or chip system in the electronic device in the embodiments of the present application.

[0169] As shown in Figure 13 , the image processing device 1300 can include an image feature extraction unit 1301, an image feature stitching unit 1302, an image feature splitting unit 1303, and an image feature decoding unit 1304.

[0170] The image feature extraction unit 1301 is configured to extract multiple scale latent space features from the first latent space feature. The image feature splicing unit 1302 is configured to splice the multiple scale latent space features to obtain a second latent space feature. The image feature splitting unit 1303 is configured to split the second latent space feature into multiple third latent space features. The image feature decoding unit 1304 is configured to decode the multiple third latent space features and the first latent space feature in parallel, and splice the decoded multiple third latent space features to obtain first image data. The image feature decoding unit 1304 is further configured to obtain second image data by using the first image data and a fourth latent space feature.

[0171] In some embodiments, one of the multiple scale latent space features is a result of one or more times of fusing a deep separable convolution operation on the first latent space feature by the image feature extraction unit 1301.

[0172] In some embodiments, after the image feature splicing unit 1302 obtains the second latent space feature, the image feature extraction unit 1301 further performs multiple convolution operations on the second latent space feature respectively, and fuses results of the multiple convolution operations.

[0173] In some embodiments, the image feature decoding unit 1304 obtains the second image data by using the first image data and the fourth latent space feature, including: the image feature decoding unit 1304 linearly stretches the first latent space feature to obtain the fourth latent space feature; the image feature decoding unit 1304 determines a first color compensation value based on the fourth latent space feature; and the image feature decoding unit 1304 compensates colors of the first image data based on the first color compensation value to obtain the second image data.

[0174] In some embodiments, the image processing apparatus 1300 further includes a display unit, and after the image feature decoding unit 1304 obtains the second image data by using the first image data and the fourth latent space feature, the display unit displays the first image based on the second image data.

[0175] Figure 14 FIG. 14 is a structural schematic diagram of a chip according to an embodiment of the present application. As shown in FIG. 14, the chip 1400 includes one or more (including two) processors 1401, a bus 1402, and a communication interface 1403. Optionally, the chip 1400 further includes a memory 1404. Figure 14

[0176] In some embodiments, the memory 1404 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0177] ​The method described in the embodiments of the present application can be applied to the processor 1401 or implemented by the processor 1401. The processor 1401 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the software form of the hardware in the processor 1401. The processor 1401 described above can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, a discrete gate or transistor logic device, or a discrete hardware component. The processor 1401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0178] The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing executed by a code processor, or executed by a combination of hardware and software modules in the code processor. Among them, the software module can be located in a mature storage medium in the field, such as random access memory, read-only memory, programmable read-only memory or electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 1404, and the processor 1401 reads the information in the memory 1404 and combines the hardware to complete the steps of the above method.

[0179] The processor 1401, the memory 1404 and the communication interface 1403 can communicate through the bus 1402.

[0180] In the above embodiments, the instructions stored in the memory for the processor to execute can be implemented in the form of a computer program product. Among them, the computer program product can be written in the memory in advance, or downloaded and installed in the memory in the form of software.

[0181] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. For example, the available media can include magnetic media (for example, floppy disk, hard disk or magnetic tape), optical media (for example, digital versatile disc (DVD)), or semiconductor media (for example, solid state disk (SSD)), etc.

[0182] The embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium stores computer programs or instructions. The computer programs or instructions are executed by a processor to implement the above method. The method described in the above embodiments can be wholly or partially implemented by software, hardware, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on or transmitted over a computer-readable medium. The computer-readable medium can include computer storage medium and communication medium, and can also include any medium that can transfer computer programs from one place to another. The storage medium can be any target medium accessible by a computer.

[0183] As a non-limiting example only, such computer-readable media can be in the form of persistent and / or nonpersistent memory, magnetic and / or optical storage disks, magnetic and / or optical storage cards, and the like. As another non-limiting example, computer-readable media can be in the form of software program product. As a possible design, the computer-readable medium can include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disk storage; magnetic disk storage or other magnetic storage devices. Also, any appropriate combination of the above can be appropriately deemed a computer-readable medium. In addition, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, DVD, floppy disk, and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0184] The above-described embodiments are merely intended to illustrate the technical solutions of the present application, but not to limit the same; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0185] In the above-described embodiments, according to the context, the term “when” can be interpreted as meaning “if” or “after” or “in response to determining” or “in response to detecting”. Similarly, according to the context, the phrase “upon determining” or “if detecting (a stated condition or event)” can be interpreted as meaning “if determining” or “in response to determining” or “upon detecting (a stated condition or event)” or “in response to detecting (a stated condition or event)”.

[0186] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like.

[0187] Those of ordinary skill in the art understand that all or part of the processes in the above embodiments can be implemented by a computer program to instruct the relevant hardware, which can be stored in a computer readable storage medium. The program can include the processes of the above method embodiments when executed. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, and the method includes: The electronic device extracts latent space features of multiple scales from the first latent space features and splices the latent space features of multiple scales to obtain a second latent space feature, wherein the resolution of the second latent space feature is higher than that of the first latent space feature. The electronic device splits the second latent space feature into multiple third latent space features, the resolution of which is lower than that of the second latent space feature; The electronic device decodes the plurality of third latent space features and the first latent space features in parallel, and then stitches the decoded plurality of third latent space features together to obtain the first image data. The electronic device uses the first image data and the fourth latent space feature to obtain the second image data. The fourth latent space feature is obtained by linearly stretching the decoded first latent space feature. The fourth latent space feature has the same resolution as the first image data. The fourth latent space feature is used to determine the first color compensation value. The first color compensation value is used to compensate the color of the first image data to obtain the second image data.

2. The method according to claim 1, characterized in that, One of the latent space features at multiple scales is obtained by fusing the first latent space feature with the result of one or more depthwise separable convolution operations, wherein the size of the convolution kernels in the multiple depthwise separable convolution operations are different from each other.

3. The method according to claim 2, characterized in that, After the electronic device obtains the second latent space feature, the method further includes: the electronic device performing multiple convolution operations on the second latent space feature, and fusing the results of the multiple convolution operations.

4. The method according to claim 1, characterized in that, The electronic device obtains second image data using the first image data and the fourth latent space features, including: The electronic device linearly stretches the first latent space feature to obtain the fourth latent space feature; The electronic device determines a first color compensation value based on the fourth hidden space feature; The electronic device compensates for the color of the first image data based on the first color compensation value to obtain the second image data.

5. The method according to claim 1, characterized in that, The first latent space feature is a latent space feature used to describe the image generated by the user's intent.

6. The method according to claim 5, characterized in that, The image generated by the user intent is determined based on the first text input by the user, and the latent space features of the image generated by the user intent are generated based on random noise and the first text.

7. The method according to claim 5, characterized in that, The image generated by the user intent is determined based on the first text and the first image input by the user, and the latent space features of the image generated by the user intent are generated based on the first image and the first text with added random noise.

8. The method according to claim 5, characterized in that, The first latent space feature is any one of the following: the latent space feature of the image input by the user, the latent space feature of the image stored by the electronic device, or the latent space feature of the image obtained by the electronic device from the network.

9. The method according to claim 8, characterized in that, After the electronic device obtains the second image data using the first image data and the fourth latent space features, the method further includes: The electronic device displays a first image based on the second image data, wherein the resolution of the first image is greater than the resolution of the image input by the user, the resolution of the first image is greater than the resolution of the image stored by the electronic device, and the resolution of the first image is greater than the resolution of the image obtained by the electronic device from the network.

10. The method according to any one of claims 1-9, characterized in that, The electronic device extracts latent space features at multiple scales from the first latent space features and concatenates these features using a multi-scale feature extraction model. This model converts low-resolution latent space features into high-resolution latent space features. The electronic device decodes the multiple third latent space features and the first latent space features in parallel using a decoding model, which converts the latent space features into image data. Before extracting the latent space features at the multiple scales, the method further includes: The electronic device trains the parameters of the multi-scale feature extraction model using a latent space domain loss function. The latent space domain loss function is used to describe the degree of deviation between the latent space features of the first sample image and the latent space features of the second sample image. The latent space features of the second sample image are high-resolution latent space features obtained by inputting the latent space features of the downsampled first sample image into the multi-scale feature extraction model. The electronic device trains the parameters of the multi-scale feature extraction model and the decoding model using an image domain loss function. The image domain loss function is used to describe the degree of deviation between the predicted image and the first sample image. The predicted image is an image obtained by inputting the latent space features of the second sample image into the decoding model.

11. The method according to claim 10, characterized in that, The latent space domain loss function includes one or more of the following: a first loss function and a second loss function, wherein the first loss function is used to calculate the deviation between the latent space features of the first sample image and the latent space features of the second sample image, and the second loss function is used to calculate the probability that the latent space features of the second sample image are estimated to be the latent space features of the first sample image.

12. The method according to claim 11, characterized in that, The image domain loss function includes one or more of the following: a third loss function, a fourth loss function, and a fifth loss function. The third loss function is used to calculate the deviation between the pixel values ​​of the first sample image and the pixel values ​​of the second sample image. The second loss function is used to calculate the perceptual similarity between the first sample image and the second sample image. The third loss function is used to calculate the deviation between the pixel values ​​of the blurred first sample image and the pixel values ​​of the blurred second sample image.

13. An electronic device, characterized in that, The electronic device includes: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display a user interface, the memory stores a computer program, and the processor executes the computer program to cause the electronic device to implement the method as described in any one of claims 1-12.

14. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed on the processor, they cause the method as described in any one of claims 1-12 to be performed.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-12.

Images