Image processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device and electronic equipment. The method comprises the steps that super-resolution processing is carried out on a first image through a non-diffusion model to obtain a second image, and the resolution of the second image is higher than that of the first image; super-resolution processing is carried out on the second image through semantic information of the first image and a diffusion model to obtain a third image, and the resolution of the third image is higher than that of the second image; and taking the third image as an output image. In this way, after the first image with the relatively low resolution is obtained, super-resolution processing can be carried out in sequence in combination with the non-expansion model and the diffusion model to obtain the finally output third image with the relatively high resolution, and therefore the quality of the image output after image super-resolution processing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and more particularly, to an image processing method and device and electronic equipment. BACKGROUND

[0002] Image resolution refers to the amount of details contained in an image, reflecting the ability of an imaging system to reflect the actual details of an object. Super-resolution technology originally refers to a technology that can improve the resolution of an optical imaging system, and now commonly refers to a method that can convert a single or multiple low-resolution images into a high-resolution image through processing. However, the related super-resolution technology still has the problem that the quality of the generated image needs to be improved. SUMMARY

[0003] In view of the above problems, the present application provides an image processing method, device and electronic equipment to improve the above problems.

[0004] In a first aspect, the present application provides an image processing method, which comprises: performing super-resolution processing on a first image through a non-diffusion model to obtain a second image, the resolution of the second image being higher than that of the first image; performing super-resolution processing on the second image through semantic information of the first image and a diffusion model to obtain a third image, the resolution of the third image being higher than that of the second image; and taking the third image as an output image.

[0005] In a second aspect, the present application provides an image processing device, which comprises: a first processing unit configured to perform super-resolution processing on a first image through a non-diffusion model to obtain a second image, the resolution of the second image being higher than that of the first image; a second processing unit configured to perform super-resolution processing on the second image through semantic information of the first image and a diffusion model to obtain a third image, the resolution of the third image being higher than that of the second image; and an image output unit configured to take the third image as an output image.

[0006] In a third aspect, the present application provides an electronic device, which comprises at least a processor and a memory, and one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0007] In a fourth aspect, the present application provides a computer readable storage medium, which stores a program code, wherein the program code is executed by a processor to perform the above method.

[0008] The image processing method, device and electronic equipment provided in the application, in the method, the first image is subjected to super-resolution processing by a non-diffusion model to obtain a second image, then the second image can be subjected to super-resolution processing by semantic information of the first image and a diffusion model to obtain a third image, and the third image is taken as an output image. Thus, by the above manner, after the first image with relatively low resolution is obtained, the non-diffusion model and the diffusion model can be combined to perform super-resolution processing in sequence to obtain the third image with relatively high resolution as the final output image, thereby improving the quality of the output image subjected to image super-resolution processing. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0010] Figure 1 A schematic diagram of an application scenario of the image processing method proposed in the embodiments of the application is shown;

[0011] Figure 2 A schematic diagram of another application scenario of the image processing method proposed in the embodiments of the application is shown;

[0012] Figure 3 A flowchart of an image processing method proposed in the embodiments of the application is shown;

[0013] Figure 4 A flowchart of an image processing method proposed in the embodiments of the application is shown;

[0014] Figure 5 A flowchart of an image processing method proposed in another embodiment of the application is shown;

[0015] Figure 6 A flowchart of an image processing method proposed in another embodiment of the application is shown;

[0016] Figure 7 A schematic diagram of a diffusion model in the embodiments of the application is shown;

[0017] Figure 8 A schematic diagram of another diffusion model in the embodiments of the application is shown;

[0018] Figure 9 A flowchart of an image processing method proposed in another embodiment of the application is shown;

[0019] Figure 10 a flow chart of an image processing method according to an embodiment of the present application is shown;

[0020] Figure 11 a structural block diagram of an image processing device according to an embodiment of the present application is shown;

[0021] Figure 12 a structural block diagram of an electronic device for performing the image processing method according to an embodiment of the present application is shown;

[0022] Figure 13 a storage unit for storing or carrying program codes for implementing the image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0024] Image resolution, as an important indicator of image quality, is directly related to the richness of details that can be displayed in the image. It is usually measured in pixels, and each pixel is a basic point in the image, carrying color, brightness and other information. High-resolution images mean that under the same display size, more pixels can be included, thus displaying more delicate and clear picture details such as texture and edges, which is crucial for medical imaging, satellite remote sensing, security monitoring and daily photography.

[0025] The purpose of super-resolution technology (SR) is to solve the problem that high-resolution images cannot be directly obtained under certain conditions (such as hardware limitations, poor shooting environment, etc.). This technology initially focuses on the field of optical imaging, aiming to optimize the physical design or algorithm processing of the imaging system to directly improve the ability of imaging devices to capture details. However, with the rapid development of computer vision and machine learning technology, the connotation and application range of super-resolution technology have been greatly expanded.

[0026] Modern super-resolution technology refers more to using advanced algorithms to process existing low-resolution images to generate high-resolution images that are visually close to or even exceed the resolution limits of the original device. Although super-resolution technology has achieved remarkable results in many applications, the quality of the generated images still needs to be further improved.

[0027] Therefore, the inventors, after finding the above problems in research, propose the image processing method, device and electronic equipment in the present application, which can improve the above problems. In the present method, the first image is super-resolution processed by a non-diffusion model to obtain a second image, and then the second image can be super-resolution processed by semantic information of the first image and a diffusion model to obtain a third image, and the third image is taken as the output image. In this way, after obtaining the first image with relatively low resolution, the non-diffusion model and the diffusion model are combined to perform super-resolution processing in sequence to obtain the third image with relatively high resolution as the final output, thereby improving the quality of the output image in the image super-resolution processing.

[0028] Before the embodiments of the present application are further described in detail, an application environment involved in the embodiments of the present application is introduced.

[0029] The application scenario involved in the embodiments of the present application is introduced as follows.

[0030] In the embodiments of the present application, the image processing method provided can be executed by an electronic device. In this way executed by the electronic device, all steps in the image processing method provided by the embodiments of the present application can be executed by the electronic device. For example, as shown in Figure 1 In the case where all steps in the image processing method provided by the embodiments of the present application can be executed by the electronic device, all steps can be executed by the processor of the electronic device 100.

[0031] Further, the image processing method provided by the embodiments of the present application can also be executed by a server. Correspondingly, in this way executed by the server, the server can start to execute the steps in the image processing method provided by the embodiments of the present application in response to a trigger instruction. The trigger instruction can be sent by the electronic device used by the user, or can be triggered locally by the server in response to some automatic events.

[0032] In addition, the image processing method provided by the embodiments of the present application can also be executed by the electronic device and the server in cooperation. In this way executed by the electronic device and the server in cooperation, part of the steps in the image processing method provided by the embodiments of the present application are executed by the electronic device, and the other part of the steps are executed by the server. For example, taking the image processing method in the present application as an example, as shown in Figure 2As shown, the electronic device 100 can perform an image processing method including: acquiring a first image. Then, the electronic device 100 can transmit the first image to a server 200, whereby the server 200 performs super-resolution processing on the first image using a non-diffusion model to obtain a second image, the second image having a higher resolution than the first image; using the semantic information of the first image and a diffusion model, the server performs super-resolution processing on the second image to obtain a third image. Afterwards, the server 200 can either save the third image locally or return it to the electronic device 100.

[0033] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.

[0034] It should be noted that the electronic equipment 100, in addition to being for Figure 1 and Figure 2 Besides smartphones, the device can also be a tablet, smart wearable device, smart voice assistant, etc. Server 200 can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. In the case where the image processing method provided in this embodiment is executed by a server cluster or distributed system consisting of multiple physical servers, different steps in the image processing method can be executed by different physical servers, or can be executed in a distributed manner by servers built on a distributed system.

[0035] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0036] Please see Figure 3 This application provides an image processing method, the method comprising:

[0037] S110: Perform super-resolution processing on the first image using a non-diffusion model to obtain a second image, wherein the resolution of the second image is higher than that of the first image.

[0038] In this embodiment, the first image can be understood as the image to be upgraded in resolution. Alternatively, the first image can also be understood as an image with a relatively low resolution.

[0039] In this embodiment, the first image can be acquired in various ways. Optionally, the first image can be an image captured by a camera of an electronic device. Optionally, the first image can be an image obtained from a network. Optionally, the first image can be an image transmitted from another electronic device.

[0040] After obtaining the first image, a non-diffusion model can be used to perform super-resolution processing on the first image to obtain the second image.

[0041] Diffusion models are a class of deep generative models inspired by nonequilibrium thermodynamics. In computer vision, these models first define a Markov chain of a forward diffusion process, gradually adding random noise to the data to make the data distribution gradually approach a prior distribution (such as a Gaussian distribution). Then, the model learns a backward diffusion process to recover the original data by progressively removing noise, thereby generating the desired samples. In this embodiment, if the diffusion model is considered as a process of generating data by progressively adding and removing noise, then a "non-diffusion model" may refer to a model that generates data without relying on this noise diffusion and removal mechanism. For example, the non-diffusion model in this embodiment could be a Generative Adversarial Network (GAN). Alternatively, the non-diffusion model in this embodiment could be a lightweight super-resolution model based on GANs.

[0042] The generative adversarial network, which serves as a non-diffusion model in the embodiments of this application, will be described below.

[0043] A GAN consists of two main parts: a generator and a discriminator. The generator's task is to generate fake images that are as close as possible to real high-resolution images, while the discriminator's task is to distinguish between the generated images and real high-resolution images. Through adversarial training, the generator can be specifically used for generating super-resolution images. The discriminator also uses a convolutional neural network to distinguish between the generated high-resolution images and real high-resolution images. The discriminator should be designed to capture details and texture differences in the image to better guide the generator's training.

[0044] During training, a noise vector can be randomly generated as input to the generator, which can then use this noise vector to generate a high-resolution image. The discriminator receives both the generated and real high-resolution images and attempts to distinguish between them. Based on the discriminator's feedback, the parameters of both the generator and the discriminator are updated to optimize the loss function. This process is repeated until the generator can produce sufficiently realistic high-resolution images that the discriminator struggles to distinguish between them.

[0045] S120: Using the semantic information of the first image and the diffusion model, super-resolution processing is performed on the second image to obtain a third image, the resolution of which is higher than that of the second image.

[0046] In this context, speech information can represent the semantic information of the content included in the first image. Optionally, image tagging can be used to extract features from the first image to output semantic labels contained in the first image, which can then be used as the semantic information of the first image. Image tagging is an image content recognition and description technology based on deep learning. It automatically labels or tags elements such as objects, scenes, and emotions in images to facilitate image management, classification, search, and retrieval.

[0047] After obtaining the semantic information of the first image, the semantic information and the second image can be input into the diffusion model for super-resolution processing to obtain the third image output by the extended model.

[0048] As mentioned earlier, the diffusion model is a generative model that gradually adds Gaussian noise to corrupt the training data, then trains a neural network to reverse this corruption process, thereby recovering the original data from the noise. In image super-resolution tasks, the diffusion model can be used to progressively remove noise from low-resolution images to generate high-resolution images. Compared with traditional super-resolution methods, the diffusion model has stronger generative capabilities and better generalization performance. Furthermore, in image super-resolution tasks, incorporating the semantic information of the image into the diffusion model for super-resolution processing not only improves the image resolution but also maintains the semantic consistency of the image content, making the generated image (the third image) more realistic and clearer.

[0049] Specifically, in the diffusion model, super-resolution processing can be accomplished through a series of iterative steps. In each step, the diffusion model attempts to remove a portion of the noise from the image (the image obtained from the previous noise removal) and gradually recovers high-resolution details. Guided by semantic information, this process can more accurately recover key features in the image, resulting in a more realistic and clearer image. Finally, when all noise has been removed, the model outputs a high-resolution image (e.g., the third image) that not only has a higher pixel density but also maintains consistency in content with the original image (either the first or second image).

[0050] S130: The third image is output as the image.

[0051] The third image as output can be understood as the image obtained after the final super-resolution processing. Outputting the third image can be understood as storing it, or transmitting it to other devices.

[0052] Next, we will proceed through...Figure 4 The processing flow involved in this embodiment will be described below.

[0053] like Figure 4 As shown, for low-resolution images (i.e., the first image in this embodiment), they can first be input into a non-diffusion model to obtain super-resolution image 1 (i.e., the second image) output by the non-diffusion model. Alternatively, the low-resolution image can be input into a lightweight text parsing module to obtain text tags (i.e., semantic information) output by the lightweight text analysis module. Then, super-resolution image 1 and text tags are input together into an extended model to obtain super-resolution image 2 (i.e., the third image).

[0054] This embodiment provides an image processing method in which a first image is super-resolution processed using a non-diffusion model to obtain a second image. Then, the second image is further super-resolution processed using the semantic information of the first image and a diffusion model to obtain a third image, which is then used as the output image. This method allows for the improvement of image quality by combining a non-diffusion model and a diffusion model sequentially after obtaining a first image with relatively low resolution, resulting in a higher-resolution third image.

[0055] Please see Figure 5 This application provides an image processing method, the method comprising:

[0056] S210: Perform super-resolution processing on the first image using a non-diffusion model to obtain a second image, wherein the resolution of the second image is higher than that of the first image.

[0057] S220: Detect whether the user is satisfied with the second image.

[0058] One approach is to display the second image after obtaining it. While displaying the second image, a prompt asking the user whether they are satisfied, along with a first selection control and a second selection control, can be simultaneously displayed. The first selection control indicates user satisfaction with the second image, while the second selection control indicates user dissatisfaction. This allows the user to determine their satisfaction with the displayed second image using both the first and second selection controls. For example, if the user touches the first selection control, it indicates satisfaction with the second image; if the user touches the second selection control, it indicates dissatisfaction.

[0059] S230: If the user is not satisfied with the second image, the second image is super-resolution processed using the semantic information of the first image and the diffusion model to obtain a third image, the resolution of which is higher than that of the second image.

[0060] S231: The third image is used as the output image.

[0061] S240: If the second image is satisfactory, the second image is used as the output image.

[0062] This embodiment provides an image processing method that, after obtaining a first image with relatively low resolution, combines a non-diffusion model and a diffusion model sequentially for super-resolution processing to obtain a third image with relatively higher resolution, thereby improving the quality of the image output from the super-resolution processing. Furthermore, in this embodiment, after obtaining the second image from the non-diffusion model, it can first detect whether the user is satisfied with the second image. If the user is satisfied, the second image can be directly used as the final output image, thus enabling faster acquisition of the higher-resolution image required by the user while simultaneously performing super-resolution processing on the first image. If the user is not satisfied with the second image, the semantic information of the first image and the diffusion model can be further combined to obtain a third image that satisfies the user. This method allows for a better balance between efficiency and quality in the process of obtaining the super-resolution image.

[0063] Please see Figure 6 This application provides an image processing method, the method comprising:

[0064] S310: Perform super-resolution processing on the first image using a non-diffusion model to obtain a second image, wherein the resolution of the second image is higher than that of the first image.

[0065] S320: Obtain the corresponding first image feature vector through the image encoder and the second image.

[0066] In one embodiment of this application, the diffusion model includes an image encoder, a text encoder, a cross-attention module, and an image generation module.

[0067] The image encoder is used to extract corresponding image features from the input image. For example, it can be used to extract features from a second input image to obtain the corresponding first image feature vector.

[0068] Optionally, the diffusion model may also include a Linear layer and a LayerNorm layer.

[0069] In the diffusion model, the Linear layer is primarily responsible for performing linear transformations. These transformations produce outputs by applying a weight matrix and bias terms to the input data. Specifically, the roles of the Linear layer can be summarized as follows: Feature extraction and transformation: Through linear transformations, the Linear layer extracts useful features from the input data and transforms these features into a format that subsequent layers of the model can process. Model parameter learning: During training, the weights and bias terms of the Linear layer are learnable parameters. Through backpropagation, these parameters are adjusted according to the model's loss function to optimize model performance. Increasing model complexity: By stacking multiple Linear layers, more complex neural network models can be built, thereby improving the model's ability to model complex data.

[0070] LayerNorm is a normalization technique used in deep neural networks. It primarily controls the distribution of activation values ​​in hidden layers to improve model training stability and performance. In diffusion models, the role of LayerNorm can be summarized as follows: Stabilizing the training process: By normalizing the activation values ​​of hidden layers, LayerNorm reduces internal covariate shift, thus stabilizing the model's training process. This helps the model converge to a better solution faster during training. Accelerating convergence: Normalization can accelerate the model's convergence speed, allowing it to achieve better performance in fewer iterations. Improving model generalization ability: By controlling the distribution of hidden layer activation values, LayerNorm can also improve the model's generalization ability, enabling it to perform well on unseen data.

[0071] In this case, the feature vector output by the image encoder can be processed sequentially through the Linear layer and the LayerNorm layer, and then the feature vector output by the LayerNorm layer can be used as the first image feature vector.

[0072] S330: Obtain the corresponding text feature vector through the text encoder and the semantic information.

[0073] The text encoder is used to extract features from the input language information and convert them into corresponding text feature vectors.

[0074] S340: Input the first image feature vector and the text feature vector into the cross-attention module to obtain the attention feature vector output by the cross-attention module.

[0075] The Decoupled Cross-Attention module is a deep learning architecture specifically designed to process data from different modalities, such as images and text. In this module, feature vectors from different modalities (in this case, the feature vectors of the first image and the text feature vectors of the second image) are processed together to produce an output that combines information from both modalities.

[0076] In one approach, the cross-attention module includes a first attention submodule (Cross-Attention) and a second attention submodule (Cross-Attention). In this approach, inputting a first image feature vector and a text feature vector into the cross-attention module to obtain the attention feature vector output by the cross-attention module may include: inputting the first image feature vector into the first attention submodule to obtain the first attention feature vector output by the first attention submodule; and inputting the text feature vector into the second attention submodule to obtain the second attention feature vector output by the second attention submodule, so that the first attention feature vector and the second attention feature vector are used as the attention feature vector output by the cross-attention module.

[0077] As one approach, before inputting the first image feature vector and the text feature vector into the cross-attention module to obtain the attention feature vector output by the cross-attention module, the method may further include: adding positional encoding information (Position_Embedding) corresponding to the second image to the first image feature vector to obtain the second image feature vector. In this approach, the second image feature vector and the text feature vector can be input into the cross-attention module to obtain the attention feature vector output by the cross-attention module. The positional encoding information can characterize the relative positional relationships between various image elements in the second image.

[0078] S350: Super-resolution processing is performed using the image generation module and the attention feature vector to obtain a third image.

[0079] Optionally, the image generation module can be a UNet model. During single-step generation, the UNet utilizes the latest LoRA acceleration modules such as HyperSD or PCM to achieve single-step generation. To further adapt to specific super-resolution tasks, sub-modules such as ControlNet or StyleLora can be added to reduce the quality loss in super-resolution generation caused by acceleration.

[0080] When using UNet as the image generation module, the training process for the diffusion model can be as follows:

[0081] (1) First, fix UNet and do not introduce the acceleration module required for single-step generation. Only train the image encoder and position encoding information (Position_Embedding) so that the resulting image generated by the text-to-image task can be better aligned with the low-resolution image (e.g., the second input image) in terms of structure.

[0082] (2) Next, fix the image encoder, position encoding information, UNet and acceleration module, and train the super-resolution task module ControlNet or StyleLora so that the pre-trained text-to-image task can be better aligned with the super-resolution task, while maintaining the acceleration effect of the pre-trained acceleration module.

[0083] (3) Finally, fix UNet and further train the image encoder, position encoding information and ControlNet or StyleLora, and perform end-to-end tuning.

[0084] For example, the structure of the diffusion model involved in this embodiment can be as follows: Figure 7 As shown, the diffusion model primarily relies on an image encoder and an IP-Adapter. The image encoder and IP-Adapter are kept structurally unchanged; however, the learning ability of the cross-attention module is enhanced by adding positional encoding information corresponding to the image representation. For example... Figure 8 As shown, for the first feature vector output by the normalization layer, positional encoding information can be further added to obtain the second feature vector.

[0085] This embodiment provides an image processing method in which a first image is super-resolution processed using a non-diffusion model to obtain a second image. Then, the second image is further super-resolution processed using the semantic information of the first image and a diffusion model to obtain a third image, which is then used as the output image. This method allows for the generation of a higher-resolution third image by combining a non-diffusion model and a diffusion model sequentially after obtaining a relatively low-resolution first image, thereby improving the quality of the super-resolution output image. Furthermore, this embodiment introduces positional encoding information corresponding to the second image, enabling the generation of the third image based on the feature vector of the second image with this added positional encoding information. This results in a third image with no significant structural distortion and good semantic consistency.

[0086] Please see Figure 8 This application provides an image processing method, the method comprising:

[0087] S410: Perform super-resolution processing on the first image using a non-diffusion model to obtain a second image, wherein the resolution of the second image is higher than that of the first image.

[0088] S420: Using the semantic information of the first image and the diffusion model, super-resolution processing is performed on the second image to obtain a third image, wherein the resolution of the third image is higher than that of the second image.

[0089] S430: The third image and the second image are fused to obtain a fourth image.

[0090] S440: The fourth image is output as the image.

[0091] For example, such as Figure 9 As shown. After obtaining super-resolution image 1 (second image) and super-resolution image 2 (third image), super-resolution image 1 and super-resolution image 2 can be fused to obtain super-resolution image 3.

[0092] This embodiment provides an image processing method that, through the above-described manner, after obtaining a first image with a relatively low resolution, can combine a non-expanded model and a diffusion model to perform super-resolution processing sequentially to obtain a third image with a relatively higher resolution. Furthermore, the third image and the second image can be further fused to obtain a fourth image, which is then used as the output image, thereby improving the quality of the image output by the super-resolution processing.

[0093] As one approach, before performing super-resolution processing on the first image using a non-diffusion model to obtain the second image, the method may further include: detecting the image scene complexity of the first image; if the image scene complexity is lower than a complexity threshold, then performing super-resolution processing on the first image using a non-diffusion model to obtain the second image. If the image scene complexity is not lower than the complexity threshold, then the first image can be directly input into a diffusion model, and the fifth image output by the diffusion model can be used as the final output image.

[0094] It should be noted that in some cases, the diffusion model can outperform the non-diffusion model in terms of image generation capability. Image generation capability can be understood as the degree of resolution improvement in the image. The greater the improvement in image resolution, the better the quality of the resulting image.

[0095] However, while diffusion models offer superior generation capabilities, they may take longer to process the same image. In such cases, for images with relatively simple scenes, a non-diffusion model can be used first. This leverages the shorter processing time of the non-diffusion model, allowing for faster display of super-resolution results to the user. Furthermore, because the scene is simpler, the processing power requirements for the model are lower, meaning the non-diffusion model's results may, to some extent, meet the user's needs. If the super-resolution result obtained using the non-diffusion model (e.g., the second image) is unsatisfactory, a diffusion model can be used to further super-resolution the super-resolution result output by the non-diffusion model. For images with relatively complex scenes, the non-diffusion model and then the diffusion model can be used sequentially, with the diffusion model's output then displayed to the user.

[0096] In the embodiments of this application, there are multiple ways to determine the image scene complexity of an image.

[0097] One method to determine the complexity of an image scene is by calculating image entropy. Image entropy is an indicator of the information content of an image, reflecting its complexity. Generally, the higher the entropy value, the more information the image contains, and the higher its complexity. To calculate image entropy, the probability distribution of each pixel or gray level can be calculated, and this information can then be used to calculate the entropy value.

[0098] As another approach, edge detection algorithms (such as Canny and Sobel) can be used to detect edges in an image. Images with a large number and variety of edges (such as straight lines and curves) are generally considered complex. The complexity can be quantified by statistically analyzing the total length, number, or density of edges.

[0099] As another approach, image complexity can be analyzed using texture features. Methods such as Gray-Level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP) can be used to extract texture features from an image. The richness and complexity of textures are often proportional to the overall complexity of the image.

[0100] Please see Figure 10 This application provides an image processing apparatus 500, which includes:

[0101] The first processing unit 510 is used to perform super-resolution processing on the first image using a non-diffusion model to obtain a second image, wherein the resolution of the second image is higher than that of the first image.

[0102] The second processing unit 520 is used to perform super-resolution processing on the second image using the semantic information of the first image and a diffusion model to obtain a third image, wherein the resolution of the third image is higher than that of the second image.

[0103] Image output unit 530 is used to output the third image as an image.

[0104] In one manner, the second processing unit 520 is specifically used to detect whether the user is satisfied with the second image; if the user is not satisfied with the second image, the second image is subjected to super-resolution processing using the semantic information of the first image and the diffusion model to obtain a third image; if the user is satisfied with the second image, the second image is used as the output image.

[0105] In one approach, the diffusion model includes an image encoder, a text encoder, a cross-attention module, and an image generation module. In this approach, the second processing unit 520 is specifically configured to: obtain a corresponding first image feature vector using the image encoder and the second image; obtain a corresponding text feature vector using the text encoder and the semantic information; input the first image feature vector and the text feature vector into the cross-attention module to obtain an attention feature vector output by the cross-attention module; and perform super-resolution processing using the image generation module and the attention feature vector to obtain a third image.

[0106] Optionally, the second processing unit 520 is specifically used to add positional encoding information corresponding to the second image to the first image feature vector to obtain the second image feature vector. In this way, the second processing unit 520 is specifically used to input the second image feature vector and the text feature vector into the cross-attention module to obtain the attention feature vector output by the cross-attention module.

[0107] In one embodiment, the cross-attention module includes a first attention submodule and a second attention submodule. The second processing unit 520 is specifically configured to input the first image feature vector into the first attention submodule to obtain a first attention feature vector output by the first attention submodule; and to input the text feature vector into the second attention submodule to obtain a second attention feature vector output by the second attention submodule, so that the first attention feature vector and the second attention feature vector are used as the attention feature vector output by the cross-attention module.

[0108] In one embodiment, the image output unit 530 is specifically used to fuse the third image and the second image to obtain a fourth image; and to output the fourth image as the output image.

[0109] In one approach, the first processing unit 510 is specifically used to detect the image scene complexity of the first image. If the image scene complexity is lower than a complexity threshold, super-resolution processing of the first image is performed using a non-diffusion model to obtain a second image.

[0110] This embodiment provides an image processing apparatus that, after obtaining a first image with a relatively low resolution, combines a non-expanded model and a diffusion model to perform super-resolution processing sequentially, thereby obtaining a third image with a relatively higher resolution, thus improving the quality of the image output by the super-resolution processing.

[0111] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0112] The following will combine Figure 11 This application describes an electronic device.

[0113] Please see Figure 11 Based on the aforementioned image processing methods and apparatus, this application also provides another electronic device 2000 capable of executing the aforementioned image processing methods. The electronic device 2000 includes one or more (only one shown in the figure) processors 202, a memory 204, a network module 206, a sensor module 208, and an audio acquisition device 210, all coupled to each other. The memory 204 stores programs capable of executing the contents of the aforementioned embodiments, and the processor 202 can execute the programs stored in the memory 204.

[0114] The processor 202 may include one or more processing cores. The processor 202 connects to various parts of the electronic device 2000 via various interfaces and lines, and performs various functions and processes data of the electronic device 2000 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 204, and by calling data stored in the memory 204. Optionally, the processor 202 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 202 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 202 and may be implemented separately using a communication chip.

[0115] The memory 204 may include random access memory (RAM) or read-only memory (ROM). The memory 204 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below.

[0116] The network module 206 is used to enable information interaction between the electronic device 2000 and other devices, such as transmitting device control commands, manipulation request commands, and status information acquisition commands. However, the network module 206 may differ depending on the specific device being used.

[0117] Sensor module 208 may include at least one sensor. Specifically, sensor module 208 may include, but is not limited to, a light sensor, a motion sensor, a pressure sensor, an infrared thermal sensor, a distance sensor, an acceleration sensor, and other sensors.

[0118] The pressure sensor is a sensor that detects pressure generated by pressing on the electronic device 2000. That is, the pressure sensor detects pressure generated by contact or pressing between the user and the electronic device, such as the pressure generated by contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether contact or pressing has occurred between the user and the electronic device 2000, and the magnitude of the pressure.

[0119] The accelerometer can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. It can be used for applications such as identifying the posture of electronic devices 2000 (e.g., screen orientation switching, related games, magnetometer posture calibration), and vibration recognition functions (e.g., pedometer, tapping). In addition, electronic devices 2000 can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, and thermometers, which will not be elaborated upon here.

[0120] Audio acquisition device 210 is used for acquiring audio signals. Optionally, audio acquisition device 210 includes multiple audio acquisition devices. These audio acquisition devices can be microphones. For example, in one embodiment, audio acquisition device 210 may include two microphones, in which one microphone may correspond to one analog-to-digital converter (ADC), and the other microphone may correspond to two ADCs with different analog gains. In another embodiment, audio acquisition device 210 may include three microphones. In this embodiment, two microphones (e.g., a main microphone and a secondary microphone) may each correspond to one ADC, and the other microphone (e.g., a camera microphone) may correspond to two ADCs with different analog gains.

[0121] In one approach, the network module of the electronic device 2000 is a radio frequency (RF) module. This RF module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with a communication network or other devices. The RF module may include various existing circuit elements used to perform these functions, such as antennas, RF transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. For example, the RF module can interact with external devices by transmitting or receiving electromagnetic waves, thereby receiving audio signals transmitted by external devices.

[0122] Furthermore, the electronic device 2000 may also include an image acquisition device for image acquisition. For example, the image acquisition device can be used to capture video, still images, or moving images.

[0123] Please refer to Figure 12This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0124] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0125] In summary, the image processing method, apparatus, and electronic device provided in this application involve performing super-resolution processing on a first image using a non-diffusion model to obtain a second image. Then, the second image can be further super-resolution processed using the semantic information of the first image and a diffusion model to obtain a third image, which is then used as the output image. This approach allows for the improvement of image quality by combining a non-diffusion model and a diffusion model sequentially after obtaining a first image with relatively low resolution, resulting in a higher-resolution third image.

[0126] In this embodiment, the fusion of non-diffusion generation and diffusion generation results in a super-resolution image without significant structural distortion and possessing good semantic consistency. This image super-resolution method, which integrates non-diffusion and diffusion generation, can return the super-resolution image to the user in stages, comprehensively considering both super-resolution quality and runtime latency, thus providing a better user experience.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises: performing super-resolution processing on the first image by a non-diffusion model to obtain a second image, the resolution of the second image being higher than that of the first image; performing super-resolution processing on the second image by a diffusion model and semantic information of the first image to obtain a third image, the resolution of the third image being higher than that of the second image; outputting the third image as an output image.

2. The method of claim 1, wherein, The method further comprises, before performing super-resolution processing on the second image by a diffusion model and semantic information of the first image to obtain a third image: detecting whether a user is satisfied with the second image; if the user is not satisfied with the second image, performing super-resolution processing on the second image by a diffusion model and semantic information of the first image to obtain a third image; if the user is satisfied with the second image, outputting the second image as an output image.

3. The method of claim 1, wherein, The diffusion model comprises an image encoder, a text encoder, a cross-attention module, and an image generation module; and the method further comprises: obtaining a corresponding first image feature vector from the image encoder and the second image; obtaining a corresponding text feature vector from the text encoder and the semantic information; inputting the first image feature vector and the text feature vector into the cross-attention module to obtain an attention feature vector output by the cross-attention module; and performing super-resolution processing on the attention feature vector by the image generation module to obtain a third image.

4. The method of claim 3, wherein, The method further comprises, before inputting the first image feature vector and the text feature vector into the cross-attention module to obtain an attention feature vector output by the cross-attention module: adding position encoding information corresponding to the second image to the first image feature vector to obtain a second image feature vector; and inputting the second image feature vector and the text feature vector into the cross-attention module to obtain an attention feature vector output by the cross-attention module. The cross-attention module comprises a first attention sub-module and a second attention sub-module.

5. The method of claim 3, wherein, The method further comprises, before inputting the first image feature vector and the text feature vector into the cross-attention module to obtain an attention feature vector output by the cross-attention module: inputting the first image feature vector into the first attention sub-module to obtain a first attention feature vector output by the first attention sub-module; inputting the text feature vector into the second attention sub-module to obtain a second attention feature vector output by the second attention sub-module, so that the first attention feature vector and the second attention feature vector are taken as an attention feature vector output by the cross-attention module. ​ 6. The method of claim 1, wherein, The non-diffusion model comprises a GAN-based lightweight super-resolution model.

7. The method of claim 1, wherein, The third image is outputted as an output image. The third image and the second image are fused to obtain a fourth image. The fourth image is outputted as an output image.

8. The method of claim 1, wherein, Before the first image is processed by the non-diffusion model to obtain the second image, the method further comprises: detecting a complexity of a scene of the first image; if the complexity of the scene is lower than a complexity threshold, processing the first image by the non-diffusion model to obtain the second image.

9. An image processing apparatus characterized by comprising: The apparatus comprises: a first processing unit configured to process the first image by the non-diffusion model to obtain the second image, the resolution of the second image being higher than that of the first image; a second processing unit configured to process the second image by a diffusion model and semantic information of the first image to obtain the third image, the resolution of the third image being higher than that of the second image; an image output unit configured to output the third image as an output image.

10. An electronic device, comprising: The apparatus comprises a processor and a memory, one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, wherein the program codes are executed by a processor to implement the method of any one of claims 1-8.

Images