Image processing method and electronic device

Abstract

The present disclosure provides an image processing method and an electronic device, which relates to the field of terminal technology and can generate a corresponding image quality enhanced image according to an image quality enhancement requirement arbitrarily input by a user. The method provided by the present disclosure includes: in response to an image quality enhancement trigger operation, according to the image quality enhancement requirement text feature, the first image feature, the second image feature, the text-guided enhancement model and the diffusion model, a first round of fused text image bimodal features are obtained; according to the first round of fused text image bimodal features, the image quality enhancement requirement text feature, the first image feature, the text-guided enhancement model and the diffusion model, N-1 iterations are performed to obtain the first enhanced image coding feature; the text-guided enhancement model has the ability to generate the first enhanced image coding feature by fusing the image quality enhancement requirement text feature, the first image feature and the jth round of diffusion image feature set in each iteration process; the first enhanced image coding feature is decoded to obtain the second image.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Art

[0002] In today's highly digital age, users use electronic devices (such as mobile phones) to take photos or obtain images through the Internet, which has become a part of daily life. However, in the actual acquisition process, due to the influence of various factors, the image quality may be poor. For example, shaking during shooting will cause blurred images; raindrops will appear in the image when shooting on rainy days; images obtained from the Internet often have too low pixels, etc.

[0003] In view of the above-mentioned poor image quality, the relevant technology uses specific functions to improve specific image quality effects. For example, the deblurring function is used to improve blurry images; the pixel enhancement function is used to increase the pixels of images, etc. In the process of using specific functions to improve specific image quality effects, users are usually required to perform a lot of repetitions. This is not only cumbersome, but may also not fully meet user needs. Therefore, how to provide users with a more convenient and flexible image processing solution has become a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present disclosure provide an image processing method and an electronic device, which can generate a corresponding image with enhanced image quality according to any image quality enhancement requirement input by a user, thereby improving the user's usage experience.

[0005] To achieve the above objectives, the embodiments of the present disclosure adopt the following technical solutions:

[0006] In a first aspect, the present disclosure provides an image processing method, the method comprising: receiving a quality enhancement trigger operation for a first image; the quality enhancement trigger operation comprises a quality enhancement requirement text; in response to the quality enhancement trigger operation, obtaining a first round of fused text-image bimodal features according to the quality enhancement requirement text features, the first image features, the second image features, a text-guided enhancement model and a diffusion model; wherein the quality enhancement requirement text features are obtained by encoding the quality enhancement requirement text; the first image features are obtained by encoding the first image; the second image features are obtained by adding noise to the first image and then performing feature extraction on the noise-added result; according to the first round of fused text-image bimodal features, the quality enhancement requirement text features, the first image features, the text-guided enhancement model and the diffusion model The enhancement model and the diffusion model are iterated N-1 times to obtain the first enhanced image coding features; wherein the diffusion model has the ability to perform denoising on the second image features, and extract features from the denoising results to generate a first round of diffusion image feature set; and the ability to perform j-1 iterations according to the first round of fused text image bimodal features to generate the j-th round of diffusion image feature set; the text-guided enhancement model has the ability to fuse the image quality enhancement requirement text features, the first image features and the j-th round of diffusion image feature set in each iteration process to generate the first enhanced image coding features; j is a positive integer, 2≤j≤N; N is a positive integer; the first enhanced image coding features are decoded to obtain a second image; the second image is an image obtained by making the first image have a stronger image quality according to the image quality enhancement requirement text.

[0007] Based on the image processing method of the first aspect, after the electronic device receives the user's image quality enhancement trigger operation, the diffusion model and the text-guided enhancement model can be used to perform multiple iterations to generate a second image with enhanced image quality. In each iteration, when the diffusion model generates a diffusion image feature set (i.e., the jth round of diffusion image feature set), the text-guided enhancement model can fuse the diffusion image feature set into the image quality enhancement requirement text feature and the first image feature. In this way, in each iteration, the image quality enhancement requirement text can be referenced to generate the second image. Thereby, the final second image meets the user's image quality enhancement requirements, and the implementation process is convenient and simple, and the user only needs to perform the image quality enhancement trigger operation.

[0008] In combination with the first aspect, in another possible implementation, according to the image quality enhancement requirement text feature, the first image feature, the second image feature, the text-guided enhancement model and the diffusion model, a first round of fused text-image bimodal features is obtained, including: using a first processing module of the text-guided enhancement model to process the image quality enhancement requirement text feature and the first image feature to obtain the text-image bimodal features corresponding to the first processing module in the first round; using the diffusion model to denoise the second image feature, and extracting features from the denoising result to generate a first round of diffusion image feature set; the first round of diffusion image feature set includes M features; expanding The diffusion model includes M processing modules; each processing module in the diffusion model corresponds to a feature; M is a positive integer; the text image bimodal feature corresponding to the first processing module in the first round and the first diffusion image feature are fused to obtain the fused text image bimodal feature corresponding to the first processing module in the first round; the first diffusion image feature is the feature output by the first processing module of the diffusion model; according to the fused text image bimodal feature corresponding to the first processing module in the first round, other features in the first round diffusion image feature set except the first diffusion image feature and the text-guided enhancement model, the first round fused text image bimodal feature is obtained.

[0009] Based on this solution, by fusing the bimodal features of text and image with the features output by the diffusion model, the first image quality enhancement process not only considers the features of the first image itself, but also combines the features of the user's text quality enhancement requirements. This fusion method can make the image quality enhancement more targeted, avoiding the blindness that may occur when relying solely on image features for processing, thereby significantly improving the image quality enhancement effect.

[0010] In combination with the first aspect, in another possible implementation method, N-1 iterations are performed based on the first round of fused text-image bimodal features, the text features required for image quality enhancement, the first image features, the text-guided enhancement model and the diffusion model to obtain the first enhanced image coding features, including: in the j-th round of iterations, the j-th round of fused text-image bimodal features and the diffusion model are determined, wherein j is a positive integer, 2≤j≤N; the j-th round of fused text-image bimodal features are obtained based on the j-th round of diffusion image feature set, the text features required for image quality enhancement, the first image features and the text-guided enhancement model; wherein, when j=N, the j-th round of fused text-image bimodal features are the first enhanced image coding features.

[0011] Based on this scheme, in the iterative process, each round is further processed based on the fused text-image bimodal features of the previous round. As the number of iterations increases, the analysis and adjustment of image features will become more refined, and the defects in the image can be gradually removed, thus enhancing the image quality.

[0012] In combination with the first aspect, in another possible implementation method, according to the j-th round diffusion image feature set, the text feature required for image quality enhancement, the first image feature and the text-guided enhancement model, the j-th round fused text image bimodal feature is obtained, including: using the first processing module of the text-guided enhancement model to process the text feature required for image quality enhancement and the first image feature to obtain the text image bimodal feature corresponding to the first processing module in the j-th round; fusing the feature output by the first processing module of the diffusion model in the j-th round diffusion image feature set and the text image bimodal feature corresponding to the first processing module in the j-th round to obtain the fused text image bimodal feature corresponding to the first processing module in the j-th round; according to the fused text image bimodal feature corresponding to the first processing module in the j-th round, the features in the j-th round diffusion image feature set except the features output by the first processing module of the diffusion model, and the text-guided enhancement model, the j-th round fused text image bimodal feature is obtained.

[0013] Based on this solution, the first processing module of the text-guided enhancement model is used to process the text features required for image quality enhancement and the first image features, so that information of two different modalities, text and image, can be integrated at an early stage. Through multiple rounds of subsequent iterations, the user's image quality enhancement requirements and the characteristics of the image itself can be fully referenced to perform image quality enhancement in a more targeted manner.

[0014] In combination with the first aspect, in another possible implementation, the j-th round fused text image bimodal features are obtained according to the fused text image bimodal features corresponding to the first processing module in the j-th round, the features in the j-th round diffusion image feature set except the features output by the first processing module of the diffusion model, and the text-guided enhancement model, including: inputting the fused text image bimodal features corresponding to the i-1-th processing module in the j-th round into the i-th processing module of the text-guided enhancement model to obtain the text image bimodal features corresponding to the i-th processing module in the j-th round; wherein the text-guided enhancement model includes M processing modules, i is a positive integer, 2≤i≤M; fusing the features output by the i-th processing module of the diffusion model in the j-th round diffusion image feature set and the text image bimodal features corresponding to the i-th processing module in the j-th round to obtain the fused text image bimodal features corresponding to the i-th processing module in the j-th round; when i=M, inputting the fused text image bimodal features corresponding to the M-th processing module in the j-th round into the output module of the text-guided enhancement model to obtain the j-th round fused text image bimodal features. Based on this scheme, an exemplary iterative process is provided.

[0015] In combination with the first aspect, in another possible implementation method, the picture quality enhancement trigger operation includes picture quality enhancement requirement audio; before responding to the picture quality enhancement trigger operation, the method also includes: when the picture quality enhancement trigger operation includes picture quality enhancement requirement audio, the picture quality enhancement requirement audio is identified to obtain the picture quality enhancement requirement text.

[0016] Based on this solution, the diversity of the image quality enhancement triggering operation can be increased. Users can not only express the image quality enhancement requirements through text input, but also use voice input. In this way, the convenience and efficiency of user operation are improved.

[0017] In combination with the first aspect, in another possible implementation, the diffusion model is a pre-trained implicit diffusion model, and the text-guided enhancement model is a u-net model. An example of a diffusion model and a text-guided enhancement model is provided.

[0018] In combination with the first aspect, in another possible implementation, the text requiring a higher image quality is to reduce noise, reduce blur and improve resolution. An example of a text requiring a higher image quality is provided.

[0019] In a second aspect, an embodiment of the present disclosure provides an image processing device, which can be applied to an electronic device to implement the method in the first aspect. The functions of the image processing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a receiving module and a processing module.

[0020] A receiving module is configured to receive a picture quality enhancement triggering operation for a first image; the picture quality enhancement triggering operation includes a picture quality enhancement requirement text;

[0021] The processing module is configured to respond to the image quality enhancement trigger operation, and obtain the first round of fused text image bimodal features according to the image quality enhancement requirement text features, the first image features, the second image features, the text-guided enhancement model and the diffusion model; wherein the image quality enhancement requirement text features are obtained by encoding the image quality enhancement requirement text; the first image features are obtained by encoding the first image; the second image features are obtained by adding noise to the first image and extracting features from the noise-added result; according to the first round of fused text image bimodal features, the image quality enhancement requirement text features, the first image features, the text-guided enhancement model and the diffusion model, N-1 iterations are performed to obtain the first enhanced image coding features; wherein, the diffusion model has the ability to perform denoising on the second image features, and extract features from the denoising results to generate a first round of diffusion image feature set; and the ability to perform j-1 iterations based on the first round of fused text image bimodal features to generate the jth round of diffusion image feature set; the text-guided enhancement model has the ability to fuse the image quality enhancement requirement text features, the first image features and the jth round of diffusion image feature set in each iteration process to generate the first enhanced image coding features; j is a positive integer, 2≤j≤N; N is a positive integer; the first enhanced image coding features are decoded to obtain a second image; the second image is an image obtained by enhancing the image quality of the first image according to the image quality enhancement requirement text.

[0022] In combination with the second aspect, in a possible implementation method, the processing module is also configured to use the first processing module of the text-guided enhancement model to process the text features required for image quality enhancement and the first image features to obtain the text-image bimodal features corresponding to the first processing module in the first round; use the diffusion model to denoise the second image features, and perform feature extraction on the denoising results to generate a first-round diffusion image feature set; the first-round diffusion image feature set includes M features; the diffusion model includes M processing modules; each processing module in the diffusion model corresponds to a feature; M is a positive integer; the text-image bimodal features corresponding to the first processing module in the first round and the first diffusion image features are fused to obtain the fused text-image bimodal features corresponding to the first processing module in the first round; the first diffusion image features are features output by the first processing module of the diffusion model; according to the fused text-image bimodal features corresponding to the first processing module in the first round, other features in the first-round diffusion image feature set except the first diffusion image features, and the text-guided enhancement model, the first-round fused text-image bimodal features are obtained.

[0023] In combination with the second aspect, in a possible implementation method, the processing module is also configured to determine the j-th round diffusion image feature set according to the j-1-th round fused text image bimodal features and the diffusion model during the j-th round of iteration; wherein j is a positive integer, 2≤j≤N; and obtain the j-th round fused text image bimodal features according to the j-th round diffusion image feature set, the image quality enhancement required text features, the first image features and the text-guided enhancement model; wherein, when j=N, the j-th round fused text image bimodal features are the first enhanced image coding features.

[0024] In combination with the second aspect, in a possible implementation method, the processing module is also configured to use the first processing module of the text-guided enhancement model to process the text features required for image quality enhancement and the first image features to obtain the text-image bimodal features corresponding to the first processing module in the j-th round; fuse the features output by the first processing module of the diffusion model in the j-th round diffusion image feature set and the text-image bimodal features corresponding to the first processing module in the j-th round to obtain the fused text-image bimodal features corresponding to the first processing module in the j-th round; obtain the j-th round fused text-image bimodal features based on the fused text-image bimodal features corresponding to the first processing module in the j-th round, the features in the j-th round diffusion image feature set except the features output by the first processing module of the diffusion model, and the text-guided enhancement model.

[0025] In combination with the second aspect, in a possible implementation method, the processing module is also configured to input the fused text image bimodal features corresponding to the i-1th processing module in the jth round into the i-th processing module of the text-guided enhancement model to obtain the text image bimodal features corresponding to the i-th processing module in the jth round; wherein the text-guided enhancement model includes M processing modules, i is a positive integer, 2≤i≤M; the features output by the i-th processing module of the diffusion model in the j-th round diffusion image feature set and the text image bimodal features corresponding to the i-th processing module in the j-th round are fused to obtain the fused text image bimodal features corresponding to the i-th processing module in the j-th round; when i=M, the fused text image bimodal features corresponding to the M-th processing module in the j-th round are input into the output module of the text-guided enhancement model to obtain the j-th round fused text image bimodal features.

[0026] In combination with the second aspect, in a possible implementation method, the picture quality enhancement trigger operation includes picture quality enhancement requirement audio; the processing module is also configured to identify the picture quality enhancement requirement audio when the picture quality enhancement trigger operation includes picture quality enhancement requirement audio to obtain the picture quality enhancement requirement text.

[0027] In combination with the second aspect, in a possible implementation, the diffusion model is a pre-trained implicit diffusion model, and the text-guided enhancement model is a u-net model.

[0028] In combination with the second aspect, in a possible implementation, the requirements for better image quality are to reduce noise, reduce blur and improve resolution.

[0029] In a third aspect, the present disclosure provides an electronic device, comprising: a memory, a display screen, and one or more processors; the memory, the display screen, and the processor are coupled. The memory is used to store computer program codes, and the computer program codes include computer instructions; when the electronic device is running, the processor is used to execute one or more computer instructions stored in the memory, so that the electronic device performs the image processing method as described in any one of the first aspects above.

[0030] In a fourth aspect, the present disclosure provides a computer storage medium, comprising computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method as described in any one of the first aspects.

[0031] In a fifth aspect, the present disclosure provides a computer program product. When the computer program product is executed on an electronic device, the electronic device executes the image processing method as described in any one of the first aspects.

[0032] In a sixth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor, which is used to support a first device to implement the functions involved in the first aspect. In a possible design, the device also includes a memory, which is used to store program instructions and data necessary for the first device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0033] It should be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of an image generation network provided in an embodiment of the present disclosure.

[0035] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure.

[0036] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present disclosure.

[0037] Figure 4 One of the flowcharts of an image processing method provided in an embodiment of the present disclosure.

[0038] Figure 5 This is one of the display interface schematic diagrams provided in an embodiment of the present disclosure.

[0039] Figure 6 The present invention provides a second display interface schematic diagram of an embodiment of the present invention.

[0040] Figure 7 The second flowchart of an image processing method provided in an embodiment of the present disclosure.

[0041] Figure 8 The third flowchart of an image processing method provided in an embodiment of the present disclosure.

[0042] Fig. 9 A structural block diagram of a chip system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The technical solution in the embodiment of the present disclosure will be described below in conjunction with the drawings in the embodiment of the present disclosure. Among them, in the description of the present disclosure, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present disclosure is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present disclosure, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, in the embodiments of the present disclosure, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for easy understanding.

[0044] In addition, the network architecture and business scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0045] With the widespread popularity of electronic devices (such as mobile phones) and the rapid development of the Internet, users often encounter various situations in which the quality of the images they obtain is poor when taking photos with their mobile phones or obtaining images through the Internet. Therefore, users have a need to improve image quality.

[0046] For example, when taking photos with a mobile phone, due to the influence of various factors, some poor image quality may occur. For example, when a user holds the mobile phone to take photos, if the hand is not stable enough, it is easy to take a shaky photo. In this case, the object in the photo may have a blurred outline and lost details, which seriously affects the quality and viewing experience of the photo.

[0047] For example, weather conditions can also affect the photo quality. For example, when taking photos on rainy days, raindrops will appear on the photos, which may block some of the scenery, making the photos look messy, and also affect the color and contrast of the photos. Therefore, in this case, users need to remove raindrops and restore the original clear picture of the photos.

[0048] For another example, the Internet is an important platform for information dissemination, and people often download various images from the Internet. However, in order to facilitate storage and transmission, many images are compressed to a certain extent. These compressed images often have problems such as low clarity, color distortion, and blurred details. When users copy these images to their local computer, if they need to use these images, such as for printing, display, or further editing, there will be a need to enhance the clarity and restore the original quality of the image. In other words, in multiple possible image application scenarios (i.e., the above scenarios), users generally have a demand for image quality enhancement.

[0049] In the related art, image generation network can be used to process the image quality enhancement requirements. Figure 1 , the embodiment of the present disclosure provides a structural diagram of an image generation network. The image generation network includes a stable diffusion model (SD model), a control network (ControlNet), a text encoder (TextEncoder) and a time encoder (Time Encoder).

[0050] Among them, the SD model is used to t ) generates a preset image. The control network is used to add training parameters in the diffusion stage of the SD model to assist in adjusting the weight parameters in the SD model. The text encoder is used to generate the prompt text. t ) is used to encode the text to get the text embedding vector. The time encoder is used to convert the time step (Time t) into a time embedding vector.

[0051] The input data of the SD model includes the requirement text c t , time step t and input data z t , the output data is Output θ (z t ,t,c i ,c f ), the output data is used to represent the time t, the demand text c t and conditional input (Condition c f ) under the condition that according to the input data z t Generate output data θ . Among them, the condition input c f is the input data of the control network. For example, z t Can be a noise image, output data θ It can be a denoised image.

[0052] The SD model includes multiple encoders and multiple decoders, and the multiple encoders and multiple decoders correspond to each other one by one. The multiple encoders are used to gradually downsample the required text c t , time step t and input data z t , to extract features of different scales. Multiple decoders are used to upsample features step by step to reconstruct the output data θ .

[0053] For example, Figure 1 As shown in FIG. 1 , the SD model can be a U-shaped network. The multiple encoders in the SD model include SD Encoder Block A, SD Encoder Block B, SD Encoder Block C, and SD Encoder Block D. Among them, SD Encoder Block A, SD Encoder Block B, SD Encoder Block C, and SD Encoder Block D each contain three identical sub-blocks. The output data of SD Encoder Block A is 64 64 matrix, the output data of SD encoder B is 32 32 matrix, the output data of SD encoder C is 16 16 matrix, the output data of SD encoder D is 8 The matrix of 8, the output data of the intermediate module of the SD model is also 8 8 matrix.

[0054] The multiple decoders include SD Decoder Block D, SD Decoder Block C, SD Decoder Block B, and SD Decoder Block A. Among them, SD Decoder Block A, SD Decoder Block B, SD Decoder Block C, and SD Decoder Block D, each of which contains three identical sub-blocks. The output data of SD Decoder Block D is 8 8 matrix, the output data of SD encoder C is 16 16 matrix, the output data of SD decoder B is 32 32 matrix, the output data of SD decoder A is 64 That is, SD encoder A and SD decoder A, SD encoder B and SD decoder B, SD encoder C and SD decoder C, SD encoder D and SD decoder D correspond one to one.

[0055] SD encoder D and SD decoder D are connected through SD Middle Block. The scale of the output data of SD Middle Block is the same as the scale of the output data of SD encoder D and SD decoder D. The output data of SDSD Middle Block is 8 8 matrix.

[0056] The input data of the control network is the conditional input c f For example, the conditional input c f It can be an edge map or a semantic segmentation map, etc.

[0057] The control network includes multiple zero convolution layers, multiple encoders, and multiple zero convolution layers. f After inputting the zero convolution layer of the control network, the zero convolution layer treats the conditional input c f Perform convolution processing to obtain the convolution result. Then, the input data z of the SD model is merged into t Combined with the convolution result to get the fused feature vector.

[0058] Next, the control network uses multiple encoders to fused feature vectors and demand text c t and time step t for further processing to output feature vectors of different scales. The feature vectors of different scales are then input into multiple decoders of the SD model through the zero convolution layer corresponding to each encoder to assist in adjusting the weight parameters in the SD model. The feature vectors of different scales can also be called training copies.

[0059] For example, refer to Figure 1 , the multiple encoders in the control network include SD encoder A (SD Encoder Block A), SD encoder B (SD Encoder Block B), SD encoder C (SD Encoder Block C), SD encoder D (SD Encoder Block D) and SD middle block (SD Middle Block). The SD encoder A in the control network corresponds to the SD encoder A in the U-type network, the SD encoder B corresponds to the SD encoder B in the U-type network, the SD encoder C corresponds to the SD encoder C in the U-type network, the SD encoder D corresponds to the SD encoder D in the U-type network, and the SD model middle block corresponds to the SD model middle block in the U-type network. Among them, the scale of the output data of the SD encoder AD in the control network is consistent with the scale of the output data of the SD encoder AD in the U-type network.

[0060] After each encoder outputs data (i.e., feature vectors of different scales), the output data of each encoder can be processed by a corresponding zero convolution layer, and the processing result can be input into the decoder corresponding to each one in the SD model as a training parameter, so that the SD model can obtain the final output data θ .

[0061] Combining the above SD model and the structure of the control network, we can see that the SD model encodes and decodes the latent variable z step by step. t , and incorporate the required text into each step of the encoding and decoding process t , time step t, to generate a preset image. The control network inputs control conditions through zero convolution during the process of the SD model generating the preset image, thereby generating a more targeted preset image. In other words, the above-mentioned image generation network usually takes image generation as the main processing core under the control of the demand text and conditions entered by the user, rather than image quality enhancement. Therefore, when faced with the demand for image quality enhancement, the above-mentioned image generation network cannot achieve a better processing effect.

[0062] In other examples, when users have a need for image quality enhancement, the relevant image processing methods often prompt users to perform a large number of repeated operations (e.g., adjusting noise parameters by sliding, adjusting deblurring parameters by sliding, increasing resolution by sliding, etc.) under specific functions (e.g., noise reduction function, blur removal function, resolution increase function, etc.) to achieve image quality enhancement. This method is not only cumbersome, but may not fully meet the user's needs for image quality enhancement. Therefore, how to build a more convenient and flexible image processing solution to effectively meet the user's needs for image quality enhancement has become an important issue that needs to be solved urgently.

[0063] To this end, the embodiment of the present disclosure provides an image processing method. After the electronic device receives the user's image quality enhancement trigger operation, the diffusion model and the text-guided enhancement model can be used to perform multiple iterations to generate a second image with enhanced image quality. In each iteration, after the diffusion model generates a diffusion image feature set (i.e., the Nth round of diffusion image feature set), the text-guided enhancement model can fuse the diffusion image feature set into the image quality enhancement requirement text features and the first image features. In this way, in each iteration, the image quality enhancement requirement text can be referenced to generate the second image. Thereby, the final second image meets the user's image quality enhancement requirements, and the implementation process is convenient and simple, and the user only needs to perform the image quality enhancement trigger operation.

[0064] The technical solution provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0065] The technical solution provided in the present application can be applied to electronic devices with image display functions. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device, etc. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0066] For example, taking the electronic device as a mobile phone, Figure 2 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0067] Reference Figure 2 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a display screen 193, a subscriber identification module (SIM) card interface 194, and a camera 195, etc. Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0068] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0069] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0070] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0071] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0072] The charging management module 140 is used to receive charging input from a power supply device (e.g., a charger, a laptop power supply, etc.). The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device.

[0073] While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141. The battery 142 can be composed of multiple batteries connected in series. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.

[0074] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 193, the camera 195, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as the battery voltage, current, battery cycle number, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110.

[0075] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0076] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc. The non-volatile memory can also store executable programs and store user and application data, etc., which can be loaded into the random access memory in advance for direct reading and writing by the processor 110. In an embodiment of the present application, a diffusion model may be stored in the internal memory 121. The internal memory 121 may also store a related model that can convert an image into a noise image and a text identifier, or may also store noise images and text identifiers corresponding to multiple images.

[0077] A touch sensor is also called a "touch control device". The touch sensor can be arranged on the display screen 193. The touch sensor and the display screen 193 form a touch screen, also called a "touch control screen". The touch sensor is used to monitor touch operations acting on or near it. The touch sensor can pass the monitored touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 193. In other embodiments, the touch sensor can also be arranged on the surface of the electronic device, which is different from the position of the display screen 193.

[0078] The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor can be set on the display screen 193. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 193, the electronic device monitors the touch operation intensity according to the pressure sensor. The electronic device can also calculate the touch position based on the monitoring signal of the pressure sensor. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0079] In some embodiments, the electronic device may include 1 or N cameras 195, where N is a positive integer greater than 1. In the embodiment of the present application, the type of camera 195 can be distinguished according to the hardware configuration and the physical location. For example, the camera arranged on the side of the display screen 193 of the electronic device can be called a front camera, and the camera arranged on the side of the back cover of the electronic device can be called a rear camera; for another example, a camera with a short focal length and a larger viewing angle can be called a wide-angle camera, and a camera with a long focal length and a small viewing angle can be called a normal camera. Among them, the length of the focal length and the size of the viewing angle are relative concepts, and there is no specific parameter limitation. Therefore, wide-angle cameras and ordinary cameras are also a relative concept, and can be specifically distinguished according to physical parameters such as focal length and viewing angle.

[0080] The electronic device implements the display function through a GPU, a display screen 193, and an application processor. The GPU is a microprocessor for image editing, connecting the display screen 193 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0081] The electronic device can realize the shooting function through ISP, camera 195, video codec, GPU, display screen 193 and application processor. GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. In the embodiment of the present application, the function of the GPU is used in the frame drawing process of each image frame so that the final displayed picture obtains better display effect and performance.

[0082] The ISP is used to process the data fed back by the camera 195. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 195. The camera 195 is used to capture static images or videos.

[0083] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0084] The display screen 193 is used to display images, videos, etc. The display screen 193 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 193, where N is a positive integer greater than 1.

[0085] In an embodiment of the present application, the display screen 193 can be used to display the interface of the electronic device (for example, the desktop, the lock screen interface, etc.), and display images stored in the electronic device (for example, wallpapers, photos, etc.), or images captured by any one or more cameras 195 in the interface.

[0086] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem and baseband processor.

[0087] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0088] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0089] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 193. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0090] The wireless communication module 160 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0091] The SIM card interface 194 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device by inserting it into or removing it from the SIM card interface 194. The electronic device can support one or more SIM card interfaces. The SIM card interface 194 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 194 at the same time. The SIM card interface 194 can also be compatible with external memory cards. The electronic device interacts with the network through the SIM card to implement functions such as calls and data communications. One SIM card corresponds to one user number.

[0092] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0093] Of course, it is understandable that the above Figure 2 The figure is only an exemplary description when the electronic device is in the form of a mobile phone. If the electronic device is in the form of a tablet computer, a handheld computer, a PC, a PDA, a wearable device (such as a smart watch, a smart bracelet), etc., the structure of the electronic device may include Figure 2 The structure shown in the figure is less than Figure 2 More structures are shown in the figure, which are not limited here.

[0094] It is understandable that, in general, the realization of electronic device functions requires not only hardware support but also software cooperation. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android® system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0095] Figure 3 A schematic diagram of the layered architecture of the software system of the electronic device provided in the embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces (such as APIs).

[0096] In some examples, reference Figure 3 As shown, in the embodiment of the present application, the software of the electronic device is divided into five layers, from top to bottom, namely, the application layer, the framework layer (or called the application framework layer), the system library and the Android runtime (android runtime), the HAL layer (hardware abstraction layer, hardware abstraction layer) and the driver layer (or called the kernel layer). Among them, the system library and the Android runtime can also be called the local framework layer or the native layer.

[0097] The application layer can include a series of applications. Figure 3 As shown, the application layer may include applications (APP) such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, instant messaging, and short message.

[0098] In an embodiment of the present application, the gallery application may enhance the image quality of a certain image (eg, the first image) based on a user's image quality enhancement instruction to improve the image quality of the image.

[0099] The framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions or services. For example, the application framework layer may include an activity manager, a window manager, a content provider, an audio service, a view system, a phone manager, a resource manager, a notification manager, a package manager, etc., and the embodiments of the present application do not impose any restrictions on this.

[0100] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0101] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0102] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images. In some embodiments, the view system can also include or start a rendering thread to complete operations such as drawing a frame buffer.

[0103] The phone manager is used to provide communication functions of the electronic device. For example, the phone manager can manage the call status of the call application (including initiation, connection, hang up, etc.).

[0104] The resource manager provides various resources for the application, such as localized strings, icons, images, layout files, video files, and so on.

[0105] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0106] Package Manager on Android ®The system is used to manage application packages. It allows applications to obtain detailed information about installed applications and their services, permissions, etc. The package manager is also used to manage events such as application installation, uninstallation, and upgrades.

[0107] In the embodiment of the present application, the framework layer may also include a picture quality enhancement service having the functions of the aforementioned gallery application. If the gallery application does not exist in the application layer or the gallery application cannot be used or the user needs to change the display effect of the image but the gallery application cannot operate it, the picture quality enhancement service performs the same action as the gallery application on the corresponding image, so that the corresponding image presents a display effect consistent with the picture quality enhancement direction.

[0108] The system library can include multiple functional modules. For example: surface manager, media library, OpenGL ES, SGL, etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. SGL is a drawing engine for 2D drawing. The Android runtime includes the core library and the ART virtual machine. Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one is the function function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the ART virtual machine. The ART virtual machine executes the Java files of the application layer and the application framework layer as binary files. The ART virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0109] The HAL layer is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The HAL layer provides a standard interface to display device hardware functions to the higher-level Java API framework (i.e., the framework layer). The HAL layer contains multiple library modules, each of which implements an interface for a specific type of hardware component, such as: audio HAL audio module, bluetooth HAL Bluetooth module, camera HAL camera module (also called camera HAL or camera hardware abstraction module), sensors HAL sensor module (or Isensor service, sensor service).

[0110] The kernel layer is a layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, a battery driver, etc., which are not limited in this application. Among them, the sensor driver can specifically include the driver of each sensor included in the electronic device, such as an ambient light sensor driver, etc. Exemplarily, the ambient light sensor driver can respond to the indication or instruction of the sensor module to obtain the detection data, and send the detection data of the ambient light sensor to the sensor module in a timely manner.

[0111] The technical solutions provided in the embodiments of the present application can be implemented in electronic devices having the above-mentioned hardware architecture or software architecture.

[0112] The following combination Figure 4 As shown, the image processing method provided in the embodiment of the present application is introduced. Figure 4 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. Figure 4 As shown, taking the electronic device as a mobile phone as an example, the image processing method may include S401-S406:

[0113] S401: A first application of a mobile phone receives a first input operation for a first image.

[0114] The first application may be any application with an image quality enhancement function, for example, the first application may be a gallery application, a camera application, a beauty application, and a smart assistant application, etc. The first application may also have an image viewing function.

[0115] The first image is an image that needs to be enhanced in image quality. The first image may be an image viewed locally by a user on a mobile phone (eg, an image in a gallery application), or an image viewed on a network (eg, an image in a browser).

[0116] The first input operation includes first input data, and the first input operation is used to trigger the first application to perform image quality enhancement on the first image based on the first input data. The first input operation can also be called an image quality enhancement trigger operation. Exemplarily, the image quality enhancement performed on the first image includes but is not limited to: reducing noise in the first image; brightening the first image; improving the resolution of the first image; removing blur in the first image; and removing artifacts in the first image.

[0117] The first input data is used to represent the user's image quality enhancement requirement. The first input data is first text data (also referred to as image quality enhancement requirement text), or first audio data (also referred to as image quality enhancement requirement audio). The first text data is used to express the user's image quality enhancement requirement in the form of text. The first audio data is used to express the user's image quality enhancement requirement in the form of audio.

[0118] Exemplarily, the first input operation is used to trigger the first application to perform image quality enhancement on the first image based on the first text data or the first audio data. The first text data may be first pan-language text data. The first pan-language text data refers to text data covering a wide range of language categories such as multiple languages, language variants, language expressions, and language-related information.

[0119] For example, the first pan-language text data may be "remove raindrops from the image", "I want to make this image clearer", etc. When the first input data is the first pan-language text data, the user can freely express the image quality enhancement requirement for the first image. In this way, the user does not need to repeatedly consider how to express it accurately, so that the image quality enhancement function can be used more conveniently.

[0120] When the user needs to enhance the image quality of the screen content of the first image, the user can trigger the first input operation so that the mobile phone responds to the first input operation and turns on the image quality enhancement function for the first image. When the mobile phone turns on the image quality enhancement function for the first image, the mobile phone can perform an image quality enhancement operation on the first image that complies with the first input operation. Among them, the first input operation can be any interactive instruction that can turn on the image quality enhancement function for the first image, such as a text instruction, an audio instruction, etc. When the image quality enhancement function for the first image is turned on, the mobile phone will perform an image quality enhancement operation on the first image that complies with the first input operation, so that the first image produces an image quality enhancement effect that complies with the first input operation. That is, the subsequent S402-S406 are executed.

[0121] In some examples, when the first input operation includes the first text data, the first application of the mobile phone can receive the first input operation including the first text data.

[0122] For example, when the user needs to enable the image quality enhancement function, he can first use any feasible trigger operation to open the display interface of the first image. In response to the trigger operation, the mobile phone can display the following Figure 5 The display interface 501 of the first image shown in (a) of FIG. 5 includes a picture quality enhancement function control 502. The picture quality enhancement function control 502 is used to implement the picture quality enhancement function for the first image. Of course, the display interface 501 of the first image may also include other controls (not shown in the figure), such as an edit control, a delete control, a forwarding control, etc., which will not be described one by one here.

[0123] The user may perform an operation (such as a click operation) to enable the image quality enhancement function control 502. In response to the user's operation to enable the image quality enhancement function control 502, the mobile phone may enable the image quality enhancement function for the first image. At this time, the mobile phone may display the following Figure 5 The image quality enhancement operation interface 503 shown in (b) of FIG. The image quality enhancement operation interface 503 includes a command input area 504 , a cancel control 507 and a completion control 508 .

[0124] The instruction input area 504 is used to receive first input data input by the user in the instruction input area 504. Afterwards, the user's expectation of the image quality enhancement effect for the first image can be determined according to the first input data.

[0125] The cancel control 507 is used to cancel the image quality enhancement and restore the first image to its original state. The cancel control 507 can be triggered at any time, for example, after the user inputs the first input data.

[0126] The completion control 508 is used to trigger the first application to enhance the image quality of the first image according to the first input data input by the user in the instruction input area 504. Alternatively, the completion control 508 is used to return to the display interface 501 of the first image after the first application enhances the image quality of the first image according to the first input data. The present disclosure does not limit the function of the completion control, which is subject to actual use.

[0127] In some examples, the instruction input area 504 may include at least one input control, so that the user can input the first input data through the input control. For example, the at least one input control may include a text input control. For another example, the at least one input control may include a voice input control. For another example, Figure 5 As shown in (b), the instruction input area 504 includes a text input control 505 and a voice input control 506. The text input control 505 is shown in the form of a text input box. The present disclosure does not limit the specific form of the input control in the instruction input area 504, which shall be subject to actual use.

[0128] The text input control 505 is used to receive the first text data input by the user. For example, the first text data input by the user is "remove the raindrops in the picture", "I want to make this picture clearer", etc. The voice input control 506 is used to receive the first audio data input by the user. For example, the first audio data input by the user is: audio including "remove the raindrops in the picture", audio including "I want to make this picture clearer", etc.

[0129] In other embodiments, in the display interface of the first image, the first input operation may also be a voice command input by the user to the mobile phone. The voice command is used to instruct the mobile phone to enhance the image quality of the first image to make the display effect of the first image better.

[0130] Specifically, in the display interface of the first image, the mobile phone can first use its own microphone to receive the smart assistant wake-up command spoken by the user, such as "Hello, YOYO". Then, in response to the smart assistant wake-up command, the mobile phone can display the following: Figure 6 The smart assistant interface 601 shown in (a).

[0131] Then, refer to Figure 6 As shown in (b), the smart assistant of the mobile phone will use the microphone of the mobile phone to obtain and recognize the user's next voice command, such as "remove the raindrops in the picture", and display the following Figure 6 The smart assistant interface 602 is shown in (b).

[0132] Finally, refer to Figure 6 In the smart assistant interface 603 shown in (c), the mobile phone can remove the raindrops in the first image and display the second image in response to the user's voice command "remove the raindrops in the image". The second image is the image after the raindrops in the first image are removed.

[0133] Of course, in practice, the wake-up command of the voice assistant can be determined according to the design of different mobile phones, and the voice command input / spoken by the user can also be any voice command that can indicate the activation of the image quality enhancement function. Of course, in practice, there are any other feasible first input operations for image quality enhancement of the first image, and the above are only examples and are not intended to be specific limitations on the embodiments of the present application.

[0134] S402 . In response to a first input operation, identify first input data to obtain a first text feature E (also referred to as a text feature required for image quality enhancement).

[0135] Among them, the first text feature E is used to characterize the image quality enhanced text encoding vector corresponding to the first input data.

[0136] In some examples, the first text feature E is obtained by encoding the first text data. Exemplarily, when the first input data is the first text data, in response to the first input operation, the first text data is encoded by a text encoder to obtain the first text feature E.

[0137] The text encoder (also referred to as a pre-trained text encoder) may be an open source neural network model. For example, the text encoder may be based on any of the bidirectional encoder representation from transformers (BERT), GPT-3, etc.

[0138] In some examples, in the embodiments of the present application, the text encoder used can be trained using any feasible training method, as long as the text encoder finally trained can generate a first text feature E that matches the first text data based on the first text data.

[0139] In some examples, such as Figure 7 As shown, in response to a first input operation, a text encoder is used to encode the first text data to obtain a first text encoding feature, which may include: first, segmenting the first text data to obtain at least one word; then converting the at least one word into a word vector corresponding to at least one word; and then encoding the word vector corresponding to the at least one word to obtain a first text feature E.

[0140] The first text feature E is a high-dimensional representation of the first text data. The first text feature E can be stored in the form of Embedding, and the first text feature E can represent a vector of a fixed length, for example, the first text feature E is a 768-dimensional vector.

[0141] Exemplarily, the process of converting at least one vocabulary word into at least one word vector corresponding to the vocabulary word may include: inputting at least one vocabulary word into a word embedding layer of a text encoder (e.g., a word to vector conversion model (Word to Vector, Word2Vec), a global word vector (GloVe), or a built-in embedding layer of BERT), and the word embedding layer outputs a word vector corresponding to the at least one vocabulary word.

[0142] The process of encoding the word vector corresponding to at least one word to obtain the first text feature E may include: inputting the word vector corresponding to at least one word into a neural network in a text encoder (e.g., a sequence-to-sequence model (Transformer) structure with a multi-layer self-attention mechanism) for encoding processing, and the neural network outputs the first text feature E.

[0143] In some examples, when the first input data is first audio data, in response to the first input operation, the first audio data is recognized using a speech recognition model to obtain first text data, and then the first text data is encoded using a text encoder to obtain a first text feature E.

[0144] Exemplarily, the speech recognition model may be any one of DeepSpeech or Wav2Vec 2.0, etc. In the embodiment of the present application, the speech recognition model used may be trained by any feasible training method, as long as the speech recognition model finally trained can generate first text data matching the first audio data based on the first audio data.

[0145] The process of using a speech recognition model to recognize the first audio data to obtain the first text data may include: first, the speech recognition model divides the first audio data into at least one frame segment, and then the speech recognition model extracts features from the at least one frame segment to obtain acoustic features corresponding to the at least one frame segment; then, the speech recognition model converts the acoustic features corresponding to the at least one frame segment into the first text data.

[0146] Exemplarily, the acoustic features corresponding to at least one frame segment may include Mel Frequency Cepstral Coefficients (MFCC), a spectrogram, and the like.

[0147] In some examples, the process of converting the acoustic features corresponding to at least one frame segment into text by the speech recognition model may include: the speech recognition model converts the acoustic features corresponding to at least one frame segment into first text data through a neural network (e.g., a recurrent neural network (RNN), a convolutional neural network (CNN), a Transformer, etc.).

[0148] After the first audio data is processed to obtain the first text data, the first text data is encoded using a text encoder to obtain the first text feature E. The process is similar to the above content and will not be repeated here.

[0149] S403 , input the first image into a first image encoder to obtain a first image feature q1 .

[0150] In some examples, the first image feature is obtained by encoding the first image. Exemplarily, after determining that the first input operation is for the first image, the first image can be input into a first image encoder to preliminarily obtain the first image feature q1 of the first image.

[0151] The first image encoder may also be referred to as a pre-trained first image encoder. The first image encoder is used to generate a first image feature q1 of the first image.

[0152] The first image encoder may be any one of CNN, Transformer, Residual Network (ResNet), Visual Geometry Group (VGG), Inception Network (Inception), etc. In the embodiment of the present application, the first image encoder used may be trained by any feasible training method, as long as the first image encoder finally trained can generate the first image feature q1 matching the first image based on the first image.

[0153] In some examples, such as Figure 7 As shown, the process of inputting the first image into the first image encoder to obtain the first image feature q1 may include: inputting the first image into the first image encoder, the first image encoder passes through a series of convolutional layers, pooling layers and fully connected layers, extracts the features of the first image, and finally outputs a feature vector of a fixed length (i.e., the first image feature q1).

[0154] Exemplarily, the first image feature q1 is a vector of fixed length, such as 2048 dimensions or 768 dimensions. The first image feature q is used for subsequent image quality enhancement tasks.

[0155] S404: Input the first image and the first time series into a second image encoder to obtain a second image feature z1.

[0156] The second image feature z1 is obtained by adding noise to the first image and then performing feature extraction on the noise-added result.

[0157] In some examples, the first time series T includes t0, ..., t i ,…,t s Inputting the first image and the first time series into the second image encoder to obtain the second image feature process may include: from T=t0 to T=t sIn the process, the noise adding operation is gradually performed on the first image until a noise image is obtained; then, feature extraction is performed on the noise image to obtain a second image feature z1.

[0158] In some examples, the second image encoder may be a diffusion model. The noise image may be obtained by the second image encoder adding any possible type of noise (e.g., Gaussian noise) to any image multiple times. The noise type may be determined by the second image encoder during the training process. When the second image encoder is used at any subsequent time, the noise type involved in the image should be consistent to ensure that the second image encoder can successfully use the information loss effect caused by the learned noise to process the image.

[0159] The diffusion model is a generative model that generates data samples (for example, the data samples can be Gaussian noise images) by gradually adding noise to the data through a forward diffusion process. Subsequently, the noise is gradually removed through a reverse diffusion process, thereby restoring the original data from the Gaussian noise image.

[0160] It can be understood that S404 is similar to the forward diffusion process of the diffusion model, that is, through a series of fixed time steps, noise is gradually added to the image to be processed (ie, the first image), thereby obtaining a noisy image.

[0161] In the embodiment of the present application, the second image encoder used can be trained by any feasible training method, as long as the second image encoder finally trained can generate a matching second image feature z1 based on the first image and the first time series.

[0162] S405 . Based on the first text feature E, the first image feature q1 , the text-guided enhancement model B, the second image feature z1 and the diffusion model P, a first enhanced image coding feature is obtained after a preset number of iterations.

[0163] Among them, the text-guided enhancement model B is used to fuse the first text feature E and the first image feature q1.

[0164] The preset number of times may be N times, where N is a positive integer, and the present disclosure does not limit the value of N.

[0165] The diffusion model has the ability to perform denoising on the second image features and extract features from the denoising results to generate a first round of diffusion image feature set; and to perform N-1 iterations based on the first round of fused text image bimodal features to generate an Nth round of diffusion image feature set.

[0166] The text-guided enhancement model has the ability to generate the first enhanced image coding features by integrating the text features required for image quality enhancement, the first image features, the first round diffusion image feature set, and the Nth round diffusion image feature set.

[0167] The first enhanced image coding feature is used to characterize the coding feature corresponding to the second image after the first image is enhanced according to the first input data.

[0168] Exemplarily, the text-guided enhancement model B can be a convolutional network (u-net) model for biomedical image segmentation. In the embodiment of the present application, the text-guided enhancement model B used can be trained by any feasible training method, as long as the text-guided enhancement model B finally trained can generate a fused text-image bimodal feature q based on the first text feature E and the first image feature q1. n '.

[0169] like Figure 7 As shown, after obtaining the second image feature z1, the second image feature z1 can be used as the input data of the diffusion model in the first round of iteration. The diffusion model P outputs multiple diffusion image features X through multiple processing modules. n .

[0170] In combination with S404, it can be seen that the diffusion model P includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, the first image is gradually transformed into an image feature corresponding to the noise image, that is, the second image feature z1. Subsequently, the reverse diffusion process is started, and the second image feature z1 is used as input data. The denoising operation is gradually performed in multiple processing modules of the diffusion model P, thereby generating image features corresponding to the denoised image (that is, multiple diffusion image features X n ).

[0171] Exemplarily, the diffusion model P can be a pre-trained implicit diffusion model, and the weights of the pre-trained implicit diffusion model are locked. This means that the parameters of the pre-trained implicit diffusion model will not be updated during a specific task or further training. The weight parameters of the pre-trained implicit diffusion model have been determined in the pre-training stage, and the fixed weights are directly used to perform various operations in subsequent use.

[0172] For example, the pre-trained implicit diffusion model P can be pre-trained on large-scale unsupervised data to learn common features and patterns in the training data. In the embodiment of the present application, the diffusion model P used can obtain training results through any feasible training means, provided that the diffusion model P after training can successfully generate multiple diffusion image features X based on the second image feature z1n .

[0173] As can be seen from the foregoing, the diffusion model P is a generation model. Therefore, the present application can integrate the user's image quality enhancement requirements into the image generation process, thereby obtaining an image that meets the user's image quality enhancement requirements.

[0174] In some examples, in the process of diffusion model P generating an image, an image that meets the user's image quality enhancement requirements can be obtained by introducing a text-guided enhancement model B. The text-guided enhancement model B is used to further process the first text feature E and the first image feature q1, and output a text-image bimodal feature. Since the first text feature E is an encoding of the image quality enhancement requirement expressed by the user through the first input data, it is equivalent to inputting the user's image quality enhancement requirement into the diffusion model P through the text-guided enhancement model B. In this way, the diffusion model P can generate an image that meets the user's image quality enhancement requirements.

[0175] Exemplarily, the diffusion model P and the text-guided enhancement model B may include multiple processing modules. Based on the above, after the text-guided enhancement model B outputs the text-image bimodal features, the output results of each processing module of the diffusion model P (e.g., the diffusion image features after noise removal) and the output results of each processing module of the text-guided enhancement model B (e.g., the text-image bimodal features) may be fused through the hierarchical fusion module f. Through this fusion method, the user's image quality enhancement needs can be gradually and effectively integrated into the image generation process.

[0176] It is understandable that the above process is an iterative process, and multiple iterations (ie, N times) can be performed. In this way, the first enhanced image coding features finally output by the diffusion model P will become richer and richer, not only including the information of the first image, but also fully reflecting the user's image quality enhancement needs.

[0177] In some examples, the hierarchical fusion module f may be a separate module in the electronic device, or a module in the text-guided enhancement model B. The setting location of the hierarchical fusion module f disclosed in the present invention is not limited.

[0178] Based on this, Figure 8 As shown, in the embodiment of the present disclosure, based on the first text feature E, the first image feature q1, the text-guided enhancement model B, the second image feature z1 and the diffusion model, after a preset number of iterations, the first enhanced image coding feature is obtained, which may specifically include S1-S3.

[0179] S1. Input the second image feature z1 into the diffusion model P to obtain the first round diffusion image feature set a1.

[0180] The diffusion model includes M processing modules. The first round diffusion image feature set a1 is the M diffusion image features output by the M processing modules of the diffusion model during the first round of iterations. Each processing module may include at least one network layer, and the scale of the output data of each network layer in the same processing module is consistent.

[0181] For example, the first round diffusion image feature set a1={ X 11 ,…, X 1i ,…, X 1M}. Among them, X 11 is the feature output by the first processing module of the diffusion model in the first round of iteration; X 1i is the feature output by the i-th processing module of the diffusion model in the first round of iteration; X 1M is the feature output by the Mth processing module of the diffusion model in the first round of iteration, where i is a positive integer.

[0182] For ease of understanding, an example is given with M=3 to illustrate the contents related to the number of all involved processing modules.

[0183] For example, the diffusion model includes three processing modules. The first round of diffusion image feature set a1={ X 11 , X 12 , X 13}. X 11 is the feature output by the first processing module of the diffusion model in the first round of iteration; X 12 is the feature output by the second processing module of the diffusion model in the first round of iteration; X 13 It is the feature output by the third processing module of the diffusion model in the first round of iteration.

[0184] The process of inputting the second image feature z1 into the diffusion model P to obtain the first round diffusion image feature set may include: in the first round of iteration, inputting the second image feature z1 into the first processing module of the diffusion model P to obtain X in the first round diffusion image feature set a1 11 . X 11 Input the second processing module of the diffusion model P to obtain X in the first round of diffusion image feature set a1 12 . X 12 Input the third processing module of the diffusion model P to obtain X in the first round of diffusion image feature set a1 13 The first processing module, the second processing module and the third processing module in the diffusion model P are used to perform denoising on the second image features and perform feature extraction on the denoising results, thereby generating a first round of diffusion image feature set.

[0185] S2. According to the first round diffusion image feature set a1, the first text feature E, the first image feature q1, and the text-guided enhancement model B, the first round fusion text-image bimodal feature q1' is obtained.

[0186] The text-guided enhancement model B includes M processing modules. The scale of the output data of each processing module in the diffusion model P is consistent with the scale of the output data of each processing module in the text-guided enhancement model B. Each processing module in the text-guided enhancement model B may include at least one network layer, and the scale of the output data of each network layer in the same processing module is consistent.

[0187] In some examples, according to the first round diffusion image feature set a1, the first text feature E, the first image feature q1, and the text-guided enhancement model B, the process of obtaining the first round fusion text image bimodal feature q1' may include: first, inputting the first text feature E and the first image feature q1 into the first processing module of the text-guided enhancement model B, and obtaining the text image bimodal feature C corresponding to the first processing module in the first round 11 ; Then according to C 11 , the first round of diffusion image feature set a1 and the text-guided enhancement model B, and obtain the first round of fused text-image bimodal features q1'.

[0188] Among them, according to C 11 , the first round of diffusion image feature set a1 and the text-guided enhancement model B, the process of obtaining the first round of fusion text image bimodal feature q1' may include: first, C 1i-1 and X in the first round of diffusion image feature set a1 1i-1 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the i-1th processing module in the first round 1i '; Then, the fused text image bimodal feature C corresponding to the i-1th processing module 1i 'Input the i-th processing module of the text-guided enhancement model B, and obtain the text-image bimodal feature C corresponding to the i-th processing module in the first round 1i ; Finally, C 1i and X in the first round of diffusion image feature set a1 1i Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the i-th processing module in the first round 1i '; where i=2,3,…,m. When i=M, the fused text image bimodal feature C corresponding to the Mth processing module in the first round is 1i 'Input text to guide the output module of the enhancement model B and obtain the first round of fused text-image bimodal features q1'.

[0189] For example, when M=3, the first text feature E and the first image feature q1 are input into the first processing module of the text-guided enhancement model B, and the text-image bimodal feature C corresponding to the first processing module in the first round is output. 11 Then C 11 and X in the first round of diffusion image feature set a1 11 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the first processing module in the first round 11 '; The fused text image bimodal feature C corresponding to the first processing module in the first round 11 'Input the second processing module of the text-guided enhancement model B, and output the text-image bimodal feature C corresponding to the second processing module in the first round 12 ; C 12 and X in the first round of diffusion image feature set a1 12 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the second processing module in the first round 12 '. The fused text image bimodal feature C corresponding to the first processing module in the first round 12 'Input the third processing module of the text-guided enhancement model B, and output the text-image bimodal feature C corresponding to the third processing module in the first round 13 ; C 13 and X in the first round of diffusion image feature set a1 13 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the third processing module in the first round 13 ', the fused text image bimodal feature C corresponding to the third processing module in the first round 1i 'Input text to guide the output module of the enhancement model B and obtain the first round of fused text-image bimodal features q1'.

[0190] Exemplarily, the text image bimodal feature C satisfies the following expression:

[0191] c=B(q,E)

[0192] For example, C can be C 11 ; B is the text-guided enhancement model B; q can be the first image feature q1; E is the first text feature E.

[0193] By fusing the bimodal features of text and image with the features output by the diffusion model, the first image quality enhancement process not only considers the features of the first image itself, but also combines the features of the user's text for quality enhancement. This fusion method can make the quality enhancement more targeted, avoiding the blindness that may occur when relying solely on image features for processing, thereby significantly improving the quality enhancement effect.

[0194] S3. According to the first round of fusion text-image bimodal feature q1', the first text feature E, the first image feature q1, the text-guided enhancement model B and the diffusion model P, a first enhanced image coding feature is obtained.

[0195] According to the first round of fusion text image bimodal feature q1', the first text feature E, the first image feature q1 and the text guided enhancement model B and the diffusion model P, N-1 iterations are performed to obtain the first enhanced image coding feature.

[0196] In some examples, the process of performing N-1 iterations according to the first round of fusion text image bimodal feature q1', the first text feature E, the first image feature q, the text-guided enhancement model B and the diffusion model P to obtain the first enhanced image coding feature may include: in the j-th round of iterations, according to the j-1th round of text image bimodal feature q j-1 ' and diffusion model, determine the j-th round diffusion image feature set a j According to the j-th round diffusion image feature set, the first text feature E, the first image feature q and the text-guided enhancement model B, the j-th round fusion text-image bimodal feature q is obtained j '; where j is a positive integer, 2≤j≤N; when j=N, the jth round of fusion text image bimodal feature q j ' is the first enhanced image encoding feature.

[0197] In the iterative process, each round is further processed based on the fused text-image bimodal features of the previous round. As the number of iterations increases, the analysis and adjustment of image features will become more refined, and the defects in the image can be gradually removed, thus enhancing the image quality.

[0198] In some examples, during the j-th iteration, the text-image bimodal feature q j-1 ' and diffusion model, determine the j-th round diffusion image feature set a j The process may include: in the j-th round of iteration, the j-1-th fusion round text image bimodal feature q j-1 'Input diffusion model, diffusion model outputs the j-th round diffusion image feature set a j For example, the j-th round diffusion image feature set a j ={ X j1 ,…, X ji ,…, X jM}. Among them, X j1 is the feature output by the first processing module of the diffusion model in the jth iteration; X ji is the feature output by the i-th processing module of the diffusion model in the j-th iteration; X jMis the feature output by the Mth processing module of the diffusion model during the jth iteration, where i is a positive integer.

[0199] For example, taking the number of iterations N=2 and the number of processing modules M=3 as an example, after obtaining the first round of fused text image bimodal features q1', it can be considered that the first round of iterations is completed, and the second round of iterations is continued, that is, j=2. In the second round of iterations, the first round of fused text image bimodal features q1' are input into the diffusion model, and the diffusion model outputs the second round of diffusion image feature set a2. a2={X 21 , X 22 , X 23}. Among them, X 21 is the feature output by the first processing module of the diffusion model in the second round of iteration; X 22 is the feature output by the second processing module of the diffusion model in the second round of iteration; X 23 It is the feature output by the third processing module of the diffusion model in the second round of iteration.

[0200] In some examples, in combination with S404, during the j-th round of iterations, the diffusion model outputs the j-th round diffusion image feature set a j The process is called the reverse diffusion process. In the reverse diffusion process, the input data of the diffusion model is the text image bimodal feature q j-1 '(i.e. input data z j ), also includes the demand text and the second time series, so that the diffusion model can gradually remove the noise in the noise image based on the second time series during the reverse diffusion process to generate image features corresponding to the second image.

[0201] For example, in the embodiment of the present application, during the reverse diffusion process, the demand text can be set to empty. For example, during the jth iteration, the feature X output by the i-th processing module of the diffusion model ji Satisfies the following expression:

[0202] x ji = P(z j ,t,null)

[0203] Where P is the diffusion model P; z j is the input data of the diffusion model in the jth iteration; t is the second time series, and null means that the content corresponding to the demand text is empty.

[0204] According to the j-th round diffusion image feature set, the first text feature E, the first image feature q and the text-guided enhancement model B, the j-th round fusion text-image bimodal feature q is obtained. jThe process may include: first, inputting the first text feature E and the first image feature q into the first processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the first processing module in the jth round. j1 ; Then, the j-th round diffusion image feature set a j X in j1 The text-image bimodal feature C corresponding to the first processing module in round j j1 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the first processing module in the jth round j1 ';Finally, according to the fusion text image bimodal feature C corresponding to the first processing module in the jth round j1 ', the j-th round diffusion image feature set a j X in ji And the text-guided enhancement model B, we get the j-th round of fused text-image bimodal feature q j '.

[0205] For example, taking the number of iterations N=2 and the number of processing modules M=3 as an example, the first text feature E and the first image feature q are input into the first processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the first processing module in the second round. 21 ; Then, the X in the second round of diffusion image feature set a2 is 21 And the first processing module text image bimodal feature C 21 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the first processing module in the second round 21 ';Finally, according to the fusion text image bimodal feature C corresponding to the first processing module in the second round 21 ', the diffusion image feature X in the second round of diffusion image feature set 2i And the text-guided enhancement model B is used to obtain the second round of fused text-image bimodal features q2' (i.e., the first enhanced image encoding features).

[0206] According to the fused text-image bimodal feature C corresponding to the first processing module in the jth round j1 ', the diffusion image feature X in the j-th round diffusion image feature set ji And the text-guided enhancement model B, we get the j-th round of fused text-image bimodal feature q j The process may include: transforming the fused text image bimodal feature C corresponding to the i-1th processing module in the jth round j1 'Input the i-th processing module of the text-guided enhancement model B, and obtain the text-image bimodal feature C corresponding to the i-th processing module in the j-th round ji ; Then, the j-th round diffusion image feature set a jThe diffusion image feature X in ji and the text-image bimodal feature C corresponding to the i-th processing module in the j-th round ji Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the i-th processing module in the j-th round ji '. When i=M, the fused text image bimodal feature C corresponding to the Mth processing module jm 'Input text to guide the output module of the enhancement model B to obtain the j-th round of fused text-image bimodal features q j '.

[0207] For example, taking the number of iterations N = 2 and the number of processing modules M = 3 as an example, the fused text-image bimodal feature C corresponding to the first processing module in the second round is 21 'Input the second processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the second processing module in the second round 22 ; Then, the diffusion image feature X in the second round diffusion image feature set a2 is 22 The text-image bimodal feature C corresponding to the second processing module in the second round 22 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the second processing module in the second round 22 ', the fused text image bimodal feature C corresponding to the second processing module in the second round 22 'Input the third processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the third processing module in the second round 23 ; The diffusion image feature X in the second round diffusion image feature set a2 23 The text-image bimodal feature C corresponding to the third processing module in the second round 23 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the third processing module in the second round 23 ', finally, the fused text image bimodal feature C corresponding to the third processing module in the second round 23 'Input the output module of the text-guided enhancement model B to obtain the second round of fused text-image bimodal features q2' (i.e., the first enhanced image encoding features).

[0208] For example, in the jth iteration, the output data z of the diffusion model j Satisfies the following expression:

[0209] z j = B(q,E, f(x j-1m , C j-1m ))

[0210] Where B is the text-guided enhancement model B; q can be the first image feature q1; E is the first text feature E. f is the fusion function, x j-1m is the diffusion image feature x output by the mth processing module of the diffusion model in the j-1th round j-1m ; C j-1m is the text-image bimodal feature C corresponding to the mth processing module of the text-guided enhancement model B in the j-1th round j-1m .

[0211] In some examples, the above S1-S3 process is illustrated by taking the number of iterations of the diffusion model N=2, the number of processing modules in the diffusion model M=3, and the number of processing modules in the text-guided enhancement model B as an example. The scale of the output data of each processing module in the diffusion model P is consistent with the scale of the output data of each processing module in the text-guided enhancement model B.

[0212] The first iteration process of the diffusion model is: the second image feature Z1 is input into the first processing module of the diffusion model, and the first processing module of the diffusion model outputs X 11 . X 11 Input to the second processing module of the diffusion model, the second processing module of the diffusion model outputs X 12 . X 12 Input the third processing module of the diffusion model, and the third processing module of the diffusion model outputs X 13 .

[0213] Input the first text feature E and the first image feature q1 into the first processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the first processing module in the first round. 11 .

[0214] X 11 The text-image bimodal feature C corresponding to the first processing module in the first round 11 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the first processing module in the first round 11 '.

[0215] The fused text-image bimodal feature C corresponding to the first processing module in the first round 11 'Input the second processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the second processing module in the first round 12 .

[0216] X 12 The text-image bimodal feature C corresponding to the second processing module 12 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the second processing module in the first round12 '.

[0217] The fused text-image bimodal feature C corresponding to the second processing module in the first round 12 'Input the third processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the third processing module in the first round 13 .

[0218] X 13 The text-image bimodal feature C corresponding to the third processing module in the first round 13 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the third processing module in the first round 13 ', the fused text image bimodal feature C corresponding to the third processing module in the first round 13 'Input text to guide the output module of the enhancement model B and obtain the first round of fused text-image bimodal features q1'.

[0219] The second iteration process of the diffusion model is: the first round of fusion text image bimodal feature q1' is input into the first processing module of the diffusion model, and the first processing module of the diffusion model outputs X 21 . X 21 Input to the second processing module of the diffusion model, the second processing module of the diffusion model outputs X 22 . X 22 Input the third processing module of the diffusion model, and the third processing module of the diffusion model outputs X 23 .

[0220] Input the first text feature E and the first image feature q1 into the first processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the first processing module in the second round. 21 .

[0221] X 21 The text-image bimodal feature C corresponding to the first processing module in the second round 21 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the first processing module in the second round 21 '.

[0222] The fused text-image bimodal feature C corresponding to the first processing module in the second round 21 'Input the second processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the second processing module in the second round 22 .

[0223] X 22 The text-image bimodal feature C corresponding to the second processing module in the second round 22Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the second processing module in the second round 22 '.

[0224] The fused text-image bimodal feature C corresponding to the second processing module in the second round 22 'Input the third processing module of the text-guided enhancement model B to obtain the text-image bimodal feature C corresponding to the third processing module in the second round 23 .

[0225] X 23 And the text image bimodal feature C corresponding to the third processing module in the second round is obtained 23 Fusion is performed to obtain the fused text-image bimodal feature C corresponding to the third processing module 23 ', the fused text image bimodal feature C corresponding to the third processing module in the second round 23 'Input the output module of the text-guided enhancement model B to obtain the second round of fused text-image bimodal features q2' (the first enhanced image encoding features).

[0226] S406: Input the first enhanced image coding feature into an image decoder to obtain a second image.

[0227] The second image is an image obtained by enhancing the quality of the first image according to the first input data.

[0228] After obtaining the first enhanced image coding features through multiple iterations, the first enhanced image coding features can be input into an image decoder (also called a pre-trained image decoder), and the image decoder outputs the first enhanced image. The image encoder can be any one of Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), U-Net, and ResNets.

[0229] In some examples, in the embodiments of the present application, the image decoder used can be trained using any feasible training method, as long as the image decoder finally trained can generate a second image that matches the first enhanced image coding feature based on the first enhanced image coding feature.

[0230] In some instances, the first enhanced image coding features are input into an image decoder (also referred to as a pre-trained image decoder), and the process of the image decoder outputting the first enhanced image may include: first, the first enhanced image coding features are input into the image decoder; then, the image decoder performs a series of operations (for example, deconvolution operations, upsampling operations, or transposed convolution operations, etc.) on the input first enhanced image coding features, and gradually restores the dimensions of the first enhanced image coding features from a high-dimensional compressed state to a second image of the same size as the first image.

[0231] Exemplary, reference Figure 6 ,against Figure 6 For the first image shown in (a) in FIG. 1 , when the first input data is “remove raindrops from the image”, after the above steps S401 to S406, the following can be obtained: Figure 6 The second image shown in (c).

[0232] Based on the above scheme, after the electronic device receives the user's image quality enhancement trigger operation, the diffusion model and the text-guided enhancement model can be used to perform multiple iterations to generate a second image with enhanced image quality. In each iteration, when the diffusion model generates a diffusion image feature set (i.e., the Nth round of diffusion image feature set), the text-guided enhancement model can fuse the diffusion image feature set into the image quality enhancement requirement text features and the first image features. In this way, in each iteration, the image quality enhancement requirement text can be referenced to generate the second image. Thereby, the final second image meets the user's image quality enhancement requirements, and the implementation process is convenient and simple, and the user only needs to perform the image quality enhancement trigger operation.

[0233] It should be understood that each step in the above method embodiment provided by the present disclosure can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present disclosure can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0234] In one example, the unit in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0235] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system on a chip SOC.

[0236] In one implementation, the units of the above apparatus implementing the corresponding steps in the above method can be implemented in the form of a processing element scheduling program. For example, the apparatus may include a processing element and a storage element, and the processing element calls the program stored in the storage element to execute the method of the above method embodiment. The storage element may be a storage element on the same chip as the processing element, that is, an on-chip storage element.

[0237] In another implementation, the program for executing the above method may be in a storage element on a different chip from the processing element, i.e., an off-chip storage element. In this case, the processing element calls or loads the program from the off-chip storage element to the on-chip storage element to call and execute the method of the above method embodiment.

[0238] For example, the embodiments of the present disclosure may also provide a device, such as an electronic device, which may include a processor and a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the data processing method of the above embodiment. The memory may be located inside the electronic device or outside the electronic device. And the processor includes one or more.

[0239] In another implementation, the unit of the device implementing each step in the above method may be configured as one or more processing elements, which may be arranged on the corresponding electronic device, and the processing element here may be an integrated circuit, for example: one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of these integrated circuits. These integrated circuits may be integrated together to form a chip.

[0240] For example, the present disclosure also provides a chip, such as Fig. 9 As shown, the chip system includes at least one processor 901 and at least one interface circuit 902. The processor 901 and the interface circuit 902 can be interconnected through a line. For example, the interface circuit 902 can be used to receive signals from other devices. For another example, the interface circuit 902 can be used to send signals to other devices (such as the processor 901).

[0241] For example, the interface circuit 902 can read the instructions stored in the memory in the device and send the instructions to the processor 901. When the instructions are executed by the processor 901, the electronic device (such as Figure 2 The electronic device shown in FIG. 10 ) performs each step in the above embodiment. Of course, the chip system may also include other discrete devices, which is not specifically limited in the embodiment of the present disclosure.

[0242] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device can implement the above-mentioned data processing method.

[0243] The disclosed embodiments also provide a computer program product, including computer instructions for the electronic device to run as described above, and when the computer instructions are run in the electronic device, the electronic device can implement the data processing method as described above. Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0244] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0245] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0246] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0247] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, including a number of instructions to enable a terminal device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or optical disk and other media that can store program codes.

[0248] For example, the embodiments of the present disclosure may also provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the data processing method in the aforementioned method embodiment.

[0249] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Receiving a picture quality enhancement triggering operation for a first image; the picture quality enhancement triggering operation includes a picture quality enhancement requirement text; In response to the image quality enhancement trigger operation, a first round of fused text-image bimodal features is obtained according to the image quality enhancement requirement text features, the first image features, the second image features, the text-guided enhancement model and the diffusion model; wherein the image quality enhancement requirement text features are obtained by encoding the image quality enhancement requirement text; the first image features are obtained by encoding the first image; and the second image features are obtained by adding noise to the first image and then performing feature extraction on the noise-added result; According to the first round of fused text image bimodal features, the image quality enhancement required text features, the first image features, the text-guided enhancement model and the diffusion model, N-1 iterations are performed to obtain the first enhanced image coding features; wherein the diffusion model has the ability to perform denoising on the second image features and perform feature extraction on the denoising results to generate the first round diffusion image feature set; and according to the first round of fused text image bimodal features, j-1 iterations are performed to generate the j-th round diffusion image feature set; the text-guided enhancement model has the ability to fuse the image quality enhancement required text features, the first image features and the j-th round diffusion image feature set in each iteration process to generate the first enhanced image coding features; j is a positive integer, 2≤j≤N; N is a positive integer; The first enhanced image coding feature is decoded to obtain a second image; the second image is an image obtained by enhancing the image quality of the first image according to the image quality enhancement requirement text.

2. The method according to claim 1, characterized in that The step of obtaining a first round of fused text-image bimodal features according to the image quality enhancement requirement text features, the first image features, the second image features, the text-guided enhancement model and the diffusion model includes: Using the first processing module of the text-guided enhancement model to process the image quality enhancement required text features and the first image features, to obtain the text-image bimodal features corresponding to the first processing module in the first round; De-noising the second image features using the diffusion model, and extracting features from the denoising results to generate a first-round diffusion image feature set; the first-round diffusion image feature set includes M features; the diffusion model includes M processing modules; each processing module in the diffusion model corresponds to one feature; M is a positive integer; Fusing the text image bimodal feature corresponding to the first processing module in the first round and the first diffusion image feature to obtain a fused text image bimodal feature corresponding to the first processing module in the first round; the first diffusion image feature is a feature output by the first processing module of the diffusion model; The first round of fused text image bimodal features is obtained according to the fused text image bimodal features corresponding to the first processing module in the first round, other features in the first round of diffuse image feature set except the first diffuse image features, and the text-guided enhancement model.

3. The method according to claim 1, characterized in that The method of performing N-1 iterations according to the first round of fusion text-image bimodal features, the image quality enhancement requirement text features, the first image features, the text-guided enhancement model and the diffusion model to obtain the first enhanced image coding features includes: In the j-th round of iteration, the j-th round diffusion image feature set is determined according to the j-1-th round fusion text image bimodal features and the diffusion model; wherein j is a positive integer, 2≤j≤N; According to the j-th round diffusion image feature set, the image quality enhancement required text features, the first image features and the text-guided enhancement model, the j-th round fused text image bimodal features are obtained; wherein, when j=N, the j-th round fused text image bimodal features are the first enhanced image coding features.

4. The method according to claim 3, characterized in that The step of obtaining the j-round fused text-image bimodal feature according to the j-round diffusion image feature set, the image quality enhancement requirement text feature, the first image feature and the text-guided enhancement model includes: Using the first processing module of the text-guided enhancement model to process the image quality enhancement required text feature and the first image feature, to obtain the text-image bimodal feature corresponding to the first processing module in the jth round; Fusing the features output by the first processing module of the diffusion model in the j-th round diffusion image feature set with the text image bimodal features corresponding to the first processing module in the j-th round to obtain the fused text image bimodal features corresponding to the first processing module in the j-th round; The j-round fused text image bimodal features are obtained according to the fused text image bimodal features corresponding to the first processing module in the j-round, the features in the j-round diffusion image feature set except the features output by the first processing module of the diffusion model, and the text-guided enhancement model.

5. The method according to claim 4, characterized in that The step of obtaining the j-th round fused text image bimodal features according to the fused text image bimodal features corresponding to the first processing module in the j-th round, the features in the j-th round diffusion image feature set except the features output by the first processing module of the diffusion model, and the text guided enhancement model comprises: Inputting the fused text image bimodal features corresponding to the i-1th processing module in the jth round into the i-th processing module of the text-guided enhancement model to obtain the text image bimodal features corresponding to the i-th processing module in the jth round; wherein the text-guided enhancement model includes M processing modules, i is a positive integer, 2≤i≤M; The features output by the i-th processing module of the diffusion model in the j-th round diffusion image feature set and the text image bimodal features corresponding to the i-th processing module in the j-th round are fused to obtain the fused text image bimodal features corresponding to the i-th processing module in the j-th round; when i=M, the fused text image bimodal features corresponding to the M-th processing module in the j-th round are input into the output module of the text-guided enhancement model to obtain the j-th round fused text image bimodal features.

6. The method according to any one of claims 1 to 5, characterized in that The picture quality enhancement trigger operation includes picture quality enhancement requirement audio; Before responding to the image quality enhancement triggering operation, the method further includes: In a case where the picture quality enhancement triggering operation includes the picture quality enhancement requirement audio, the picture quality enhancement requirement audio is identified to obtain the picture quality enhancement requirement text.

7. The method according to any one of claims 1 to 5, characterized in that The diffusion model is a pre-trained implicit diffusion model, and the text-guided enhancement model is a u-net model.

8. The method according to any one of claims 1 to 5, characterized in that The requirements for better image quality are to reduce noise, reduce blur and improve resolution.

9. An electronic device, characterized in that: include: A touch screen, which includes a touch sensor and a display screen; one or more processors; Memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the electronic device, the electronic device executes the image processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having computer program instructions stored thereon; characterized in that: When the computer program instructions are executed by an electronic device, the electronic device implements the image processing method according to any one of claims 1 to 8.

Images