Image processing methods and training methods for stable diffusion models of graph-generated images

Abstract

This application relates to the field of image processing technology, providing an image processing method and a training method for a stable diffusion model of an image-generated image. The stable diffusion model of the image-generated image includes an image cross-attention mechanism layer. The method includes: inputting a first image with a first illumination effect and a target ambient light image into the trained stable diffusion model of the image-generated image; the stable diffusion model of the image-generated image performs cross-processing between the first image and the target ambient light image through the image cross-attention mechanism layer, enabling the stable diffusion model of the image-generated image to re-illuminate the first image under the guidance of the target ambient light image, outputting a second image with a second illumination effect. This achieves the reconstruction of the image illumination effect, improves the image illumination effect, and enhances image quality and image effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and a method for training a stable diffusion model of an image-generated image. Background Technology

[0002] With the development of electronic device technology, the shooting technology of electronic devices (such as mobile phones, tablets, cameras, etc.) has become increasingly sophisticated. As a result, users are taking pictures with electronic devices more and more frequently.

[0003] However, due to the different shooting angles of different users and the limitations of the environment in which the user was shooting, the lighting of the images taken by the user may be relatively poor, thereby reducing the image effect and image quality. Summary of the Invention

[0004] This application provides an image processing method and a stable diffusion model training method for image-generated images, which are used to reconstruct the lighting effects of images, so that the images achieve better lighting effects, thereby improving image effects and image quality.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, an image processing method is provided for application in an electronic device, including a display screen; simultaneously, a trained, stable diffusion model for generated images is deployed within the electronic device. The method includes:

[0007] The system displays a first image with a first illumination effect on a first interface; receives a first operation from a user on the first interface; wherein the first operation is used to trigger an electronic device to relight the image; in response to the first operation, outputs a second image with a second illumination effect based on the first image and the target ambient light image using a stable diffusion model of the image-generated image; and displays the second image with the second illumination effect on the first interface.

[0008] The stable diffusion model of the image-generated image includes an image cross-attention mechanism layer. This layer performs cross-processing between the first image and the target ambient light image, allowing the model to re-illuminate the first image under the influence of the target ambient light image, resulting in a second image. The second image, after re-illumination, corresponds to the illumination effect of the target ambient light image; in other words, the second illumination effect corresponds to the illumination effect of the target ambient light image.

[0009] Therefore, this method can relight the first image by using the stable diffusion model of the image-generated image constructed by the stable diffusion technique, thereby improving the lighting effect of the first image and thus improving the image quality and image effect of the first image.

[0010] Meanwhile, the relighting of the first image is completed by the stable diffusion model of the image-generated image under the guidance of the target ambient light image. The target ambient light image can generally be a pre-configured image with good lighting effects, or an image given by the user that meets the user's lighting effect requirements. Therefore, the stable diffusion model of the image-generated image, based on the ambient light image, not only further improves the lighting effect of the first image, enhancing its image quality and overall image effect, but also produces an image that satisfies the user, ensuring a better user experience.

[0011] Additionally, since stable diffusion is a technique commonly used in AI painting, it inherently handles details such as faces (e.g., human faces) quite finely. Therefore, even if the first aspect doesn't additionally guide the stable diffusion model of the raw image to perform fine processing on the face, if the first image input to the stable diffusion model of the raw image includes a face (e.g., human face), then the output second image will not only have improved lighting effects, but the face may also be clearer than the input first image.

[0012] In one possible implementation of the first aspect, the image processing method can be applied to image post-processing scenarios.

[0013] Based on this, the first interface can be a large image display interface in the image library of an electronic device, and the first image can be an image from the image library of the electronic device. Then, the second image is obtained by relighting the first image. Therefore, the lighting effects of the images stored in the image library can be reconstructed later, thereby improving the lighting effects of the images in the image library.

[0014] In one possible implementation of the first aspect, the image processing method can also be applied to an image capture scenario. In this case, the first interface can also be a camera preview interface in an electronic device, where the first image is a real-time captured image from the camera, and the second image is obtained by relighting the real-time captured image.

[0015] Based on this, the image processing method may further include: saving a second image in response to a user's shooting operation on the first interface; wherein the shooting operation follows the first operation. Thus, users can directly save images with good lighting effects by shooting, ensuring a better shooting experience.

[0016] In one possible implementation of the first aspect, similar to the image capture scenario, the image processing method can also be applied to the image recording scenario. Therefore, the first interface can also be the recording interface of a camera in an electronic device, and correspondingly, the first image can be an image captured in real-time by the camera, and the second image is obtained by relighting the image captured in real-time by the camera; based on this, the image processing method may further include:

[0017] In response to the user's "Start Recording" operation on the first screen, multiple second images are captured consecutively; in response to the user's "Stop Recording" operation on the first screen, the multiple second images are saved as a video; the "Start Recording" operation follows the first operation. Therefore, users can directly save videos with good lighting effects, ensuring a better recording experience.

[0018] In one possible implementation of the first aspect, the electronic device may include multiple candidate ambient light images. The target ambient light image can then be any one or more of these candidate ambient light images.

[0019] Based on this, the image processing method may further include: displaying multiple candidate ambient light images on a first interface; wherein the candidate ambient light images include pre-configured ambient light images and / or user-specified ambient light images;

[0020] Furthermore, in response to the first operation, the second image, based on the first image and the target ambient light image, outputs a second image with a second illumination effect using a stable diffusion model derived from the image, which may include:

[0021] The system receives a first operation and uses the candidate ambient light image corresponding to the first operation as the target ambient light image. Based on the stable diffusion model of the image-generated image, it outputs a second image with a second lighting effect. Thus, the electronic device can select a candidate ambient light image as the target ambient light image based on the user's operation, and relight the device according to the selected candidate ambient light image, thereby achieving a lighting effect that is as satisfactory as possible to the user and ensuring a good user experience.

[0022] In one possible implementation of the first aspect, since both the first image and the target ambient light image need to be processed, the stable diffusion model of the image-generated image in this implementation may include two parallel branches that process these two images. These two parallel branches perform relevant processing on the first image and the target ambient light image respectively.

[0023] That is, in this implementation, the stable diffusion model of the graph-generated graph may include two parallel first stable diffusion branches and second stable diffusion branches; the first stable diffusion branch and the second stable diffusion branch include an image cross-attention mechanism layer, and the first stable diffusion branch and the second stable diffusion branch are interconnected through the image cross-attention mechanism layer.

[0024] Based on this, the stable diffusion model of the image-generated image, which outputs the second image of the second illumination effect based on the first image and the target ambient light image, may include:

[0025] A first image is input into a first stable diffusion branch, and a target ambient light image is input into a second stable diffusion branch. The first stable diffusion branch obtains image information of the target ambient light image from the second stable diffusion branch through an image cross-attention mechanism layer, enabling the first stable diffusion branch to output a second image under the guidance of the target ambient light image. The image information includes lighting information. Therefore, the electronic device can reconstruct the lighting effect of the image under the guidance of the target ambient light image, improving image quality and image effect, and ensuring a better user experience.

[0026] In one possible implementation of the first aspect, the stable diffusion model of the graph-generated image can be processed iteratively by multiple U-net units to improve the image generation effect of the model.

[0027] Therefore, in this implementation, the first stable diffusion branch includes a first U-net branch, and the second stable diffusion branch includes a second U-net branch; wherein, the first U-net branch includes n cascaded first U-net units, and the second U-net branch includes n unconnected second U-net units; all n cascaded first U-net units include an image cross-attention mechanism, and all n unconnected second U-net units also include an image cross-attention mechanism layer; furthermore, these n cascaded first U-net units and n unconnected second U-net units are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1; based on this, inputting the first image into the first stable diffusion branch and inputting the target ambient light image into the second stable diffusion branch, the first stable diffusion branch obtains the image information of the target ambient light image from the second stable diffusion branch through the image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the target ambient light image, which may include:

[0028] The first image is input into the first stable diffusion branch, and the first stable diffusion branch performs image encoding and noise addition on the first image to obtain the corresponding noisy first potential image; the target ambient light image is input into the second stable diffusion branch, and the second stable diffusion branch performs image encoding and noise addition on the target ambient light image to obtain the corresponding noisy target potential image.

[0029] Then, the noisy target latent image is transmitted to n second U-net units respectively. The n cascaded first U-net units in the first U-net branch obtain the noisy target latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer. The noisy first latent image is transmitted to the first U-net branch. The first U-net unit in the first U-net branch can perform cross-processing on the noisy first latent image and the noisy target latent image, so that the noisy first latent image is fused with the image information of the noisy target latent image, and then the noise of the noisy first latent image is predicted. The image information includes illumination information.

[0030] Therefore, the first U-net branch can output a first predicted noise image after n iterations of cross-processing and noise prediction by n cascaded first U-net units; finally, the noisy first latent image is denoised based on the first predicted noise image, and the denoised noisy first latent image is image decoded to output a second image.

[0031] In one possible implementation of the first aspect, the second stable diffusion branch performs image encoding and noise addition on the target ambient light image to obtain a corresponding noisy target latent image, including: performing image encoding on the target ambient light image to obtain a target latent image, and adding noise to the target latent image n times with different intensities to obtain n noisy target latent images.

[0032] Then, the noisy target latent image is transmitted to each of the second U-net units. The n cascaded first U-net units in the first U-net branch obtain the noisy target latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer, which may include:

[0033] The n noisy target potential images are transmitted one-to-one to the n unconnected second U-net units. The n cascaded first U-net units in the first U-net branch obtain the n noisy target potential images from the corresponding connected second U-net units through the image cross-attention mechanism layer.

[0034] In this context, according to the cascaded connection order, the noise intensity of the noisy target potential image obtained by the n cascaded first U-net units is distributed from strong to weak. The noise intensity of the noisy target potential image obtained by the first first U-net unit is the strongest, and the noise intensity of the noisy target potential image obtained by the last first U-net unit is the weakest.

[0035] Therefore, the target ambient light image is noise-added with different intensities in descending order of noise strength. Then, each noise-added target ambient light image is input to n disconnected second U-net units in descending order of noise strength. This allows the first U-net branch to shift and reconstruct the lighting effect of the first image of the first lighting effect based on the target ambient light images with different noise levels, thereby accurately outputting the second image of the second lighting effect under the guidance of the target ambient light image.

[0036] In one possible implementation of the first aspect, the image can be encoded using an image encoder, and the first image and the target ambient light image can use different image encoders. Therefore, the first stable diffusion branch may further include a first image encoder, a first noise-adding module, a denoising module, and an image decoder; wherein the first image encoder is used to encode the first image to obtain a first latent image; the first noise-adding module is used to add noise to the first latent image to obtain a noisy first latent image; the denoising module is used to denoise the noisy first latent image based on a first predicted noise image; and the image decoder is used to decode the denoised noisy first latent image to output a second image.

[0037] In one possible implementation of the first aspect, the second stable diffusion branch may further include a second image encoder and a second noise-adding module; wherein the second image encoder is used to perform image encoding on the target ambient light image to obtain a target latent image; and the second noise-adding module is used to add noise to the target latent image to obtain a noisy target latent image.

[0038] In one possible implementation of the first aspect, the image cross-attention mechanism layer includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer, a first attention mechanism layer, and a second attention mechanism layer.

[0039] The input to the first linear layer is the output of the residual network layer in the first U-net unit, the input to the second linear layer is the output of the residual network layer in the second U-net unit, the input to the third linear layer is the output of the first attention mechanism layer, the input to the fourth linear layer is the output of the second attention mechanism layer, and the inputs to the first and second attention mechanism layers are the outputs of the first and second linear layers.

[0040] In one possible implementation of the first aspect, since different first images can include different types of objects (e.g., faces, animals, plants, etc.), and the lighting requirements may differ for different types of objects, this implementation can add a ControlNet to provide additional control over the reconstruction of the image-to-image SD model in order to specifically reconstruct lighting effects that better suit the object.

[0041] Therefore, the stable diffusion model of graph-generated graphs can also include ControlNet. The output of ControlNet is transmitted to the cascaded layers in the U-net units; in this implementation, the U-net units include n first U-net units connected in series and / or n second U-net units that are not connected.

[0042] In one possible implementation of the first aspect, denoising the noisy first latent image based on the first predicted noise image, and performing image decoding on the denoised noisy first latent image to output a second image, may include:

[0043] The difference between two corresponding pixel values ​​in the noisy first latent image and the first predicted noisy image is calculated to obtain the denoised first latent image; the denoised first latent image is then decoded to output the second image.

[0044] In one possible implementation of the first aspect, the target ambient light image includes i target ambient light images; the stable diffusion model of the graph includes i+1 parallel stable diffusion branches; wherein, the i+1 parallel stable diffusion branches include a first stable diffusion branch and i second stable diffusion branches; both the first stable diffusion branch and the i second stable diffusion branches include an image cross-attention mechanism layer and are interconnected through the image cross-attention mechanism layer; i is an integer, i≥1;

[0045] A stable diffusion model derived from the image-generated image outputs a second image with a second illumination effect based on a first image and a target ambient light image, including:

[0046] The first image is input into the first stable diffusion branch, and i target ambient light images are input one-to-one into i second stable diffusion branches. The first stable diffusion branch obtains image information from the i target ambient light images from the i second stable diffusion branches through an image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the i target ambient light images. The image information includes illumination information. Therefore, the image-generated image stable diffusion model can reconstruct the illumination effect of an image under the guidance of multiple ambient light images.

[0047] Secondly, a training method for a stable diffusion model of graph-generated graphs is provided, wherein the stable diffusion model of graph-generated graphs includes a U-net structure; the U-net structure includes an image cross-attention mechanism layer;

[0048] The training method includes:

[0049] Obtain the first training image and the training ambient light image; initialize the weights of the U-net structure;

[0050] The first training image and the training ambient light image are input into the stable diffusion model of the image-generated image. The stable diffusion model of the image-generated image outputs the second training image based on the first training image and the training ambient light image. The stable diffusion model of the image-generated image performs cross-processing between the first training image and the training ambient light image through an image cross-attention mechanism layer. This allows the stable diffusion model of the image-generated image to relight the first training image under the guidance of the training ambient light image, thus obtaining the second training image.

[0051] By training on the first image, the ambient light image, and the second image, a stable diffusion model for the image-generated image is determined. The loss is applied to the relighting using the ambient light image during training. The weights of the U-net structure are adjusted based on the loss until the training termination condition is met, thus obtaining a well-trained stable diffusion model for the image-generated image.

[0052] In one possible implementation of the second aspect, the U-net structure includes a first U-net branch and a second U-net branch; the first U-net branch includes n serially connected first U-net units, and the second U-net branch includes n unconnected second U-net units; wherein, the n serially connected first U-net units all include an image cross-attention mechanism, and the n unconnected second U-net units also all include an image cross-attention mechanism layer; and the n serially connected first U-net units and the n unconnected second U-net units are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1.

[0053] Based on this, the first training image and the training ambient light image are input into the stable diffusion model of the image-generated image. The stable diffusion model of the image-generated image outputs the second training image based on the first training image and the training ambient light image, including:

[0054] The first training image is image encoded and noise-added to obtain the corresponding training noisy first latent image; and the training ambient light image is image encoded and noise-added to obtain the corresponding training noisy ambient light latent image.

[0055] The training noisy ambient light latent image is transmitted to n second U-net units respectively. The n cascaded first U-net units in the first U-net branch obtain the training noisy ambient light latent image from the corresponding connected second U-net unit through the image cross attention mechanism layer.

[0056] The training noisy first latent image is transmitted to the first U-net branch. The first U-net unit in the first U-net branch performs cross-processing on the training noisy first latent image and the training noisy ambient light latent image, so that the training noisy first latent image is fused with the image information of the training noisy ambient light latent image, and the noise of the training noisy first latent image is predicted; wherein, the image information includes illumination information.

[0057] The first U-net branch iteratively performs n cross-processing and noise prediction through n cascaded first U-net units, and outputs a training first predicted noise image; based on the training first predicted noise image, the training noisy first latent image is denoised, and the denoised training noisy first latent image is image decoded to output a training second image.

[0058] In one possible implementation of the second aspect, the stable diffusion model of the graph-generated image further includes an image encoder and an image decoder. The image encoder is used to encode the images (including a first training image and a training ambient light image). The image decoder is used to decode the denoised, noisy first latent training image.

[0059] Therefore, the training method further includes: determining and fixing the weights of the image encoder, and determining and fixing the weights of the image decoder; wherein the weights of the image encoder and / or the image decoder are open-source weights; or, the weights of the image encoder and / or the image decoder are determined through training. Open-source weights refer to the open-source weights of the image encoder and the open-source weights of the image decoder.

[0060] In one possible implementation of the second aspect, the image encoder includes a first image encoder and a second image encoder; the first image encoder is used to perform image encoding on a training first image; the second image encoder is used to perform image encoding on a training ambient light image; then determining and fixing the weights of the image encoders may include: determining and fixing the weights of the first image encoder, and determining and fixing the weights of the second image encoder; wherein the weights of the first image encoder and the weights of the second image encoder are the same or different.

[0061] In one possible implementation of the second aspect, initializing the weights of the U-net structure includes: obtaining the open-source weights corresponding to the U-net structure in the open-source stable diffusion model, and initializing the weights of the U-net structure according to the open-source weights.

[0062] It should be noted that the model structure of the stable diffusion model for graph-generated images in the second aspect, as well as the beneficial effects of the training method provided in the second aspect, can be found in the description of the first aspect above, and will not be repeated here. For example, the stable diffusion model for graph-generated images in the second aspect may also include a first noise-adding module, a second noise-adding module, and a noise-reducing module.

[0063] Thirdly, this application provides an electronic device, including: a display screen, a camera, one or more processors, and a memory; the display screen, camera, and memory are coupled to the processors; the display screen is used to display images, and the camera is used to capture images; the memory stores one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, the electronic device performs the following steps:

[0064] A first image with a first illumination effect is displayed on a first interface; a first operation by a user on the first interface is received; wherein the first operation is used to trigger an electronic device to relight the image; in response to the first operation, a second image with a second illumination effect is output based on the first image and the target ambient light image by a stable diffusion model of the image-generated image; wherein the stable diffusion model of the image-generated image includes an image cross-attention mechanism layer; the stable diffusion model of the image-generated image performs cross-processing between the first image and the target ambient light image through the image cross-attention mechanism layer, so that the stable diffusion model of the image-generated image relights the first image under the guidance of the target ambient light image to obtain the second image; wherein the second illumination effect corresponds to the illumination effect of the target ambient light image; the second image with the second illumination effect is displayed on the first interface.

[0065] In one possible implementation of the third aspect, the first interface is a large image display interface of the image library in the electronic device, and the first image is an image in the image library of the electronic device; therefore, when the above computer instructions are executed by the processor, the electronic device also performs the following steps: the second image is obtained by relighting the first image.

[0066] In one possible implementation of the third aspect, the first interface is a camera preview interface in the electronic device, the first image is an image captured in real time by the camera, and the second image is obtained by relighting the image captured in real time by the camera; therefore, when the above computer instructions are executed by the processor, the electronic device also performs the following steps: in response to the user's shooting operation on the first interface, save the second image; wherein the shooting operation is after the first operation.

[0067] In one possible implementation of the third aspect, the first interface is the recording interface of the camera in the electronic device, the first image is an image captured in real time by the camera, and the second image is obtained by relighting the image captured in real time by the camera; therefore, when the above computer instructions are executed by the processor, the electronic device also performs the following steps:

[0068] In response to the user's start recording operation on the first interface, multiple second images are continuously captured; in response to the user's end recording operation on the first interface, the multiple second images are saved as a video; wherein, the start recording operation is performed after the first operation.

[0069] In one possible implementation of the third aspect, when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: displaying a plurality of candidate ambient light images on a first interface; wherein the candidate ambient light images include a pre-configured ambient light image and / or a user-specified ambient light image; receiving a first operation and using the candidate ambient light image corresponding to the first operation as the target ambient light image; and outputting a second image of a second illumination effect based on the first image and the target ambient light image using a stable diffusion model of the graph.

[0070] In one possible implementation of the third aspect, the stable diffusion model of the graph includes two parallel first stable diffusion branches and a second stable diffusion branch; the first and second stable diffusion branches include an image cross-attention mechanism layer and are interconnected through the image cross-attention mechanism layer; therefore, when the above computer instructions are executed by the processor, the electronic device further performs the following steps:

[0071] The first image is input into the first stable diffusion branch, and the target ambient light image is input into the second stable diffusion branch. The first stable diffusion branch obtains the image information of the target ambient light image from the second stable diffusion branch through the image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the target ambient light image; wherein, the image information includes illumination information.

[0072] In one possible implementation of the third aspect, the first stable diffusion branch includes a first U-net branch, and the second stable diffusion branch includes a second U-net branch; wherein, the first U-net branch includes n cascaded first U-net units, and the second U-net branch includes n unconnected second U-net units; the n cascaded first U-net units all include an image cross-attention mechanism, and the n unconnected second U-net units also all include an image cross-attention mechanism layer, and the n cascaded first U-net units and the n unconnected second U-net units are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1; therefore, when the above computer instructions are executed by the processor, the electronic device further performs the following steps:

[0073] The first image is input into the first stable diffusion branch, and the first stable diffusion branch performs image encoding and noise addition on the first image to obtain the corresponding noisy first potential image; the target ambient light image is input into the second stable diffusion branch, and the second stable diffusion branch performs image encoding and noise addition on the target ambient light image to obtain the corresponding noisy target potential image.

[0074] The noisy target latent image is transmitted to n second U-net units respectively. The n cascaded first U-net units in the first U-net branch obtain the noisy target latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer respectively.

[0075] The noisy first latent image is transmitted to the first U-net branch. The first U-net unit in the first U-net branch performs cross-processing on the noisy first latent image and the noisy target latent image, so that the noisy first latent image is fused with the image information of the noisy target latent image, and then the noise of the noisy first latent image is predicted; wherein, the image information includes illumination information.

[0076] The first U-net branch outputs a first predicted noise image after iteratively performing n cross-processing and noise prediction on n cascaded first U-net units.

[0077] The first noisy first potential image is denoised based on the first predicted noisy image, and the denoised first noisy first potential image is then image decoded to output the second image.

[0078] In one possible implementation of the third aspect, when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: image encoding of the target ambient light image to obtain a target latent image, and adding noise to the target latent image n times with different intensities to obtain n noisy target latent images; transmitting the n noisy target latent images one-to-one to n disconnected second U-net units, wherein the n cascaded first U-net units in the first U-net branch respectively obtain the n noisy target latent images from the corresponding connected second U-net units through the image cross-attention mechanism layer; wherein, according to the cascaded connection order, the noise intensity of the noisy target latent images obtained by the n cascaded first U-net units is distributed from strong to weak, with the noise intensity of the noisy target latent image obtained by the first first U-net unit being the strongest, and the noise intensity of the noisy target latent image obtained by the last first U-net unit being the weakest.

[0079] In one possible implementation of the third aspect, the first stable diffusion branch further includes a first image encoder, a first noise-adding module, a noise-reducing module, and an image decoder; when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: image encoding the first image by the first image encoder to obtain a first latent image; adding noise to the first latent image by the first noise-adding module to obtain a noisy first latent image; denoising the noisy first latent image by the noise-reducing module based on the first predicted noise image; and image decoding the denoised noisy first latent image by the image decoder to output a second image.

[0080] In one possible implementation of the third aspect, the second stable diffusion branch further includes a second image encoder and a second noise-adding module; when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: image encoding of the target ambient light image by the second image encoder to obtain a target latent image; and adding noise to the target latent image by the second noise-adding module to obtain a noisy target latent image.

[0081] In one possible implementation of the third aspect, when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: calculating the difference between two corresponding pixel values ​​in the noisy first latent image and the first predicted noise image to obtain a denoised first latent image; and performing image decoding on the denoised first latent image to output a second image.

[0082] In one possible implementation of the third aspect, the target ambient light image includes i target ambient light images; the stable diffusion model of the image-generated image includes i+1 parallel stable diffusion branches; wherein, the i+1 parallel stable diffusion branches include a first stable diffusion branch and i second stable diffusion branches; both the first stable diffusion branch and the i second stable diffusion branches include an image cross-attention mechanism layer and are interconnected through the image cross-attention mechanism layer; i is an integer, i≥1; when the above computer instructions are executed by the processor, the electronic device further performs the following steps:

[0083] The first image is input into the first stable diffusion branch, and i target ambient light images are input one-to-one into i second stable diffusion branches. The first stable diffusion branch obtains image information of the i target ambient light images from the i second stable diffusion branches through an image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the i target ambient light images. The image information includes illumination information.

[0084] Fourthly, this application provides an electronic device, comprising: one or more processors and a memory, the memory being coupled to the processor; the memory storing one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, the electronic device performs the following steps:

[0085] Acquire the first training image and the training ambient light image; and initialize the weights of the U-net structure;

[0086] The first training image and the training ambient light image are input into the stable diffusion model of the image-generated image. The stable diffusion model of the image-generated image outputs the second training image based on the first training image and the training ambient light image. The stable diffusion model of the image-generated image performs cross-processing between the first training image and the training ambient light image through an image cross-attention mechanism layer. This allows the stable diffusion model of the image-generated image to relight the first training image under the guidance of the training ambient light image, thus obtaining the second training image.

[0087] By training on the first image, the ambient light image, and the second image, a stable diffusion model for the image-generated image is determined. The loss is applied to the relighting using the ambient light image during training. The weights of the U-net structure are adjusted based on the loss until the training termination condition is met, thus obtaining a well-trained stable diffusion model for the image-generated image.

[0088] In one possible implementation of the fourth aspect, the U-net structure includes a first U-net branch and a second U-net branch; the first U-net branch includes n serially connected first U-net units, and the second U-net branch includes n unconnected second U-net units; wherein, each of the n serially connected first U-net units includes an image cross-attention mechanism, and each of the n unconnected second U-net units also includes an image cross-attention mechanism layer; furthermore, the n serially connected first U-net units and the n unconnected second U-net units are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1; therefore, when the above computer instructions are executed by the processor, the electronic device further performs the following steps:

[0089] The training first image is image encoded and noise-added to obtain the corresponding training noisy first latent image; and the training ambient light image is image encoded and noise-added to obtain the corresponding training noisy ambient light latent image; the training noisy ambient light latent images are respectively transmitted to n second U-net units, and the n cascaded first U-net units in the first U-net branch respectively obtain the training noisy ambient light latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer; the training noisy first latent image is transmitted to the first U-net branch, and the first U-net in the first U-net branch... The unit performs cross-processing on the training noisy first latent image and the training noisy ambient light latent image, so that the training noisy first latent image is fused with the image information of the training noisy ambient light latent image, and then predicts the noise of the training noisy first latent image; wherein, the image information includes illumination information; the first U-net branch performs cross-processing and noise prediction n times through n cascaded first U-net units, and outputs the training first predicted noise image; the training noisy first latent image is denoised based on the training first predicted noise image, and the denoised training noisy first latent image is image decoded to output the training second image.

[0090] In one possible implementation of the fourth aspect, the stable diffusion model of the image-generated image further includes an image encoder and an image decoder; wherein the image encoder includes a first image encoder and a second image encoder; the first image encoder is used to encode a training first image; the image decoder is used to decode a denoised training noisy first latent image; and the second image encoder is used to encode a training ambient light image. Therefore, when the above computer instructions are executed by the processor, the electronic device further performs the following steps:

[0091] The weights of the image encoder are determined and fixed, and the weights of the image decoder are determined and fixed; wherein the weights of the image encoder and / or the image decoder are open-source weights; or, the weights of the image encoder and / or the image decoder are determined through training; wherein the weights of the first image encoder and the second image encoder are the same or different.

[0092] In one possible implementation of the fourth aspect, when the aforementioned computer instructions are executed by the processor, the electronic device further performs the following steps: obtaining the open-source weights corresponding to the U-net structure in the open-source stable diffusion model, and initializing the weights of the U-net structure according to the open-source weights.

[0093] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor in an electronic device, causes the electronic device to perform an image processing method as described in the first aspect and any possible implementation thereof, and / or a training method for a stable diffusion model of a graph-generated image as described in the second aspect and any possible implementation thereof.

[0094] Sixthly, this application provides a computer program product that, when run on a computer, causes the computer to execute the method of the first aspect and any possible implementation thereof, and / or the training method of the stable diffusion model of the graph-generated graph of the second aspect and any possible implementation thereof. The computer may be the aforementioned electronic device.

[0095] Understandably, the beneficial effects that can be achieved by the electronic device of any possible implementation of the third aspect, the electronic device of any possible implementation of the fourth aspect, the computer-readable storage medium of the fifth aspect, and the computer program product of the sixth aspect can be referred to as the beneficial effects in the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0096] Figure 1 A flowchart illustrating the processing of a stable diffusion model for graph-generated images, provided in an embodiment of this application;

[0097] Figure 2 A flowchart illustrating the processing of another stable diffusion model for graph-generated images provided in this application embodiment;

[0098] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0099] Figure 4 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0100] Figure 5 and Figure 6This is a schematic diagram of a scene for relighting in image post-processing, provided as an embodiment of this application.

[0101] Figure 7 This is a schematic diagram illustrating another scenario of relighting in image post-processing, provided as an embodiment of this application.

[0102] Figure 8 This application provides a schematic diagram of a scene where relighting is applied during image capture, as illustrated in an embodiment of the present application. Figure 1 ;

[0103] Figure 9 This application provides a schematic diagram of a scene where relighting is applied during image capture, as illustrated in an embodiment of the present application. Figure 2 ;

[0104] Figure 10 This application provides a schematic diagram of a scene where relighting is applied during image capture, as illustrated in an embodiment of the present application. Figure 3 ;

[0105] Figure 11 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0106] Figure 12 A schematic diagram of an interface for a candidate ambient light image provided in an embodiment of this application;

[0107] Figure 13 A schematic diagram of an interface for another candidate ambient light image provided in an embodiment of this application;

[0108] Figure 14 This is a schematic diagram of a conventional stable diffusion model provided in an embodiment of this application;

[0109] Figure 15 This is a schematic diagram of the structure of a U-net in a conventional stable diffusion model provided in an embodiment of this application;

[0110] Figure 16 A schematic diagram of the structure of a graph-based SD model provided in this application embodiment.

[0111] Figure 17 This application provides a schematic diagram of the structure of a U-net unit in a graph-based SD model.

[0112] Figure 18 This application provides a schematic diagram of the structure of a cross two-image attention layer according to an embodiment of the present application.

[0113] Figure 19 A flowchart illustrating another image processing method provided in this application embodiment;

[0114] Figure 20 A schematic diagram illustrating the connection between ControlNet and a graph-based SD model provided in this application embodiment;

[0115] Figure 21 This is a structural block diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0116] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., are not necessarily different. Also, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0117] With the development of electronic device technology, the shooting technology of electronic devices (such as mobile phones, tablets, cameras, etc.) has become increasingly sophisticated. As a result, users are taking pictures with electronic devices more and more frequently.

[0118] However, due to differences in the user's shooting angle and the limitations of the environment at the time of shooting, the lighting effect of the user's captured image may be relatively poor, thereby reducing the image effect and image quality.

[0119] Based on this, embodiments of this application provide an image processing method. This image processing method, applied to electronic devices, can improve the lighting effect of images, thereby enhancing image quality and producing images that satisfy the user, thus ensuring a better user experience.

[0120] In summary, the image processing method provided in this application embodiment is implemented by relighting an image using an electronic device. Relighting is an image illumination technique, which can be simply understood as a technique used to supplement the light in an image. In other words, this application embodiment improves the lighting effect of an image by relighting it using an electronic device, thereby improving the image effect and quality, ultimately resulting in a satisfactory image for the user and ensuring a better user experience.

[0121] Specifically, this application embodiment relights the image based on diffusion technology. The main difference lies in the fact that this application embodiment constructs a stable diffusion model for the image-to-image model using stable diffusion (SD) model technology. Then, this application embodiment relights the image based on this constructed stable diffusion model for the image-to-image model.

[0122] First, after the stable diffusion model of the raw image is constructed, the electronic device trains the constructed stable diffusion model on relighting-related parameters. Then, after fixing the parameters of the trained stable diffusion model, it is deployed to the electronic device. It is understood that, in this embodiment, the electronic device used to train the stable diffusion model of the raw image and the electronic device used to deploy the stable diffusion model can be the same type of electronic device or different types of electronic devices.

[0123] For example, a stable diffusion model for graph-native graphs can be trained on a server, and then deployed to electronic devices such as mobile phones and tablets after the parameters are fixed. Alternatively, a stable diffusion model for graph-native graphs can also be trained on electronic devices such as mobile phones and tablets and then directly deployed thereafter; this application does not impose any limitations on this approach.

[0124] Then, after the electronic device receives the first operation from the user to trigger the relighting of the image, the electronic device responds to this first operation by calling the trained stable diffusion model of the original image to relight the image that needs to be relighted (hereinafter referred to as the image to be processed), thereby outputting the relighted image (hereinafter referred to as the output image).

[0125] In addition, in this embodiment of the application, since the stable diffusion model of the image-generated image is an SD model that generates images from images, the input of the model includes not only the image that needs to be relit, but also the ambient light image, that is, in addition to the image to be processed, the electronic device also needs to input the stable diffusion model.

[0126] The ambient light image input to the electronic device can be understood as a template image of lighting effects, used to guide or influence the stable diffusion model of the image-generated image to relight the image to be processed. For the stable diffusion model of the image-generated image, the lighting effect of the input ambient light image is the target lighting effect that the model needs to obtain in the generated image.

[0127] That is, in this embodiment, the stable diffusion model of the image-generated image is based on the input ambient light image to relight the input image to be processed, thereby outputting the relighted image, i.e., the aforementioned output image. Therefore, without additional image processing, the output image of this embodiment generally satisfies the content information of the input image to be processed, and also satisfies the lighting effect of the input ambient light image. For example, the lighting effect of the output image can correspond to the lighting effect of the ambient light image. It is understood that the corresponding lighting effects can be the same or similar. For example, when the matching degree between the lighting effect of the output image and the lighting effect of the input ambient light image reaches a threshold, it can be determined that the lighting effects of the two images correspond. For example, taking a threshold of 90% as an example, when the matching degree between the lighting effect of the output image and the lighting effect of the input ambient light image is higher than 90%, it can be determined that the lighting effect of the output image is the same or similar to the lighting effect of the input ambient light image, that is, it is determined that the lighting effect of the output image corresponds to the lighting effect of the input ambient light image.

[0128] For example, Figure 1 A flowchart of a stable diffusion model for graph-generated graphs is shown.

[0129] like Figure 1 As shown, the electronic device inputs the image to be processed and the ambient light image into the stable diffusion model 100 of the image-generated image. Then, the stable diffusion model 100 of the image-generated image outputs the image generated by relighting, thus obtaining the output image.

[0130] Understandably, in order to illustrate different lighting effects through the accompanying drawings, the drawings of this application embodiments mainly use image colors to represent the lighting effects of the images, with different colors representing images with different lighting effects. For example... Figure 1 As shown, the image to be processed is black, while the ambient light image and the output image are white.

[0131] Right now, Figure 1 The lighting effects of an image are expressed using only two colors: black and white. The image to be processed represents one lighting effect, while the ambient light image and the output image represent another. Simply put... Figure 1 The lighting effect of the image to be processed can be understood as black, while the lighting effect of the ambient light image and the output image can be understood as white. In other words, the lighting effect of the image to be processed is different from that of the ambient light image and the output image, while the lighting effect of the ambient light image is the same as that of the output image.

[0132] It should be noted that, based on the actual lighting effect of the input ambient light image, relighting can be done in several ways, such as... Figure 1 The image shown can also be globally relit, as shown in the image. Figure 2 The localized relighting shown is not limited in this embodiment.

[0133] like Figure 1 As shown, due to the lighting effects of the ambient light image and the image to be processed (i.e., Figure 1 The colors of the image shown are completely different, so the lighting effect of the output image is completely different from that of the image to be processed. Figure 1 The colors of the images shown have all changed, indicating that the overall lighting effect of the image has changed after the image has been relit, which can be understood as global relighting.

[0134] like Figure 2 As shown, due to the lighting effects of the ambient light image and the image to be processed (i.e., Figure 2 The colors of the images shown are only partially different, so the output image, compared to the image to be processed, only shows the lighting effects in a portion of the image area (i.e.,...). Figure 2 The color shown has changed. This indicates that after the image is relit, only a portion of the image's lighting effect has changed, which can be understood as local relighting.

[0135] Therefore, the stable diffusion model in this embodiment can be viewed as an image generation model combining the image content of the image to be processed and the lighting effect of the ambient light image. The output image of this image-generated stable diffusion model is an image generated based on the image content of the image to be processed and the lighting effect of the ambient light image. The output image can also be understood as the image reconstructed from the image to be processed by the image-generated stable diffusion model based on the lighting effect of the ambient light image. Therefore, the image content of the output image can be the same as the image content of the input image to be processed. Simultaneously, the lighting effect of the output image can also be the same as the lighting effect of the input ambient light image. In essence, the output image is an image that combines the image content of the image to be processed with the lighting effect of the ambient light image.

[0136] For example, the lighting effect in an image can include lighting direction and lighting intensity. Therefore, the lighting direction and intensity of the output image can be consistent with those of the input ambient light image. Alternatively, it can be understood that due to the influence of model loss, the lighting direction and intensity of the output image may not be completely consistent with those of the input ambient light image; for example, the lighting direction and intensity can be similar or close. For example, when the matching of lighting direction and lighting intensity between two images is higher than 90%, the lighting effects of the two images can be considered consistent, i.e., corresponding.

[0137] In some embodiments, the ambient light image can be pre-configured in the electronic device, for example, by customizing the corresponding ambient light image based on scene and user satisfaction analysis and then storing it in the electronic device. The ambient light image can also be a user-specified image. For example, the electronic device can receive images uploaded by the user or use images selected by the user as the ambient light image. In a specific embodiment, the electronic device can use an image specified by the user in a gallery (such as a photo album) as the ambient light image.

[0138] This is understandable, because the output image is generated based on the input image to be processed, guided or pulled by the input ambient light image. Therefore, under normal circumstances, the lighting effect of the input image to be processed (hereinafter referred to as the first lighting effect) is different from the lighting effect of the output image (hereinafter referred to as the second lighting effect).

[0139] However, it should be noted that, since the ambient light image in this embodiment is pre-configured or given by the user (such as uploaded or downloaded by the user), and the lighting effect of the image depends on the actual environmental conditions, this embodiment cannot completely guarantee that the lighting effect of the input image to be processed is different from the lighting effect of the ambient light image.

[0140] Therefore, this application embodiment does not exclude the possibility that there may be an extreme case where the lighting effect of the input image to be processed corresponds to the lighting effect of the input ambient light image. If such an extreme case occurs, that is, the input image to be processed and the ambient light image are two images with corresponding lighting effects, then the lighting effect of the final output image may also correspond to the lighting effect of the input image to be processed. That is, the first lighting effect and the second lighting effect may correspond (they may be the same, similar, or have a matching degree higher than a threshold, etc.).

[0141] It should be noted that, since the image processing method provided in this application mainly targets the processing of lighting effects in images, the image content mentioned in this application refers to image content other than lighting effects. For example, the image content of the input image to be processed refers to image content other than the first lighting effect, and the image content of the output image refers to image content other than the second lighting effect.

[0142] Therefore, the embodiments of this application can achieve relighting processing of the image to be processed through stable diffusion technology, thereby improving the illumination effect of the image to be processed and thus improving the image quality and image effect of the image to be processed.

[0143] Meanwhile, the relighting of the image to be processed is completed by the stable diffusion model of the image-generated image, guided or driven by the ambient light image. The ambient light image can be a pre-configured image with good lighting effects, or an image provided by the user that meets the user's lighting effect requirements. Therefore, the relighting of the image to be processed by the stable diffusion model of the image-generated image, guided or driven by the ambient light image, can not only further improve the lighting effect of the image to be processed, and improve the image quality and effect, but also obtain an image that satisfies the user, ensuring a good user experience.

[0144] Additionally, since stable diffusion is a technique commonly used in Artificial Intelligence Painting (AI Painting), it inherently provides relatively fine detail processing for faces (such as human faces). Therefore, even if the embodiments of this application do not specifically instruct or guide the constructed stable diffusion model to perform fine processing on faces, if the image to be processed input to the stable diffusion model includes a face (such as a human face), the final output image will not only have improved lighting effects, but the face may also be clearer compared to the input image to be processed.

[0145] In some embodiments, the aforementioned electronic device may include at least one of the following: mobile phone, foldable electronic device, tablet computer, camera, camcorder, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application does not impose any special limitation on the specific type of the electronic device.

[0146] Figure 3 A schematic diagram of the structure of an electronic device is shown.

[0147] like Figure 3As shown, the electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) connector 330, a charging management module 340, a power management module 341, a battery 342, an antenna 31, an antenna 32, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor module 380, buttons 390, a motor 391, an indicator 392, a camera module 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, a barometric pressure sensor 380C, a magnetic sensor 380D, an accelerometer sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0148] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0149] Processor 310 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0150] The processor can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution. For example, the processor 310 can execute the image processing method provided in the embodiments of this application. Specifically, the processor 310 can call a trained stable diffusion model of the raw image, input the image to be processed and the ambient light image into this stable diffusion model, thereby generating an image with better lighting effects.

[0151] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 may be a cache memory. This memory can store instructions or data that the processor 310 has used or that are used frequently. If the processor 310 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0152] In some embodiments, the processor 310 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) connector, etc. The processor 310 can connect to modules such as touch sensors, audio modules, wireless communication modules, displays, and camera modules through at least one of these interfaces.

[0153] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0154] The external storage interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, it can save music, video, and other files to the external memory card, or transfer music, video, and other files from the electronic device to the external memory card.

[0155] Internal memory 321 can be used to store computer executable program code, including instructions. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 300 (such as audio data, phonebook, etc.). Furthermore, internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 310 executes various functional methods or data processing of electronic device 300 by running instructions stored in internal memory 321 and / or instructions stored in memory disposed in the processor.

[0156] The USB connector 330 is a USB standard-compliant interface used to connect electronic devices 300 and peripheral devices, specifically a Mini USB connector, Micro USB connector, USB Type-C connector, etc. The charging management module 340 receives charging input from the charger. The power management module 341 connects to the battery 342, and the charging management module 340 connects to the processor 310.

[0157] The wireless communication function of electronic device 300 can be implemented through antenna 31, antenna 32, mobile communication module 350, wireless communication module 360, modem processor, and baseband processor.

[0158] Electronic device 300 can implement display functions through a GPU, display screen 394, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0159] Display screen 394 is used to display images, videos, etc. Display screen 394 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 300 may include one or more display screens 394.

[0160] In this embodiment, the electronic device can display images through the display screen 394. For example, the electronic device can display an image on the display screen 394 that has not yet undergone image processing, that is, an image on the display screen 394 that has not yet been relit, such as the image to be processed mentioned above. Alternatively, the electronic device can also display an image on the display screen 394 that has undergone image processing, that is, an image on the display screen 394 after relighting, such as the output image mentioned above. In other words, the user can visually see the images before and after relighting through the display screen 394 and determine the lighting effect of the image through the display screen 394.

[0161] Electronic device 300 can realize camera (such as taking pictures, recording videos, etc.) functions through camera module 393, ISP, video codec, GPU, display screen 394, application processor AP, neural network processor NPU, etc.

[0162] The camera module 393 can be used to acquire color image data and depth data of the subject. The Information Service Provider (ISP) can be used to process the color image data acquired by the camera module 393. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the photosensitive element transmits this electrical signal to the ISP for processing, converting it into a visible image. The ISP can also perform algorithmic optimization of image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera module 393.

[0163] In some embodiments, the camera module 393 may consist of a color camera module and a 3D sensing module.

[0164] In some embodiments, the photosensitive element of the camera in the color camera module can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats.

[0165] In some embodiments, the 3D sensing module can be a time-of-flight (TOF) 3D sensing module or a structured light 3D sensing module. Structured light 3D sensing is an active depth sensing technology. The basic components of a structured light 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of a structured light 3D sensing module is to first emit a specific pattern of light onto the object being photographed, then receive the light coding on the object's surface, compare it with the original projected light pattern, and calculate the object's three-dimensional coordinates using triangulation principles. These three-dimensional coordinates include the distance between the electronic device and the object being photographed. Similarly, TOF 3D sensing can be an active depth sensing technology. The basic components of a TOF 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of a TOF 3D sensing module is to calculate the distance (i.e., depth) between the TOF 3D sensing module and the object being photographed by measuring the time it takes for the infrared light to return, thus obtaining a 3D depth map.

[0166] Structured light 3D sensing modules can also be applied to facial recognition, motion-sensing game consoles, and industrial machine vision inspection. Time-of-flight (TOF) 3D sensing modules can also be applied to game consoles, augmented reality (AR) / virtual reality (VR) and other fields.

[0167] In other embodiments, the camera module 393 may also consist of two or more cameras. These two or more cameras may include a color camera, which can be used to acquire color image data of the object being photographed. These two or more cameras may employ stereo vision technology to acquire depth data of the object being photographed. Stereo vision technology is based on the principle of human parallax. Under natural light, two or more cameras capture images of the same object from different angles, and then triangulation and other calculations are performed to obtain the distance information, i.e., depth information, between the electronic device 300 and the object being photographed.

[0168] In some embodiments, the electronic device 300 may include one or more camera modules 393. Specifically, the electronic device 300 may include one front-facing camera module 393 and one rear-facing camera module 393. The front-facing camera module 393 is typically used to capture color image data and depth data of the photographer facing the display screen 394, while the rear-facing camera module is used to capture color image data and depth data of the subject (such as a person, landscape, etc.) being photographed.

[0169] In some embodiments, the CPU, GPU, or NPU in the processor 310 can process the color image data and depth data acquired by the camera module 393.

[0170] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs. Thus, electronic device 300 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0171] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. The NPU enables intelligent cognitive applications in electronic devices 300, such as image recognition, face recognition, speech recognition, and text understanding. In this embodiment, the NPU can also relight the color image data acquired by the camera module 393 (specifically, the color camera module) based on a constructed stable diffusion model of the image-generated image to obtain an image with better lighting effects. In some embodiments, the CPU or GPU can also run neural network algorithms, such as running the stable diffusion model of the image-generated image constructed in this embodiment, to achieve image relighting processing.

[0172] Electronic device 300 can implement audio functions such as music playback and recording through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0173] Buttons 390 may include a power button, volume buttons, etc. A motor 391 can generate vibration feedback. An indicator 392 may be an indicator light, used to indicate charging status, battery level changes, and also to indicate messages, missed calls, notifications, etc. A SIM card interface 395 is used to connect a SIM card.

[0174] In some embodiments, the software system of the electronic device 300 may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The following embodiments of this application will use a layered architecture of Android as an example. TM Taking the system as an example, the software structure of electronic device 300 is illustrated.

[0175] Figure 4 A software architecture block diagram of an electronic device is shown.

[0176] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, Android... TM The system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime (ART) and native C / C++ libraries, hardware abstraction layer (HAL) and kernel layer.

[0177] The application layer can include a series of application packages. For example... Figure 4 As shown, the application package may include applications such as gallery, camera, maps, WLAN, music, SMS, calls, navigation, Bluetooth, and video.

[0178] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0179] like Figure 4 As shown, the application framework layer may include a window manager, activity manager, input manager, resource manager, notification manager, view system, content provider, etc.

[0180] The window manager provides Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.

[0181] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0182] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0183] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0184] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0185] The Activity Manager Service (AMS) can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers), as well as manage and schedule application processes.

[0186] The Input Manager Service (IMS) provides input management services, which can be used to manage system inputs such as touchscreen input, keypad input, and sensor input. IMS retrieves events from input device nodes and, through interaction with the WMS (Windows Management System), distributes these events to appropriate windows.

[0187] The Android runtime consists of the core libraries and the Android runtime itself. The Android runtime is responsible for converting source code into machine code. The Android runtime primarily employs ahead-of-time (AOT) compilation and just-in-time (JIT) compilation techniques.

[0188] The core library primarily provides basic Java class library functionalities, such as libraries for fundamental data structures, mathematics, I / O, tools, databases, and networking. It also provides APIs for users to develop Android applications.

[0189] Native C / C++ libraries can include multiple functional modules. Examples include: surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0190] The Surface Manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The Media Framework supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. OpenGL ES provides drawing and manipulation of 2D and 3D graphics in applications. SQLite provides a lightweight relational database for applications on electronic devices.

[0191] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to higher layers. For example... Figure 4 As shown, the hardware abstraction layer can include display HAL, audio HAL, camera HAL, Bluetooth HAL, etc.

[0192] The kernel layer is the layer between hardware and software. For example... Figure 4 As shown, the kernel layer can contain display drivers, camera drivers, audio drivers, and Bluetooth drivers.

[0193] The following example, using an image capture scenario, illustrates the workflow of the software and hardware of electronic device 300.

[0194] When the touch sensor 380K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click operation as an example, where the control corresponding to the click operation is the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then calls the kernel layer to start the camera driver, capturing still images or videos through the camera. Specifically, in this embodiment, the image captured by the camera can be the first image in this embodiment. Then, if the touch sensor 380K receives another click operation from the user on the relighting function or relighting filter, the image captured by the camera can be further relit using the image-based SD model to obtain the second image in this embodiment.

[0195] The image processing method proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that the image processing methods in the following embodiments can all be implemented in the electronic device 300 equipped with the above-described hardware structure. Furthermore, for ease of description, the trained stable diffusion model of the image-generated image will be referred to as the image-generated image SD model in the following embodiments of this application.

[0196] In some embodiments, the image processing method provided in this application can be applied to image post-processing scenarios.

[0197] For example, an image post-processing scenario could be an electronic device processing a first image stored in a photo library (album). The electronic device can receive a user's operation to relight the first image stored in the photo library. This user's operation to relight the first image in the photo library can be considered as triggering the electronic device to relight the first image. Here, the first image is the image selected by the user to be relighted, such as the image to be processed mentioned above.

[0198] Then, in response to this first operation, the electronic device invokes the image-to-image (SD) model, inputting the first image selected by the user and the ambient light image into the SD model. Guided by the ambient light image, the SD model re-illuminates the first image to obtain the second image. The second image is the image output by the SD model, as shown in the output image above.

[0199] For example, the first image in the embodiments of this application may be Figure 1 and Figure 2 The image to be processed is shown. The second image can be... Figure 1 and Figure 2The output image shown. The ambient light image can be... Figure 1 and Figure 2 The image shown is of ambient light.

[0200] At this point, since the first image is an image that has already been captured and stored in the image library, if the electronic device only performs image reconstruction using the stable diffusion model of the image-generated image in this embodiment of the application to change the lighting effect of the first image, without performing additional processing on the image content of the first image, then the final second image obtained will be an image in which the image content has not changed, only the lighting effect has changed.

[0201] Therefore, the second image output by the stable diffusion model of the image-generated image can be simply understood as the first image whose lighting effect changes from the first lighting effect to the second lighting effect, and this second lighting effect is consistent with the lighting effect of the ambient light image. In other words, the image content of the obtained second image is the same as the image content of the first image, and the second lighting effect is consistent with the lighting effect of the ambient light image.

[0202] Therefore, in image post-processing scenarios, electronic devices can specifically relight the first image stored in the image library that has poor lighting effects to reconstruct the lighting effects, thereby improving the lighting effects of the stored first image, improving the image effect and image quality, obtaining an image that satisfies the user, and ensuring the user experience.

[0203] Taking the iPhone 10 as an example, Figure 5 and Figure 6 Together, they illustrate a scene diagram of relighting in image post-processing.

[0204] like Figure 5 As shown, the phone 10 can display [the following information] when powered on. Figure 5 The main interface 500 shown is shown. This main interface 500 may include application icons for apps such as Clock, Calendar, Music, Notes, etc., as well as apps such as Gallery, Settings, Camera, Contacts, Phone, and Messages. The phone 10 can respond to the user's click on the Gallery app icon on this main interface 500, entering the Gallery app and displaying the Gallery interface 501.

[0205] like Figure 5 As shown, the gallery interface 501 includes thumbnails 502 of multiple saved first images. Additionally, the bottom of the gallery interface 501 may include function options such as Photos, Albums, Moments, and Discover. Understandably... Figure 5 The first images stored in the image library interface 501 can be images taken by the camera application of the mobile phone 10, or images obtained by importing, downloading, or screenshotting. This application embodiment does not limit this in any way.

[0206] Mobile phone 10 can respond to the user's click on the thumbnail 502 in the gallery interface 501, and enter the large image display interface 503. For example... Figure 5 As shown, the large image display interface 503 may include image 504, which is the large image corresponding to the thumbnail 502 clicked by the user. It can be understood that image 504 and thumbnail 502 are essentially the same first image. Thumbnail 502 is the first image displayed as a smaller image after compression. That is, thumbnail 502 is the compressed image 504.

[0207] The large image display interface 503 also includes sharing, favorite, editing, deletion, and more function options at the bottom. Phone 10 can respond to the user's click on the edit function option in the large image display interface 503, entering the image editing interface 505. For example... Figure 5 As shown, the bottom of the image editing interface 505 may include editing options such as cropping, adjustment, filters, drawing, and text.

[0208] Understandably, after entering the image editing interface (505) on the phone, it can default to selecting one of the editing options. For example... Figure 5 The image editing interface 505 shown currently has cropping as the default selected editing option. Therefore, the image editing interface 505 also includes cropping sizes corresponding to the cropping editing option, such as free, original ratio, 1:1, 9:16, 19:9, and 3:4.

[0209] Next, the phone 10 can respond to the user's click on the filter editing option in the image editing interface 505 and enter... Figure 6 The filter editing interface 506 is shown in the image. Figure 6 As shown, the filter editing interface 506 may include a variety of selectable filters, such as Original, Vivid, Energetic, Bright, Soft, and Relight. Understandably, in some embodiments, the phone 10 may respond to the user's swipe gesture on the filter to switch between displaying more filters (not shown in the accompanying drawings).

[0210] Then, as Figure 6 As shown, the mobile phone 10 can respond to the user's click operation on the relight filter in the filter editing interface 506, triggering the mobile phone 10 to acquire the ambient light image corresponding to the relight filter, and then relight the image 504 according to the acquired ambient light image, thereby obtaining the relighted image 507. That is, the ambient light image corresponding to the relight filter and the image 504 are input into the stable diffusion model of the raw image to obtain the image 507.

[0211] Understandably, the phone 10 stores an ambient light image, and this image is associated with the relight filter. Whenever the phone 10 receives a user's tap on the relight filter, the electronic device can obtain the corresponding ambient light image based on this association.

[0212] In other words, after the mobile phone 10 displays image 504 in response to the user's operation in the mobile phone 10's gallery, the user's click operation in the filter editing interface 506 to select the "relighting" filter for image 504 can be regarded as the first operation in this application embodiment. In this case, the image 504 selected by the user is the first image in this application embodiment (i.e., the image to be processed mentioned above), and image 507 is the second image in this application embodiment (i.e., the output image mentioned above). At this time, the first image and the second image are two images with the same content but different lighting effects.

[0213] It should be noted that, Figures 5-6 The image post-processing scenarios shown are merely examples used in the embodiments of this application and do not constitute any limitation on the image post-processing scenarios applicable to the embodiments of this application. For example, relighting can also be a separate option, so the first operation in the embodiments of this application can be anything other than... Figures 5-6 In addition to clicking to select the Relight filter in the filter editing interface 506, as shown, you can also click to select the Relight option as a separate option.

[0214] Figure 7 This diagram illustrates another scenario where relighting is performed during image post-processing.

[0215] like Figure 7 As shown, another image editing interface 701 is illustrated, which also includes image 504. However, in addition to editing options such as cropping, adjustment, filters, and text, this image editing interface 701 also includes a relighting option at the bottom. In this case, relighting is a separate option.

[0216] In this case, the ambient light image saved on the phone 10 is associated with the relighting editing option. Thus, as... Figure 7 As shown, the mobile phone 10 can respond to the user's click operation on the relighting editing option in the image editing interface 701, and relight the image 504 using the ambient light image corresponding to the relighting editing option to obtain the relighted image 507. At this time, image 504 is still the first image, and image 507 is still the second image. However, in this embodiment of the application, the first operation is the user's click operation on the relighting editing option in the image editing interface 701.

[0217] In some embodiments, the image processing method provided in this application can also be applied to image capturing scenarios.

[0218] For example, an image capture scenario could be an electronic device activating its camera to capture an image. The electronic device can receive a user's activation command within the camera to enable the relighting function; this activation command can be considered as triggering a first operation to relight the first image (the image captured in real-time by the camera). Then, in response to this first operation, the electronic device activates the camera's relighting function.

[0219] Understandably, since current electronic devices (such as mobile phones and tablets) are in a state of real-time image acquisition once their cameras are activated, this state is manifested on the interface as the image displayed in the camera's viewfinder changing in real time. Therefore, in an image capture scenario, the first image input by the electronic device to the image-to-image SD model (such as the image to be processed mentioned above) is the image captured in real time by the electronic device's camera. Correspondingly, the second image generated and output by the image-to-image SD model in this embodiment (i.e., the output image mentioned above) is obtained by relighting the image captured in real time by the camera.

[0220] Therefore, when the relighting function in the camera is enabled, the electronic device will call the image-to-image (SD) model and relight the first image (the image captured by the camera in real time) based on the ambient light image to obtain the second image (obtained by relighting the image captured by the camera in real time).

[0221] Meanwhile, since the image capture scenario involves relighting the image captured in real time by the camera, the image content of the first image that the user can see on the interface before the relighting function is turned on may be the same as or different from the image content of the first image actually input to the image-generated SD model after the relighting function is turned on.

[0222] In other words, compared to the intuitive situation in image post-processing where the first image displayed on the interface is the first image input to the raw image SD model, in image shooting scenarios, the first image input to the raw image SD model is an image captured in real-time by the camera. Therefore, if the relighting function is enabled, after capturing the first image, the electronic device will first input it to the raw image SD model for relighting processing before displaying it in the viewfinder. Thus, the user cannot directly see the first image input to the raw image SD model before the electronic device displays it. Furthermore, during actual shooting, the image content captured by the camera changes with the user's shooting actions. Therefore, the image content of the first image that the user sees on the interface before the relighting function is enabled may be the same as or different from the image content of the first image input to the raw image SD model after the relighting function is enabled.

[0223] Furthermore, depending on the different image functions of the camera on an electronic device, such as the camera on a mobile phone or tablet, which includes both image preview and image capture functions, the image captured in real time by the camera of an electronic device can be either a preview image corresponding to the image preview function or a captured image corresponding to the image capture function.

[0224] In this process, the preview image is typically displayed in real-time within the viewfinder of the shooting interface after capture. Therefore, if the first image input to the image-to-image SD model by the electronic device is the preview image corresponding to the camera's image preview function, the electronic device will also display the second image output by the image-to-image SD model within the viewfinder of the shooting interface.

[0225] Images are typically saved directly after capture and are not displayed in the viewfinder. Therefore, if the first image input to the image-generated SD model is a captured image from the camera's image capture function, the electronic device can save the second image directly instead of displaying it in the viewfinder. For example, the electronic device can save the second image directly to its photo library.

[0226] In other embodiments, although the electronic device does not display the captured image within the viewfinder of the shooting interface, it can display it as a thumbnail on the shooting interface. Therefore, the electronic device in this embodiment can also display a thumbnail of the second image on the shooting interface.

[0227] Therefore, if the first image (the image captured in real-time by the camera) is a preview image, then the second image (obtained by relighting the image captured in real-time by the camera) will be displayed in the viewfinder of the shooting interface. If the first image (the image captured in real-time by the camera) is a captured image, then the second image (obtained by relighting the image captured in real-time by the camera) will be saved directly and can then be displayed in the shooting interface as a thumbnail.

[0228] In other words, during the image preview stage, before the camera's relighting function is activated (i.e., before the electronic device responds to the first operation), the image displayed in the viewfinder of the shooting interface is the image captured by the camera in real time. This image has not undergone relighting processing; it is the first image of the first lighting effect. After the camera's relighting function is activated, the electronic device performs relighting processing on the image captured by the camera in real time using a raw image SD model. Therefore, the image displayed in the viewfinder of the shooting interface is the image after relighting processing of the image captured by the camera in real time; it is the second image of the second lighting effect.

[0229] Therefore, when taking a picture, users can preview the lighting effect of the image after relighting during the image preview stage. In other words, users can directly observe the second lighting effect after relighting, which makes it easier for users to adjust the lighting effect they need for shooting, thereby obtaining the lighting effect they are satisfied with and ensuring user experience.

[0230] During the image capture stage, if the camera's relighting function is enabled and the electronic device receives a shooting operation from the user (such as the user clicking the shooting control), the electronic device can respond to the user's shooting operation by relighting the image captured by the camera in real time and saving it, thus directly saving the image with better lighting effect.

[0231] Therefore, electronic devices can reconstruct the lighting effect of the captured image based on the user's determined lighting requirements during the image capture stage, thereby directly capturing an image with better lighting effect, improving the image quality and effect of the captured image, obtaining an image that satisfies the user, and ensuring user experience.

[0232] It should be noted that, in the image capture scenario of this application embodiment, whether in the image preview stage or the image capture stage, since both the first image and the second image are images captured by the camera in real time, the image content of the first image and the second image specifically depends on the actual shooting situation of the electronic device. Whether the image content of the first image and the second image are the same depends on the actual shooting situation of the electronic device, and this application embodiment does not impose any limitations on this.

[0233] Taking the iPhone 10 as an example, Figure 8This diagram illustrates a scene where relighting is applied during image capture.

[0234] like Figure 8 As shown, the phone 10 can display [the following information] when powered on. Figure 8 The main interface 800 shown is shown. This main interface 800 may include icons for applications such as clock, calendar, gallery, memo, ... and applications such as settings, camera, contacts, phone, and messages. The phone 10 can respond to the user's click on the camera application icon on this main interface 800, entering the camera application and displaying the shooting interface 801.

[0235] The viewfinder of the shooting interface 801 displays a preview image 802. Understandably, the preview image 802 displayed in the viewfinder is captured in real-time by the camera of the mobile phone 10. Furthermore, this shooting interface 801 can include multiple shooting options.

[0236] like Figure 8 As shown, the shooting interface 801 may include shooting options such as flash, AI, filters, HDR, settings, aperture, night scene, portrait, photo, video, professional, and more. Then, the phone 10 can respond to the user's click operation on the filter shooting option within the shooting interface 801 and enter the filter selection interface 803.

[0237] The filter selection interface 803 includes a variety of filters to choose from. However, images are generally displayed without filters by default, i.e., the original image. Therefore, after the phone 10 responds to user input and enters the filter selection interface 803, the filter selected on this interface 803 will usually be the original image. For example... Figure 8 As shown, the original image filter is selected.

[0238] Furthermore, the mobile phone 10 can respond to the user's click operation on the relight filter in the filter selection interface 803. This click operation is equivalent to activating the camera's relight function. In other words, the user's click operation to select the relight filter in the camera application is the first operation in this embodiment of the application.

[0239] Next, the phone will relight the preview image captured in real time by the phone's camera before displaying it in the viewfinder, such as... Figure 8 The preview image 805 is shown in the viewfinder of interface 804.

[0240] It should be noted that, Figure 8 This is a schematic diagram shown under the premise that the content of the preview image remains unchanged. Therefore, Figure 8 The image content displayed in the viewfinder of the shooting interface 801, filter selection interface 803, and interface 804 is the same. That is, Figure 8The content of the preview image 802 without relighting and the preview image 805 after relighting is the same.

[0241] However, it is understandable that because the preview image is an image captured by the camera in real time, the image content captured by the camera will change with the user's shooting actions during the actual shooting process. Therefore, the image content of the preview image displayed in the viewfinder can change based on the changes in the user's actual shooting actions. For example, with changes in the actual shooting, the image content of the preview image 802 without relighting may be different from the image content of the preview image 805 after relighting. Therefore, whether the image content changes depends on the actual shooting situation, as described in the embodiments of this application. Figure 8 This image is for illustrative purposes only and does not constitute any limitation on the image content captured by the camera in real time.

[0242] Furthermore, taking the iPhone 10 as an example, Figure 9 This diagram illustrates a scenario where relighting is applied in another image capture situation.

[0243] With the lighting turned on again, such as Figure 8 When the relighting filter is enabled, if the electronic device receives a user's shooting operation (such as clicking the shooting control in the camera application), the electronic device responds to this operation by using the camera to capture the image. Then, it further processes the captured image with relighting, outputting the relighted image to the gallery (photo album) as the user's captured image. In this case, the image captured by the camera is still the first image with the first lighting effect without relighting, while the image output by the electronic device and saved in the gallery is the second image with the second lighting effect after relighting.

[0244] like Figure 9 As shown, when the relighting filter is enabled, the mobile phone 10 can respond to the user's click operation on the shooting control 901, call the image-generated image SD model to relight the first image of the first lighting effect captured by the camera in real time, and process and save the second image of the second lighting effect to the gallery (album).

[0245] In some embodiments, the mobile phone 10 may display a second image of the second lighting effect as a thumbnail on the shooting interface, such as... Figure 9 The thumbnail 903 is shown in the interface 902.

[0246] Understandably, in this image-taking scenario, from the user's perspective, after the user activates relighting within the camera app, the images displayed in the camera viewfinder will all be relit images. Similarly, with relighting enabled, images captured and saved to the gallery by the electronic device after clicking the shooting control will also be relit images.

[0247] It should be noted that, Figures 8-9 The relighting process shown in the image shooting scenario is only an example of the embodiments of this application and does not constitute any limitation on the image shooting scenarios to which the embodiments of this application are applicable.

[0248] For example, relighting can also be a separate option. Therefore, the first operation in the embodiments of this application can be in addition to being Figures 8-9 In addition to selecting the relight filter within the filter shooting options shown, you can also click on the relight option as a separate action. For example... Figure 10 As shown, the phone 10 can treat relighting as a separate shooting option. Then, the phone 10 can treat the user's tap on the relighting shooting option in the camera as the first operation, and the phone 10 will relight the corresponding image in response to the user's tap on the relighting shooting option.

[0249] In another embodiment, similar to image capture scenarios, the image processing method provided in this application can also be applied to video recording scenarios. For example, using... Figure 8 For example, after the user taps the image in the camera application and enters the shooting interface 801, the shooting interface 801 may include a video recording option. Then, the phone 10 can further respond to the user's tap on the video recording option in the shooting interface 801 and enter the video recording interface (not shown in the attached diagram). Thus, the phone 10 can respond to the user's operation in the video recording interface to activate the relighting, and subsequently respond to the user's recording operations, including starting and ending recording, saving a video containing multiple consecutive second images.

[0250] Figure 11 A flowchart of an image processing method is shown, including steps S1101-S1104. The following is a combination of... Figure 11 The image processing method provided in the embodiments of this application will be described.

[0251] S1101, the electronic device displays a first image with a first lighting effect on a first interface.

[0252] The first interface can be a large image display interface in the electronic device's image library; correspondingly, the first image can be an image saved in the image library. The electronic device can respond to a user's click to open the first image in the image library and display the first image in the large image display interface (i.e., the first interface). For example, the first interface could be... Figure 5 The large image display interface 503 shown can have the first image as... Figure 5 The large image displayed is shown in interface 503, which contains image 504.

[0253] Alternatively, the first interface could be the camera preview interface of an electronic device. Correspondingly, the first image could be a preview image captured in real-time by the camera, which could then be further processed and displayed within the viewfinder of the shooting interface. For example, in response to a user's camera activation, the electronic device could capture a first image, display the camera preview interface, and show this captured first image within the viewfinder of the camera preview interface (i.e., the first interface). For example, the camera preview interface (first interface) could be... Figure 8 The shooting interface 801 shown can have the first image as... Figure 8 The preview image 802 is shown in the viewfinder of the shooting interface 801. Alternatively, the first interface can also be the video recording interface of an electronic device's camera. In the video recording scenario, the first image is also an image captured in real-time by the camera. The principle is roughly the same as in the photo shooting scenario, and will not be elaborated further.

[0254] S1102, the electronic device receives the user's first operation on the first interface.

[0255] The first operation is used to trigger the electronic device to relight the image on the first interface. Understandably, the first operation may differ depending on the actual settings of the relighting function and the actual design of the first interface.

[0256] For example, when the first interface is a large image display interface of the electronic device gallery, the first operation could be... Figure 5 The image shows the click operation in filter editing interface 506 where you select the "Relight" filter. The first operation could also be... Figure 7 The image shows the clicking operation of the relighting editing option in the image editing interface 701.

[0257] For example, when the first interface is the camera preview screen of an electronic device, the first operation could be... Figure 8 The image shows the click operation of selecting the Relight filter in filter selection interface 803. The first operation can also be... Figure 10 The image shows the clicking action for the "Relighting" shooting option.

[0258] S1103, the electronic device responds to the first operation by inputting the first image and the target ambient light image into the trained stable diffusion model of the image-generated image, and outputs a second image with the second illumination effect.

[0259] After receiving the user's first operation, the electronic device indicates that the user needs relighting of the first image displayed on the first interface. Therefore, in response to this first operation, the electronic device begins to relight the first image displayed on the first interface.

[0260] Electronic devices call the graph-based stable diffusion model (SD model, i.e., a trained graph-based stable diffusion model), such as... Figure 1 or Figure 2 As shown, the electronic device will display the first image (i.e. Figure 1 or Figure 2 The image to be processed in the image and the target ambient light image (i.e., the image to be processed in the image) ... Figure 1 or Figure 2 The ambient light image (from the first image) is input into the trained image-generated image SD model. The image-generated image SD model generates and outputs a second image (i.e., the second illumination effect image) based on the target ambient light image and the first image. Figure 1 or Figure 2 (output image in the image).

[0261] The target ambient light image is the ambient light image that determines the stable diffusion model to be input into the image-generated image. It's understandable that if the electronic device only stores one candidate ambient light image, meaning the relighting process only has one usable ambient light image, then the target ambient light image is this candidate ambient light image stored in the electronic device.

[0262] If multiple candidate ambient light images exist, meaning there are multiple available ambient light images that can be used for relighting, this could be achieved by pre-storing multiple candidate ambient light images in the electronic device. Alternatively, the electronic device can receive an image specified by the user from a gallery as the target ambient light image; in this case, all images stored in the gallery can be considered candidate ambient light images. The target ambient light image is then the candidate ambient light image selected by the electronic device. In this scenario, the electronic device can select the candidate ambient light image as the target ambient light image based on the user's action.

[0263] In one specific embodiment, the electronic device can use the candidate ambient light image corresponding to the first operation as the target ambient light image. That is, if the electronic device includes multiple candidate ambient light images to choose from, it can display all the candidate ambient light images on the interface for the user to select. Then, the electronic device determines the target ambient light image based on the user's selection.

[0264] Figure 12 and Figure 13 Taking filters as an example, the interface diagrams for a candidate ambient light image are shown respectively.

[0265] like Figure 12 As shown, for image post-processing scenarios, if the electronic device includes multiple candidate ambient light images, then the electronic device can display all candidate ambient light images within the filter editing interface 506. For example... Figure 12 Relighting 1 and Relighting 2 are shown in the image. Relighting 1 can correspond to candidate ambient light image 1, and Relighting 2 can correspond to candidate ambient light image 2.

[0266] Then, as Figure 12 As shown, if the electronic device receives the user's first operation of clicking "Relight 2" in the filter editing interface 506, the electronic device can determine that the target ambient light image is the candidate ambient light image 2. Conversely, if the electronic device receives the user's first operation of clicking "Relight 1" in the filter editing interface 506, the electronic device can determine that the target ambient light image is the candidate ambient light image 1.

[0267] Similarly, such as Figure 13 As shown, for an image capture scenario, if the electronic device includes multiple candidate ambient light images, then the electronic device can display all candidate ambient light images within the filter selection interface 803. For example... Figure 13 Relighting 1 and Relighting 2 are used. Relighting 1 corresponds to candidate ambient light image 1, and Relighting 2 corresponds to candidate ambient light image 2. Then, as... Figure 13 As shown, if the electronic device receives a user's first operation of clicking "Relighting 2" in the filter selection interface 803, the electronic device can determine that the target ambient light image is the candidate ambient light image 2. Similarly, if the electronic device receives a user's first operation of clicking "Relighting 1" in the filter selection interface 803, the electronic device can determine that the candidate ambient light image 1 is the target ambient light image.

[0268] Understandable. Figure 12 and Figure 13 The names "Re-lighting 1," "Re-lighting 2," etc., used in this application are merely examples for the embodiments of this application. Specific naming can be configured according to actual needs, and this application does not impose any limitations on this. For example, they could also be named "First Re-lighting" and "Second Re-lighting." Alternatively, they could be named "First Fill Light" and "Second Fill Light."

[0269] Additionally, if relighting is not a filter option, but rather... Figure 7 The individual editing options shown or as... Figure 10The individual shooting option is shown. When multiple candidate ambient light images are available, the electronic device can respond to the user's click on the relighting editing option, or the user's click on the relighting shooting option, by further displaying all candidate ambient light images. Then, the electronic device receives the user's selection of the displayed candidate ambient light images and uses the selected candidate ambient light image as the target ambient light image.

[0270] S1104, the electronic device displays a second image on the first interface.

[0271] After the electronic device relights the first image with the first lighting effect to obtain a second image with the second lighting effect, it can display the relit second image on the first interface. The second lighting effect of the obtained second image corresponds to the lighting effect of the target ambient light image. However, depending on the application scenario, the image content of the second image may be the same as or different from the image content of the first image. For example, if the second image is obtained by relighting the first image in the image library, the image content of the second image can be the same as the first image. However, in image capture or video recording scenarios, the first image is an image captured in real time by the camera without relighting, while the second image is an image captured by the camera with relighting. Because it is captured in real time by the camera, the image content of the first image and the second image may be different.

[0272] Therefore, the embodiments of this application can obtain a second image after the illumination effect reconstruction through the stable diffusion model of the image-generated image, thereby improving the image quality and image effect, obtaining an image that satisfies the user, and ensuring the user experience.

[0273] The following will describe in detail, with reference to the accompanying drawings, the above-mentioned S1103, namely, the processing process in the embodiment of this application in which the first image and the target ambient light image are used as inputs to the image-generated image SD model, and the image-generated image SD model reconstructs the lighting effect and outputs the second image of the second lighting effect.

[0274] To facilitate understanding of the solution, the embodiments of this application first describe the traditional stable diffusion model.

[0275] Traditional stable diffusion models are typically text-to-image or text-to-image-to-image. A text-to-image model generates an image that matches the text description based on the given text. A text-to-image model, on the other hand, simultaneously inputs both text and an image, then generates an image based on both. Therefore, traditional stable diffusion models usually involve text processing in addition to image processing.

[0276] This application uses a text-based image as an example in its embodiments. Figure 14A schematic diagram of a traditional stable diffusion model is shown.

[0277] like Figure 14 As shown, a traditional stable diffusion model includes an image encoder, a text encoder, an image decoder, a noise-adding module, a noise-reducing module, and n cascaded U-net units. Here, n is an integer, n≥1.

[0278] An image encoder encodes an image into low-dimensional latent data. Simply put, it extracts the latent features of an image to obtain the corresponding latent image. After encoding, the image can undergo forward diffusion (adding noise to the latent data) and backward diffusion (removing noise from the latent data) in the latent space. An image decoder then decodes the latent data back into an image.

[0279] The text encoder encodes the text, converting it into an embedded representation of the context. The noise-adding module adds noise to the latent image, resulting in a noisy latent image. The denoising module removes the predicted noise from the image, resulting in a denoised latent image.

[0280] n cascaded U-net units are a series of layers used to iteratively process the noisy letent image to predict the noise in the noisy letent image, thus outputting a predicted noisy image. For a stable diffusion model of text-to-image generation, the input of these n cascaded U-net units includes both the image and the text. Figure 14 As shown, the inputs of these n cascaded U-net units include the noisy latent image output by the noisy module, as well as the output of the text encoder.

[0281] Generally, each of the n cascaded U-net units has the same structure. Each U-net unit operates on the output of the previous U-net unit, thus achieving iterative processing. Furthermore, each of these n cascaded U-net units may contain multiple layers, and each of these layers also operates on the output of the previous layer. Simultaneously, a portion of the output from each layer within a U-net unit is connected to a backend layer via a skip connection, allowing the backend layer to directly receive and process the output from the frontend layer.

[0282] Figure 15 A schematic diagram of the structure of a U-net in a traditional stable diffusion model is shown.

[0283] like Figure 15 As shown, the U-net unit in the stable diffusion model includes two convolutional layers (Conv layers, abbreviated as CV in the figure), four downsampling layers (Down layers, abbreviated as Dn in the figure), one intermediate connection layer (Middle layer, abbreviated as Mid in the figure), and four upsampling layers (Up layers, abbreviated as Up in the figure). Figure 15 As shown, the connection order of each layer in the U-net unit is: Conv→Down→Down→Down→Down→Middle→Up→Up→Up→Up→Conv.

[0284] like Figure 15 As shown, according to the connection order, the first Conv layer is the first layer of the U-net unit, so the input is the noisy latent image, i.e., the noisy latent image. The output of the first Conv layer is the input of the first Down layer. At the same time, the output of the first Conv layer is also transmitted to the cascaded layers in the backend Up layer.

[0285] Another Conv layer is the last layer of the U-net unit. The input of this Conv layer is the output of the last Up layer, and the output of this Conv layer is the processing result of this U-net unit, that is, the prediction of the noisy image.

[0286] The Down layers of the U-net unit include ResNet Block (ResBlock), cross attention layers, and down-sizing convolutions. The Up layers of the U-net unit include Channel Concatenation, ResBlock, cross attention layers, and up-sizing convolutions. The Middle layers include ResBlock and cross attention layers.

[0287] In the traditional stable diffusion model of text-to-image or text-to-image, the cross attention layer is mainly used to cross-process text and images, thereby fusing text features and image features, so that the model can generate images based on text guidance or traction.

[0288] like Figure 15As shown, in the connection order, the first Down layer (the first Dn1 in the attached diagram), the second Down layer (the second Dn1 in the attached diagram), and the third Down layer (the third Dn1 in the attached diagram, the structure of which is not shown in detail in the attached diagram due to space limitations) all include two ResBlocks, two cross attention layers, and one Down-sizingConv. The connection order of each layer within these three Down layers (Dn1 in the attached diagram) is: ResBlock → cross attention layer → ResBlock → cross attention layer → Down-sizing Conv.

[0289] Meanwhile, the outputs of the cross attention layer and the Down-sizing Conv in these three Down layers (Dn1 in the attached diagram) are transmitted to the cascaded layers in the backend Up layer.

[0290] The fourth Down layer (Dn2 in the attached diagram) consists of only two connected ResBlocks, and the output of the second ResBlock is also transmitted to the cascaded layer in the backend Up layer.

[0291] like Figure 15 As shown, the Middle layer (Mid in the attached diagram) consists of two ResBlocks and one cross attention layer, connected in the following order: ResBlock → cross attention layer → ResBlock.

[0292] like Figure 15 As shown, the first Up layer (Up1 in the attached diagram) includes three cascaded layers, three ResBlocks, and one Up-sizing Conv. The connection order of the layers within this Up layer (Up1 in the attached diagram) is: cascaded layer → ResBlock → cascaded layer → ResBlock → cascaded layer → ResBlock → Up-sizing Conv.

[0293] The second Up layer (the first Up2 in the attached diagram) and the third Up layer (the second Up2 in the attached diagram) each consist of three cascaded layers, three ResBlocks, three cross attention layers, and one Up-sizing Conv.

[0294] The connection order of each layer within these two Up layers (the first Up2 in the attached diagram and the second Up2 in the attached diagram) is as follows: cascaded layer → ResBlock → cross attention layer → cascaded layer → ResBlock → cross attention layer → cascaded layer → ResBlock → cross attention layer → Up-sizing Conv.

[0295] The fourth Up layer (Up3 in the attached diagram) consists of three cascaded layers, three ResBlock layers, and three crossattention layers. The connection order of the layers within this Up layer (Up3 in the attached diagram) is as follows: cascaded layer → ResBlock → crossattention layer → cascaded layer → ResBlock → cross attention layer → cascaded layer → ResBlock → crossattention layer.

[0296] like Figures 14-15 It is known that traditional stable diffusion models are only applicable to text-based or text-image-based scenarios. However, the embodiments of this application relight a first image with a first lighting effect based on a target ambient light image to obtain a second image with a second lighting effect. Therefore, the embodiments of this application are for image-based scenarios, which are different from traditional text-based or text-image-based scenarios. Therefore, traditional stable diffusion models cannot be applied to the image-based scenario of this application, so this application requires a stable diffusion model capable of achieving image-based rendering. Based on this, the embodiments of this application construct a stable diffusion model that can be used for image-based rendering, namely the aforementioned image-based SD model.

[0297] Next, this application will provide a detailed description of the constructed graph-generated graph SD model.

[0298] The graph-generated image SD model in this application mainly includes an image encoder, an image decoder, a noise-adding module, a noise-reducing module, and 2n graph-generated image U-net units. Here, n is an integer, n≥1.

[0299] Because this application focuses on graph-based images, it does not require text features. Therefore, compared to traditional stable diffusion models, the graph-based SD model in this application does not have a text encoder overall.

[0300] Meanwhile, this embodiment of the application requires processing two images: the first image and the target ambient light image. Therefore, the image-to-image SD model of this embodiment includes two parallel branches, which process the first image and the target ambient light image respectively. Hereinafter, this embodiment of the application refers to the branch that processes the first image as the first stable diffusion branch and the branch that processes the target ambient light image as the second stable diffusion branch.

[0301] Furthermore, the cross attention layer of the U-net unit in the traditional stable diffusion model focuses on the cross processing of text and images, and is not applicable to the processing of two images in the embodiments of this application.

[0302] Therefore, the cross-attention mechanism of the image-to-image SD model in this embodiment adopts an image-to-image cross-attention mechanism. That is, in this embodiment, the image-to-image SD model includes an image cross-attention mechanism layer. Specifically, both the first stable diffusion branch and the second stable diffusion branch include an image cross-attention mechanism layer, and the first stable diffusion branch and the second stable diffusion branch are connected through this image cross-attention mechanism layer.

[0303] Therefore, the first image is input to the first stable diffusion branch, and the target ambient light image is input to the second stable diffusion branch. Since the first and second stable diffusion branches include an image cross-attention mechanism layer and are connected based on this layer, this image cross-attention mechanism layer can simultaneously acquire both the first image and the target ambient light image. It then performs cross-processing on the input first and target ambient light images, fusing the features of the two images. Simultaneously, this image cross-attention mechanism layer also transmits the cross-processing result to the first and second stable diffusion branches respectively. In this way, the first stable diffusion branch, used to process the first image, can obtain the image information of the target ambient light image from the connected second stable diffusion branch through the image cross-attention mechanism layer. This allows the first stable diffusion branch to reconstruct the first lighting effect in the first image under the guidance or influence of the target ambient light image, outputting a second image with the second lighting effect.

[0304] Figure 16 A schematic diagram of a graph-generated graph SD model is shown.

[0305] like Figure 16 As shown, the first stable diffusion branch includes a first image encoder, a first noise-adding module, a first U-net branch, a denoising module, and an image decoder. The second stable diffusion branch includes a second image encoder, a second noise-adding module, and a second U-net branch. Wherein, as... Figure 16As shown, the first U-net branch consists of n serially connected first U-net units, and the second U-net branch consists of n unconnected second U-net units. Simultaneously, the aforementioned image cross-attention mechanism layer is set within these n first U-net units and n second U-net units. That is, the first U-net units in the first U-net branch and the second U-net units in the second U-net branch include the image cross-attention mechanism layer, and these n serially connected first U-net units are connected one-to-one with these n unconnected second U-net units through the image cross-attention mechanism layer. Therefore, the graph-generated graph SD model in this embodiment has n sets of connected first U-net units and second U-net units.

[0306] Understandably, since the image-to-image SD model in this embodiment is based on two parallel branches that process the first image and the target ambient light image respectively, the final image-to-image SD model will predict and output two predicted noise images.

[0307] That is, such as Figure 16 As shown, the first U-net branch outputs a first predicted noise image obtained by performing noise prediction on the noisy first latent image. Simultaneously, the second U-net branch outputs a second predicted noise image obtained by performing noise prediction on the noisy target latent image. However, it should be noted that since the second predicted noise image does not require further processing, it is discarded by electronic devices in practical applications.

[0308] like Figure 16 As shown, the SD model will not perform any further processing on the second predicted noise image. The second stable diffusion branch ends after outputting the second predicted noise.

[0309] Figure 17 A schematic diagram of the U-net unit structure in the graph-generated image SD model is shown. The following, combined with... Figure 17 The structure of the U-net unit in the embodiments of this application will be described.

[0310] Understandable. Figure 17 This is a schematic diagram illustrating a structure using a connected first U-net unit and a second U-net unit as an example. That is, Figure 17 The structure shown includes only one first U-net unit and one second U-net unit.

[0311] like Figure 17 As shown, both the first and second U-net units include two Convs, four Down layers, one Middle layer, and four Up layers. The connection order is also as follows:

[0312] Conv→Down→Down→Down→Down→Middle→Up→Up→Up→Up→Conv.

[0313] Understandably, since the first U-net branch to which the first U-net unit belongs is primarily used to process the first image, the input and output of the first U-net unit are mainly related to the first image. For example, the input of the first first U-net unit in the first U-net branch is the latent image corresponding to the noisy first image; that is, the input of the first first U-net unit in the first U-net branch is the noisy first latent image.

[0314] The input of the middle U-net unit in the first U-net branch is the output of the previous U-net unit. The output of the last U-net unit in the first U-net branch is the image noise of the first image predicted by the first U-net branch, i.e., the first predicted noise image.

[0315] Similarly, the second U-net branch to which the second U-net unit belongs is the U-net branch mainly used for processing the target ambient light image. Therefore, the input and output of the second U-net unit are primarily related to the target ambient light image. For example, the input of the first second U-net unit in the second U-net branch is the latent image corresponding to the noisy target ambient light image; that is, the input of the first second U-net unit in the second U-net branch is the noisy target latent image.

[0316] Similarly, the input of the middle second U-net unit in the second U-net branch is the output of the previous second U-net unit. The output of the last second U-net unit in the second U-net branch is the image noise of the target ambient light image predicted by the second U-net branch, i.e., the second predicted noise image. However, it should be noted that the second predicted noise image is not used in subsequent processing in this embodiment, so in actual use, the electronic device will discard the second predicted noise.

[0317] At the same time, such as Figure 17 As shown, both the first U-net unit and the second U-net unit include a cross-two-image attention layer, and the first U-net unit and the second U-net unit are connected based on these cross-two-image attention layers. That is, a set of connected first U-net units and second U-net units are mainly connected through cross-two-image attention layers.

[0318] Therefore, the U-net unit in the image-text-image SD model can perform cross-processing on the first image and the target ambient light image based on the cross two-image attention layer to fuse the image features of the first image and the target ambient light image. This allows the image-text-image SD model to finally output a second image with the same image content as the first image and the same lighting effect as the target ambient light image.

[0319] Specifically, such as Figure 17 As shown, according to the connection order, the first Conv in the first U-net unit is the first layer of this first U-net unit. Depending on the cascade order of the first U-net units in the first U-net branch, the input to the first Conv in this first U-net unit can be either the noisy first latent image or the output of the previous first U-net unit. The output of the first Conv in the first U-net unit is the input to the first Down layer (as shown by the first Dn1 in the attached figure) in this first U-net unit. Simultaneously, the output of the first Conv in the first U-net unit is transmitted to the cascaded layers in the backend Up layer.

[0320] The other Conv in the first U-net unit is the last layer of this first U-net unit, and its input is the output of the last Up layer in the first U-net unit (Up3 in the attached figure). Depending on the concatenation order, the output of this Conv can be either the first predicted noisy image or the input of the next first U-net unit.

[0321] The two Convs in the second U-net unit have different inputs than the two Convs in the first U-net unit, but their operating principles are the same, and will not be elaborated further here. For example, the input to the first Conv in the second U-net unit can be the noisy target latent image or the output of the previous second U-net unit, and the output is the input to the first Down layer (the first Dn1 in the attached figure) in the second U-net unit. Similarly, the input to the other Conv in the second U-net unit is the output of the last Up layer (Up3 in the attached figure) in the second U-net unit, and the output can be the second predicted noise or the input to the next second U-net unit.

[0322] like Figure 17As shown, the Down layers in both the first and second U-net units include ResBlock, a cross-two-image attention layer, and a Down-sizing Conv. Similarly, the Up layers in both units also include cascaded layers: ResBlock, a cross-two-image attention layer, and an Up-sizing Conv. The Middle layers in both units also include ResBlock and a cross-two-image attention layer.

[0323] like Figure 17 As shown, according to the connection order, the first Down layer (the first Dn1 in the attached figure), the second Down layer (the second Dn1 in the attached figure), and the third Down layer (the third Dn1 in the attached figure, the structure of which is not shown in detail in the attached figure due to space limitations) in the first U-net unit and the second U-net unit all include two ResBlocks, two cross-two-image attention layers, and one Down-sizing Conv. The connection order of each layer in these three Down layers (Dn1 in the attached figure) in the first U-net unit and the second U-net unit is: ResBlock → cross two-image attention → ResBlock → cross two-image attention → Down-sizing Conv.

[0324] Meanwhile, the outputs of the cross two-image attention layer and the output of the Down-sizing Conv in these three Down layers (Dn1 in the attached diagram) are transmitted to the cascaded layers in the backend Up layer.

[0325] The fourth Down layer (Dn2 in the attached diagram) in the first U-net unit and the second U-net unit only includes two connected ResBlocks, and the output of the second ResBlock is also transmitted to the cascaded layer in the backend Up layer.

[0326] like Figure 17As shown, the Middle layer (Mid in the attached figure) in the first U-net unit and the second U-net unit includes two ResBlocks and a cross two-image attention layer. The connection order of each layer in this Middle layer (Mid in the attached figure) is: ResBlock → cross two-image attention → ResBlock.

[0327] like Figure 17 As shown, the first Up layer (Up1 in the attached figure) in the first U-net unit and the second U-net unit includes three cascaded layers, three ResBlocks, and one Up-sizing Conv. The connection order of each layer in this Up layer (Up1 in the attached figure) is: cascaded layer → ResBlock → cascaded layer → ResBlock → cascaded layer → ResBlock → Up-sizing Conv.

[0328] The second Up layer (the first Up2 in the attached figure) and the third Up layer (the second Up2 in the attached figure) in the first U-net unit and the second U-net unit each include three cascaded layers, three ResBlocks, three cross two-image attention layers, and one Up-sizing Conv.

[0329] The connection order of each layer in these two Up layers (the first Up2 in the attached diagram and the second Up2 in the attached diagram) is as follows: cascaded layer → ResBlock → cross two-image attention → cascaded layer → ResBlock → cross two-image attention → cascaded layer → ResBlock → cross attention layer → Up-sizing Conv.

[0330] The fourth Up layer (Up3 in the attached diagram) in the first and second U-net units includes three cascaded layers, three ResBlocks, and three cross two-image attention layers. The connection order of the layers in this Up layer (Up3 in the attached diagram) is as follows: cascaded layer → ResBlock → cross two-image attention → cascaded layer → ResBlock → cross two-image attention → cascaded layer → ResBlock → cross two-image attention.

[0331] Therefore, compared to the traditional stable diffusion model, the image-to-image SD diffusion model in this embodiment mainly changes the cross-attention mechanism to an image-to-image cross-attention mechanism. Meanwhile, to facilitate simultaneous processing of the target ambient light image, the image-to-image SD diffusion model consists of two parallel SD structures, which are connected based on the image cross-attention mechanism layer corresponding to the aforementioned image-to-image cross-attention mechanism.

[0332] Therefore, this application embodiment utilizes stable diffusion technology to achieve image relighting. By inputting the target ambient light image and the first image with the first lighting effect into the image-generated image SD model, a second image with the second lighting effect can be obtained, thereby improving image quality and image effect, obtaining an image that satisfies the user, and ensuring user experience.

[0333] In some embodiments, the specific implementation of the structure of the cross two-image attention layer and the cross attention mechanism can adopt any existing cross attention mechanism capable of cross-processing two or more images, and this application embodiment does not limit it in any way.

[0334] Figure 18 A schematic diagram of a cross two-image attention layer is shown.

[0335] like Figure 18 As shown, since the cross-two-image attention layer in this embodiment connects the first U-net unit and the second U-net unit, it has two inputs and two outputs. Therefore, the cross-two-image attention layer in this embodiment also includes two branches.

[0336] like Figure 18 As shown, each branch in the cross-two-image attention layer includes two linear layers and one attention mechanism layer. That is, the cross-two-image attention layer in this embodiment includes a total of four linear layers and two attention mechanisms. The four linear layers include a first linear layer, a second linear layer, a third linear layer, and a fourth linear layer. The two attention mechanisms include a first attention mechanism and a second attention mechanism.

[0337] like Figure 18The diagram shows the first linear layer, the second linear layer, the third linear layer, the fourth linear layer, the first attention mechanism layer, and the second attention mechanism layer.

[0338] Linear is a commonly used layer in neural networks, a fundamental layer also known as a fully connected layer or dense layer. A Linear layer typically accepts an input tensor and computes the output tensor through matrix multiplication and the addition of a bias vector.

[0339] Specifically, the inputs to the first and second linear layers are the outputs of the preceding layer in the cross-two-image attention layer. (See reference) Figure 17 The preceding layer of the cross two-image attention layer is mostly a ResBlock, so the input of the first and second Linear can be the output of the ResBlock.

[0340] The input to the first attention mechanism is the output of the first linear and the output of the second linear. Similarly, the input to the second attention mechanism is the output of the first linear and the output of the second linear. Both the first and second linears have three outputs, each corresponding one-to-one with the parameters required by the cross-attention mechanism, such as query (Q), key (K), and value (V).

[0341] Understandably, the number of outputs of the Linear layer can be configured according to actual needs. For example, since the output of the Linear layer is the input of the Attention, the Linear layer needs to have three outputs. Therefore, the output calculated by the Linear layer can be split into three parts and then transmitted to the Attention separately. Alternatively, three sub-Linear layers can be configured directly in the Linear layer to obtain three outputs that are transmitted to the Attention separately.

[0342] Specifically, such as Figure 18 As shown, the Q of the first Attention is the output of the first Linear, and the K and V of the first Attention are the outputs of the second Linear. The Q of the second Attention is the output of the second Linear, and the K and V of the second Attention are the outputs of the first Linear. Then, the first and second Attentions perform cross-processing based on the outputs of the first and second Linears, respectively, to achieve the effect of fusing the first image and the target ambient light image.

[0343] The input to the third line is the output of the first attention layer, and the input to the fourth line is the output of the second attention layer. The outputs of the third and fourth lines are the inputs to the next layer in the cross-two-image attention process. For example, see reference... Figure 17 In the Down layer, the layer following the cross-two-image attention layer can be either a ResBlock or a Down-sizing Conv. Therefore, the outputs of the third and fourth Linear layers can be either the inputs to the ResBlock or the inputs to the Down-sizing Conv. For example, see reference... Figure 17 In the Up layer, the layer following the cross-two-image attention layer can be a cascaded layer, an Up-sizing Conv, or a Conv. Therefore, the outputs of the third and fourth Linear layers can be the inputs of the cascaded layer, the Up-sizing Conv, or the Conv.

[0344] The specific processing of the crossover between the first Attention and the second Attention can be referred to the calculation of existing Attention methods. The principle of the embodiments in this application is the same, so it will not be repeated here.

[0345] Therefore, the embodiments of this application can perform cross-processing on the first image and the target ambient light image through the cross two-image attention layer, thereby fusing the image features of the first image and the target ambient light image. This allows the image-to-image SD model to reconstruct the first lighting effect of the first image under the guidance or traction of the target ambient light image, thereby obtaining a second image with a second lighting effect.

[0346] In some embodiments, since the input of the stable diffusion model usually requires adding noise to the image, both the first image and the target ambient light image are input to the U-net unit after being denoised by the denoising module.

[0347] However, if the target ambient light image is subjected to strong noise and then input into the second U-net unit, the noise may affect the judgment and learning of the lighting effect in the target ambient light image by the n U-net units connected in series in the first U-net branch, which may result in the inaccurate generation of a second image with the same lighting effect as the target ambient light image.

[0348] Therefore, in order for the first U-net branch to accurately offset and reconstruct the lighting effect of the first image with the first lighting effect based on the target ambient light image, so as to generate the second image with the second lighting effect, the embodiments of this application can add noise to the target ambient light image at different intensities in descending order of intensity. Then, each noisy target ambient light image is also input to n unconnected second U-net units in descending order of noise intensity.

[0349] Figure 19 A flowchart of an image processing method is shown.

[0350] like Figure 19 As shown, the n first U-net units in the first U-net branch, which is the U-net branch used to process the first image, are still connected. Therefore, the output of each first U-net unit in the first U-net branch is transmitted to the next layer's first U-net unit as its input.

[0351] However, the n second U-net units in the second U-net branch, which is used to process the target ambient light image, are still unconnected. That is, the output of each second U-net unit in the second U-net branch is not transmitted to the next layer of second U-net units. For example... Figure 19 As shown, the input of each second U-net unit in the second U-net branch is only a noisy target latent image with different levels of noise.

[0352] Specifically, after the target ambient light image is processed by an image encoder to extract latent features and obtain the corresponding target latent image, the target latent image is input into the next layer's noise-adding module. Then, the noise-adding module adds noise to the target latent image to different degrees to obtain the corresponding noisy target latent image.

[0353] like Figure 19 As shown, the second noise-adding module adds noise to the target latent image including noise addition t1, noise addition t2, ..., noise addition tn. Here, t1, t2, ..., tn are noise addition ratios, representing different levels of noise addition. Furthermore, t1 > t2 > ... > tn, indicating a decreasing degree of noise addition. That is, the noise addition ratio of noise addition t1 is greater than that of noise addition t2, and the corresponding noise in the noisy target latent image t1 is stronger than that in the noisy target latent image t2. Similarly, the noise addition ratio of noise addition t2 is greater than that of noise addition tn, and the corresponding noise in the noisy target latent image t2 is stronger than that in the noisy target latent image tn. In other words, noise addition t1 has the strongest noise addition level, and noise addition tn has the weakest noise addition level.

[0354] In some embodiments, the image noise addition process can employ any existing noise addition method, such as Gaussian noise, salt-and-pepper noise, white noise, etc. Taking Gaussian noise as an example, the noise addition ratios from strongest to weakest can be 1, 0.5, 0.2, 0.1. For example, t1 = 1, t2 = 0.5, ..., t4 = 0.1. Or, t1 = 0.5, t2 = 0.2, ..., t4 = 0.1.

[0355] Next, following the cascading order of the n cascaded first U-net units in the first U-net branch, the n noisy target latent images are input to these n unconnected second U-net units in order of decreasing noise level.

[0356] like Figure 19 As shown, the latent image t1 of the noisy target with the largest noise intensity is input to the first second U-net unit, the latent image t2 of the noisy target with the second largest noise intensity is input to the second second U-net unit, and so on, the latent image tn of the noisy target with the weakest noise intensity is input to the last second U-net unit.

[0357] Therefore, the number of noisy target latent images corresponding to the target ambient light image is equal to the number of the second U-net units, meaning that n second U-net units correspond to n noisy target latent images with different levels of noise. Simultaneously, the number of second U-net units is equal to the number of first U-net units. Since the number of second U-net units and the number of first U-net units are both n, the graph-generated image SD model has a total of 2n U-net units.

[0358] The first image, after having its latent features extracted by the image encoder to obtain the corresponding first latent image, is also input into the next layer's noise-adding module. However, the difference lies in... Figure 19 As shown, the noise-adding module in this branch will only add noise to the first latent image to obtain a corresponding noisy first latent image, and then input this noisy first latent image to the first U-net unit.

[0359] The noise addition ratio of the first image can be set according to actual needs, and this application embodiment does not impose any limitation on it. For example, the first latent image can be noise-added according to the noise addition ratio t1.

[0360] Thus, in this embodiment of the application, the n first U-net units of the first U-net branch can continuously influence the reconstruction of the first image of the first illumination effect by the first U-net unit based on the target ambient light image with different noise levels, thereby improving the reconstruction effect and accurately obtaining the second image of the second illumination effect.

[0361] In addition, according to Figure 17 As can be seen from the structure of the U-net unit shown, the principle of the first U-net unit and the second U-net unit is actually to perform residual processing on the two images, that is, to predict the noise of the image through residual processing, and thus output the first predicted noise corresponding to the first image.

[0362] Therefore, the first predicted noise predicted by the first U-net unit is actually an unwanted residual result, so the denoising module needs to remove the predicted noise from the original image. That is, as shown... Figure 19 and Figure 15 As shown, the denoising module, in addition to receiving the first predicted noise output from the first U-net branch, also needs to receive the noisy first latent image after image encoding and noise addition. Then, the denoising module subtracts the first predicted noise image from the noisy first latent image, thereby removing the unwanted predicted noise from the noisy first latent image. The result is then output to the image decoder to reconstruct the image, thus obtaining the second image with the second lighting effect. Therefore, since the first predicted noise image is a noise image predicted after fusing the target ambient light image, removing this first predicted noise from the noisy first latent image yields a second image with a lighting effect corresponding to the target ambient light image.

[0363] In some embodiments, different first images may include different types of objects, such as faces, animals, plants, etc. Furthermore, the lighting requirements may differ for different types of objects. Therefore, to reconstruct lighting effects that better suit the specific object, this application embodiment may additionally add a ControlNet to provide additional control over the reconstruction of the image-to-image SD model. For example, when the object in the first image includes a face, the ControlNet can additionally input a face semantic segmentation image or a face depth feature map into the image-to-image SD model. The specific configuration can be tailored to actual needs, and this application embodiment does not impose any limitations on this.

[0364] For example, Figure 20 A schematic diagram of a connection between ControlNet and a graph-based SD model is shown.

[0365] like Figure 20 As shown, the output of ControlNet ( Figure 20 The dashed lines in the text can be used to jump and transmit the output of the Down layer at the front end of the U-net unit to the Up layer at the back end, and input to the cascaded layer in the Up layer at the back end.

[0366] This can be understood as follows: Before adding ControlNet, the inputs of the cascaded layers in the U-net unit's backend Up layer only include output A transmitted from the previous layer and output B transmitted from the frontend Down layer. At this time, the inputs of the cascaded layers only include output A and output B. However, after adding ControlNet, the inputs of the cascaded layers in the U-net unit's backend Up layer, in addition to the aforementioned outputs A and B, also include the ControlNet output C. At this time, the inputs of the cascaded layers include outputs A, B, and C.

[0367] It should be noted that the ControlNet structure in the embodiments of this application can adopt any existing structure. For example, the ControlNet in the embodiments of this application can directly adopt the ControlNet used in the open-source stable diffusion model.

[0368] Additionally, due to space limitations of the attached images, Figure 20 The ControlNet is illustrated only on the U-net unit of one branch. However, it is understandable that a ControlNet can also be added to the U-net unit of the other branch. That is, a ControlNet can be set up, and the output of this ControlNet can be transmitted to n cascaded first U-net units and / or n unconnected second U-net units respectively. Alternatively, a first ControlNet and / or a second ControlNet can be set up simultaneously. Then, the output of the first ControlNet is transmitted to the n cascaded first U-net units, and the output of the second ControlNet is transmitted to the n unconnected second U-net units.

[0369] Finally, since the aforementioned image-to-image SD model needs to be trained before it can be deployed to electronic devices, the electronic devices also need to perform pre-training on the image-to-image SD model related to relighting. The following embodiments of this application describe the training process of the aforementioned image-to-image SD model.

[0370] In this embodiment, the training data for the image-to-image SD model can be obtained by pre-capturing a large number of different images. For ease of distinction, the images and ambient light images required for training will be referred to as the first training image and the training ambient light image. That is, the training data mainly includes the first training image and the training ambient light image.

[0371] In some embodiments, the electronic device can also generate a large number of images as training first images and training ambient light images using open-source image generation methods. For example, the electronic device can generate images as training ambient light images using the open-source illumination estimation technique (DiffusionLight).

[0372] After the training data is determined, the electronic device can begin training the graph-to-graph SD model. In this embodiment, the training of the graph-to-graph SD model mainly includes two stages: pre-training the weights and formal training. After the pre-training stage is completed, the electronic device can begin formal training.

[0373] The pre-training weights primarily determine the weights of certain layers in the image-to-image SD model, including the image encoder, image decoder, and U-net units (first U-net unit and second U-net unit). In some embodiments, the weights determined during the pre-training weight stage can be either final, fixed weights or initial weights. If the weights are fixed, they will not be updated during the formal training phase. If they are initial weights, they can be updated during the formal training phase. For example, the weights of the image encoder and image decoder can be determined as fixed weights during the pre-training weight stage, while the weights of the U-net units can be initial weights.

[0374] In the image-to-image SD model, the weights of the image encoder and image decoder can be determined by training with image data. Alternatively, open-source weights can be used. For example, regarding the weights of the image encoder and image decoder, in this embodiment, collected image data can be used for image encoding and decoding training, and the weights of the image encoder and image decoder can be determined and fixed through this training.

[0375] Alternatively, the image encoder and image decoder can also use open-source weights. Open-source weights refer to the weights of the open-source image encoder and the open-source image decoder. For example, the image encoder and image decoder are collectively referred to as autoencoders in open source. Therefore, embodiments of this application can also avoid additional training of the image encoder and image decoder, and instead directly use any open-source autoencoder weights.

[0376] Meanwhile, the U-net units (first U-net unit and second U-net unit) can also use an open-source stable diffusion model as initial weights. For example, the weights of the U-net units in the graph-to-graph SD model in this embodiment can be used to initialize the weights of the U-net units, giving them an initial weight. In a specific embodiment, the U-net unit includes ResBlock and Conv. Since both ResBlock and Conv have relatively mature open-source structures, the initial weights of the ResBlock in the U-net unit can be the weights of the open-source ResBlock, and the initial weights of the Conv can also be the weights of the open-source Conv.

[0377] Then, during formal training, these initial weights can be further updated to determine and fix the final weights. For example, the initial weights of U-net units can be updated during formal training to determine and fix the final required weights.

[0378] In other words, after the pre-training weights stage is completed, each layer in the image-to-image SD model has defined weights. Then, the electronic device can begin training this image-to-image SD model using the constructed first training image and the training ambient light image. This training process is the formal training. During this formal training process, the image-to-image SD model will also output the corresponding second training image based on the first training image and the training ambient light image.

[0379] Understandably, the process of outputting the second training image during the formal training phase is roughly the same as the process of outputting the second image based on the first image and the target ambient light image in actual application after model deployment.

[0380] For example, an electronic device can input a first training image and a training ambient light image into a graph-based SD model, which then outputs a second training image based on these two images. Since the graph-based SD model includes an image cross-attention mechanism, this mechanism layer can perform cross-processing between the first training image and the training ambient light image. This allows the graph-based stable diffusion model to re-illuminate the first training image under the guidance of the training ambient light image, resulting in the second training image.

[0381] More specifically, because the graph-native-image SD model includes two parallel branches, as mentioned above, the U-net structure in the graph-native-image SD model includes a first U-net branch and a second U-net branch. The first U-net branch includes n cascaded first U-net units, and the second U-net unit includes n unconnected second U-net units. Therefore, during the formal training phase, the first training image can be image-encoded and noise-added to obtain the corresponding noisy first latent image; and the training ambient light image can be image-encoded and noise-added to obtain the corresponding noisy ambient light latent image. Then, the noisy ambient light latent image is transmitted to each of the n second U-net units. The n cascaded first U-net units in the first U-net branch can then obtain the noisy ambient light latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer. Simultaneously, the training noisy first latent image is transmitted to the first U-net branch. The first U-net unit in this branch can then perform cross-processing on the training noisy first latent image and the training noisy ambient light latent image. This allows the training noisy first latent image to be fused with the image information (including illumination information) of the training noisy ambient light latent image, predicting the noise in the training noisy first latent image. The first U-net branch iteratively performs n cross-processing and noise predictions through n cascaded first U-net units, outputting a training first predicted noisy image. Then, the training noisy first latent image is denoised based on the training first predicted noisy image, and the denoised training noisy first latent image is decoded to output a training second image. Image encoding of the training first image and the training ambient light image can be implemented by the first encoder and the second encoder in the image encoder. Noisy addition of the training first image and the training ambient light image can be implemented by the first noise addition module and the second noise addition module, while image decoding can be implemented by the image decoder.

[0382] For example, during the formal training phase, the electronic device can also add noise to the training ambient light image to varying degrees before feeding it to each of the n disconnected second U-net units. Therefore, the specific process of outputting the second training image during the formal training phase can be found in the description of the output of the second image in the above embodiments, and will not be repeated here.

[0383] However, it should be noted that the formal training phase differs from the actual application after model deployment described in the above embodiments. In this training process, the electronic device further determines the model's relighting loss based on the first training image, the training ambient light image, and the output second training image. Then, the weights of the U-net units are adjusted based on the loss until the training termination condition is met.

[0384] For example, the training termination condition could be a fixed number of training iterations. Therefore, the weights initialized in the pre-training stage can be updated through continuous iterations. After the preset number of training iterations is reached, training ends, and the final updated weights are fixed, thus completing the training of the image-to-image SD model. For instance, if the training iterations are three times, after the first training iteration, the initial weights of U-net are updated to the first weight. After the second training iteration, the first weight is updated to the second weight. After the third training iteration, the second weight is updated to the third weight. Thus, the third weight is the final fixed weight. The trained image-to-image SD model can then be deployed to electronic devices for image relighting.

[0385] In this embodiment, since the image-to-image SD model includes two parallel branches, the loss is the sum of the losses of the two stable diffusion branches. That is, loss = loss_img + loss_light. Here, loss_img is the loss value of the branch that primarily processes the training of the first image, i.e., the loss value of the first stable diffusion branch (first U-net branch). loss_light is the loss value of the branch that primarily processes the training of the ambient light image, i.e., the loss value of the second stable diffusion branch (second U-net branch). It is understood that the loss function used in this embodiment can be any existing loss function. For example, the loss function can be the LDM function used in the open-source stable diffusion model; this embodiment does not impose any limitations on this.

[0386] In some embodiments, the electronic device may also re-illuminate the first image based on multiple target ambient light candidate images. For example, it may include i target ambient light candidate images, where i is an integer and i≥1. Then, the above-described image-to-image SD model is configured with i+1 parallel stable diffusion models. It can be understood that when i=1, the above-described image-to-image SD model includes a first stable diffusion branch and a second stable diffusion branch. The first image is input to the first stable diffusion branch, and the target ambient light image is input to the second stable diffusion branch. The first stable diffusion branch obtains the image information of the target ambient light image from the second stable diffusion branch through an image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the target ambient light image.

[0387] When i > 1, the above-described image-generated image SD model includes a first stable diffusion branch and i second stable diffusion branches. The first image is still input to the first stable diffusion branch. The i target ambient light images are then input to the i second stable diffusion branches one by one. That is, one second stable diffusion branch corresponds to one target ambient light image. Then, the first stable diffusion branch obtains the image information of the i > 1 target ambient light images from each of the i > 1 second stable diffusion branches through an image cross-attention mechanism layer, causing the first stable diffusion branch to output the second image under the guidance of these i > 1 target ambient light images.

[0388] It should be noted that, in the embodiments of this application, the specific structures of the first stable diffusion branch and the i second stable diffusion branches can be referred to the above description of the first and second stable diffusion branches. The structure and principle are the same, and this application will not repeat them here. Therefore, the electronic device of this application embodiment can reconstruct the first illumination effect of the first image under the guidance of i target ambient light images through the image-generated image SD model, thereby outputting a second image with a second illumination effect, improving image quality and image effect, obtaining an image that satisfies the user, and ensuring user experience.

[0389] Another embodiment of this application provides an electronic device, including: a display screen, a camera, one or more processors, and a memory. The display screen, camera, and memory are respectively coupled to the processor; the display screen is used to display images; the camera is used to capture images; the memory stores one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, the electronic device implements the image processing method described in any of the above embodiments. In a specific embodiment, the electronic device in this application can be a mobile phone, tablet computer, etc.

[0390] Another embodiment of this application provides an electronic device, including: one or more processors and a memory, the memory being coupled to the processors; the memory storing one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, the electronic device causes the electronic device to implement the training method of the graph-based image SD model described in any of the above embodiments. In a specific embodiment, the electronic device in this application may be a server.

[0391] Another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor in an electronic device, causes the electronic device to implement the image processing method described in any of the above embodiments.

[0392] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps described in the above method embodiments.

[0393] This application also provides a chip system, such as... Figure 21 As shown, the chip system includes at least one processor 2101 and at least one interface circuit 2102. The processor 2101 and the interface circuit 2102 are interconnected via lines. For example, the interface circuit 2102 can be used to receive signals from other devices (e.g., a computer's memory). As another example, the interface circuit 2102 can be used to send signals to other devices (e.g., the processor 2101).

[0394] For example, interface circuit 2102 can read instructions stored in memory and send those instructions to processor 2101. When the instructions are executed by processor 2101, the computer can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0395] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0396] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0397] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0398] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0399] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0400] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, Applied to an electronic device, the electronic device including a display screen; the electronic device having a trained stable diffusion model of an image-generated graph deployed within it; the method includes: Display a first image with the first lighting effect on the first interface; The system receives a first operation from the user on the first interface; wherein the first operation is used to trigger the electronic device to relight the image. In response to the first operation, the stable diffusion model of the image-generated image outputs a second image with a second illumination effect based on the first image and the target ambient light image; wherein, the stable diffusion model of the image-generated image includes an image cross-attention mechanism layer; the stable diffusion model of the image-generated image performs cross-processing between the first image and the target ambient light image through the image cross-attention mechanism layer, so that the stable diffusion model of the image-generated image re-illuminates the first image under the guidance of the target ambient light image to obtain the second image; wherein, the second illumination effect corresponds to the illumination effect of the target ambient light image; The second image with the second lighting effect is displayed on the first interface; The stable diffusion model of the graph includes a first stable diffusion branch and a second stable diffusion branch, which are interconnected through the image cross-attention mechanism layer. The image cross-attention mechanism layer includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer, a first attention mechanism layer, and a second attention mechanism layer. The input of the first linear layer is obtained from the first stable diffusion branch; the input of the second linear layer is obtained from the second stable diffusion branch; the inputs of the first attention mechanism layer and the second attention mechanism layer are the outputs of the first linear layer and the second linear layer, respectively; the input of the third linear layer is the output of the first attention mechanism layer; and the input of the fourth linear layer is the output of the second attention mechanism layer.

2. The method of claim 1, wherein, The first interface is a large image display interface of the image library in the electronic device, and the first image is an image in the image library of the electronic device; the second image is obtained by relighting the first image.

3. The method of claim 1, wherein, The electronic device also includes a camera; The first interface is the camera preview interface in the electronic device, the first image is the image captured in real time by the camera, and the second image is obtained by relighting the image captured in real time by the camera. The method further includes: saving the second image in response to a user's shooting operation on the first interface; wherein the shooting operation is performed after the first operation.

4. The method of claim 1, wherein, The electronic device also includes a camera; The first interface is the recording interface of the camera in the electronic device, the first image is the image captured by the camera in real time, and the second image is obtained by relighting the image captured by the camera in real time. The method further includes: In response to the user's start recording operation on the first interface, multiple second images are continuously acquired; in response to the user's end recording operation on the first interface, the multiple second images are saved as a video; wherein, the start recording operation is performed after the first operation.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The first interface displays multiple candidate ambient light images; wherein, the candidate ambient light images include pre-configured ambient light images and / or user-specified ambient light images; The second image, responding to the first operation and outputting a second illumination effect based on the first image and the target ambient light image using a stable diffusion model of the generated image, includes: Receive the first operation, and use the candidate ambient light image corresponding to the first operation as the target ambient light image; Based on the stable diffusion model of the image-generated image, a second image with a second illumination effect is output based on the first image and the target ambient light image.

6. The method according to any one of claims 1-4, characterized in that, The stable diffusion model derived from the image-generated image outputs a second image with a second illumination effect based on the first image and the target ambient light image, including: The first image is input into the first stable diffusion branch, and the target ambient light image is input into the second stable diffusion branch. The first stable diffusion branch obtains the image information of the target ambient light image from the second stable diffusion branch through the image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the target ambient light image; wherein, the image information includes illumination information.

7. The method of claim 6, wherein, The first stable diffusion branch includes a first U-net branch, and the second stable diffusion branch includes a second U-net branch; wherein, the first U-net branch includes n first U-net units connected in series, and the second U-net branch includes n second U-net units that are not connected. The n cascaded first U-net units all include the image cross-attention mechanism layer, and the n unconnected second U-net units also include the image cross-attention mechanism layer. Furthermore, the n cascaded first U-net units and the n unconnected second U-net units are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1; The step of inputting the first image into the first stable diffusion branch and inputting the target ambient light image into the second stable diffusion branch, wherein the first stable diffusion branch obtains image information of the target ambient light image from the second stable diffusion branch through the image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the target ambient light image, includes: The first image is input into the first stable diffusion branch, and the first image is encoded and denoised through the first stable diffusion branch to obtain the corresponding denoised first potential image. The target ambient light image is input into the second stable diffusion branch, and the target ambient light image is image encoded and noise-added through the second stable diffusion branch to obtain the corresponding noisy target potential image. The noisy target latent image is transmitted to the n second U-net units respectively. The n cascaded first U-net units in the first U-net branch respectively obtain the noisy target latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer. The noisy first latent image is transmitted to the first U-net branch. The first U-net unit in the first U-net branch performs cross-processing on the noisy first latent image and the noisy target latent image, so that the noisy first latent image is fused with the image information of the noisy target latent image, and the noise of the noisy first latent image is predicted; wherein, the image information includes illumination information; The first U-net branch outputs a first predicted noise image after iteratively performing n cross-processing and noise prediction through the n cascaded first U-net units. The first noisy potential image is denoised based on the first predicted noisy image, and the denoised first noisy potential image is then image decoded to output the second image.

8. The method of claim 7, wherein, The second stable diffusion branch performs image encoding and noise addition on the target ambient light image to obtain a corresponding noisy target latent image, including: Image encoding is performed on the target ambient light image to obtain a target latent image, and noise is added to the target latent image n times with different intensities to obtain n noisy target latent images; The step of transmitting the noisy target latent image to each of the second U-net units, wherein the n cascaded first U-net units in the first U-net branch respectively obtain the noisy target latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer, includes: The n noisy target potential images are transmitted one-to-one to the n unconnected second U-net units. The n serially connected first U-net units in the first U-net branch obtain the n noisy target potential images from the corresponding connected second U-net units through the image cross-attention mechanism layer. In this series, according to the connection order, the noise intensity of the noisy target potential image obtained by the n series first U-net units is distributed from strong to weak. The noise intensity of the noisy target potential image obtained by the first first U-net unit is the strongest, and the noise intensity of the noisy target potential image obtained by the last first U-net unit is the weakest.

9. The method of claim 7, wherein, The first stable diffusion branch further includes a first image encoder, a first noise-adding module, a noise-reducing module, and an image decoder; Wherein, the first image encoder is used to encode the first image to obtain a first latent image; the first noise-adding module is used to add noise to the first latent image to obtain the noisy first latent image; The denoising module is used to denoise the noisy first potential image based on the first predicted noise image; the image decoder is used to perform image decoding on the denoised noisy first potential image and output the second image.

10. The method according to any one of claims 7-9, characterized in that, The second stable diffusion branch also includes a second image encoder and a second noise-adding module; The second image encoder is used to encode the target ambient light image to obtain a target latent image; the second noise-adding module is used to add noise to the target latent image to obtain a noisy target latent image.

11. The method according to any one of claims 7-9, characterized in that, The stable diffusion model of the graph-generated graph also includes a control network; wherein the output of the control network is transmitted to the cascaded layers in the U-net unit; the U-net unit includes the n first U-net units connected in series and / or the n second U-net units not connected.

12. The method according to any one of claims 7-9, characterized in that, The step of denoising the noisy first latent image based on the first predicted noise image, and then performing image decoding on the denoised noisy first latent image to output the second image includes: The difference between two corresponding pixel values ​​in the noisy first potential image and the first predicted noise image is calculated to obtain the denoised first potential image. The first potential image to be denoised is decoded to output the second image.

13. The method according to any one of claims 1-4, characterized in that, The target ambient light image includes i target ambient light images; the stable diffusion model of the graph includes i+1 parallel stable diffusion branches; The i+1 parallel stable diffusion branches include a first stable diffusion branch and i second stable diffusion branches; both the first stable diffusion branch and the i second stable diffusion branches include the image cross-attention mechanism layer and are interconnected through the image cross-attention mechanism layer; i is an integer, i≥1; The stable diffusion model derived from the image-generated image outputs a second image with a second illumination effect based on the first image and the target ambient light image, including: The first image is input into the first stable diffusion branch, and the i target ambient light images are input one-to-one into the i second stable diffusion branches. The first stable diffusion branch obtains the image information of the i target ambient light images from the i second stable diffusion branches through the image cross-attention mechanism layer, so that the first stable diffusion branch outputs the second image under the guidance of the i target ambient light images; wherein, the image information includes illumination information.

14. A training method for a stable diffusion model of a graph-generated graph, characterized in that, The stable diffusion model for the graph-generated image includes a U-net structure; wherein the U-net structure includes an image cross-attention mechanism layer; the training method includes: Acquire the first training image and the training ambient light image; Initialize the weights of the U-net structure; The first training image and the training ambient light image are input into the stable diffusion model of the image-generated image. The stable diffusion model of the image-generated image outputs a second training image based on the first training image and the training ambient light image. The stable diffusion model of the image-generated image performs cross-processing between the first training image and the training ambient light image through the image cross-attention mechanism layer, so that the stable diffusion model of the image-generated image re-lights the first training image under the guidance of the training ambient light image to obtain the second training image. Using the first training image, the training ambient light image, and the second training image, the loss of the stable diffusion model of the image-generated image is determined by the relighting loss caused by the training ambient light image. The weights of the U-net structure are adjusted based on the loss until the training termination condition is met, thus obtaining the trained stable diffusion model of the image-generated image. The stable diffusion model of the graph-generated image includes a first stable diffusion branch and a second stable diffusion branch. Both the first and second stable diffusion branches include the U-net structure and are interconnected through an image cross-attention mechanism layer within the U-net structure. The image cross-attention mechanism layer includes a first linear layer, a second linear layer, a third linear layer, a fourth linear layer, a first attention mechanism layer, and a second attention mechanism layer. The input of the first linear layer is obtained from the first stable diffusion branch; the input of the second linear layer is obtained from the second stable diffusion branch; the inputs of the first and second attention mechanism layers are the outputs of the first and second linear layers, respectively; the input of the third linear layer is the output of the first attention mechanism layer; and the input of the fourth linear layer is the output of the second attention mechanism layer.

15. The method according to claim 14, characterized in that, The U-net structure includes a first U-net branch and a second U-net branch; the first U-net branch includes n first U-net units connected in series, and the second U-net branch includes n second U-net units that are not connected. Wherein, the n cascaded first U-net units all include the image cross-attention mechanism, and the n unconnected second U-net units also all include the image cross-attention mechanism layer; and the n cascaded first U-net units and the n unconnected second U-nets are connected one-to-one through the image cross-attention mechanism layer; n is an integer, n≥1; The step of inputting the first training image and the training ambient light image into the stable diffusion model of the image-generated image, and outputting a second training image based on the first training image and the training ambient light image from the stable diffusion model of the image-generated image, includes: The training first image is image encoded and noise-added to obtain a corresponding training noisy first latent image; and the training ambient light image is image encoded and noise-added to obtain a corresponding training noisy ambient light latent image. The training noisy ambient light latent image is transmitted to the n second U-net units respectively. The n cascaded first U-net units in the first U-net branch obtain the training noisy ambient light latent image from the corresponding connected second U-net unit through the image cross-attention mechanism layer. The training noisy first latent image is transmitted to the first U-net branch. The first U-net unit in the first U-net branch performs cross-processing on the training noisy first latent image and the training noisy ambient light latent image, so that after the training noisy first latent image is fused with the image information of the training noisy ambient light latent image, the noise of the training noisy first latent image is predicted; wherein, the image information includes illumination information; The first U-net branch outputs a training first predicted noise image after iteratively performing n cross-processing and noise prediction through the n cascaded first U-net units. The training noisy first latent image is denoised based on the training first predicted noisy image, and the denoised training noisy first latent image is image decoded to output the training second image.

16. The method according to claim 15, characterized in that, The stable diffusion model of the graph-generated image further includes an image encoder and an image decoder; wherein the image encoder includes a first image encoder and a second image encoder; The first image encoder is used to encode the training first image; the image decoder is used to decode the denoised training noisy first latent image; the second image encoder is used to encode the training ambient light image. The method further includes: The weights of the image encoder and the image decoder are determined and fixed; wherein the weights of the image encoder and / or the image decoder are open-source weights; or, the weights of the image encoder and / or the image decoder are determined through training; wherein the weights of the first image encoder and the second image encoder are the same or different.

17. The method according to any one of claims 14-16, characterized in that, The initialization of the weights of the U-net structure includes: obtaining the open-source weights corresponding to the U-net structure in the open-source stable diffusion model, and initializing the weights of the U-net structure according to the open-source weights.

18. An electronic device, characterized in that, include: The display screen, camera, one or more processors, and memory, wherein the display screen, the camera, and the memory are respectively coupled to the processor; The display screen is used to display images, and the camera is used to capture images; the memory stores one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, the electronic device performs the image processing method as described in any one of claims 1-13.

19. A computer device, characterized in that, include: One or more processors and a memory, the memory being coupled to the processors; the memory storing one or more computer program codes, the computer program codes including computer instructions; when the processor executes the computer instructions, causing the computer device to perform a training method for a stable diffusion model of a graph as described in any one of claims 14-17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor of the electronic device, the electronic device performs the image processing method as described in any one of claims 1-13 and / or performs the training method for a stable diffusion model of an image as described in any one of claims 14-17.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor in an electronic device, the electronic device performs the image processing method as described in any one of claims 1-13 and / or the training method for a stable diffusion model of an image as described in any one of claims 14-17.

Images