Image processing method, training method, device and equipment of deep learning model

Abstract

This disclosure provides an image processing method, relating to the field of artificial intelligence technology, particularly augmented reality, virtual reality, computer vision, and deep learning, applicable to scenarios such as metaverse and virtual digital humans. The specific implementation involves: encoding the initial luminance channel image of the original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result includes first luminance image features; obtaining a target luminance channel image based on the target spherical harmonic coefficients and the first encoding result; and fusing the target luminance channel image with the initial chroma channel image of the original image to obtain the target image. This disclosure also provides a training method, apparatus, electronic device, and storage medium for a deep learning model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to augmented reality, virtual reality, computer vision, and deep learning technologies, and can be applied to scenarios such as the metaverse and virtual digital humans. More specifically, this disclosure provides an image processing method, a deep learning model training method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence technology, image processing technology is constantly being improved. For example, based on artificial intelligence technology, the lighting effects of objects in the original image can be adjusted. Summary of the Invention

[0003] This disclosure provides an image processing method, a deep learning model training method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, the method comprising: encoding an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result includes first luminance image features; obtaining a target luminance channel image based on a target spherical harmonic coefficient and the first encoding result; and fusing the target luminance channel image with an initial chroma channel image of the original image to obtain a target image.

[0005] According to another aspect of this disclosure, a method for training a deep learning model is provided. The deep learning model includes an encoding network and a decoding network. The method includes: inputting an initial luminance channel sample image of a sample image into the encoding network to obtain a first sample encoding result of the initial luminance channel sample image, wherein the first sample encoding result includes first sample spherical harmonic coefficients; inputting the first sample encoding result into the decoding network to obtain an output luminance channel image; fusing the output luminance channel image and the initial chroma channel sample image of the sample image to obtain an output image; and training the deep learning model based on at least one of the output image, the output luminance channel image, and the first sample spherical harmonic coefficients.

[0006] According to another aspect of this disclosure, an image processing apparatus is provided, the apparatus comprising: an encoding module for encoding an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result includes first luminance image features; a first obtaining module for obtaining a target luminance channel image based on a target spherical harmonic coefficient and the first encoding result; and a first fusing module for fusing the target luminance channel image with an initial chroma channel image of the original image to obtain a target image.

[0007] According to another aspect of this disclosure, a training apparatus for a deep learning model is provided, the apparatus comprising: an encoding module for encoding an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result includes first luminance image features; a first obtaining module for obtaining a target luminance channel image based on a target spherical harmonic coefficient and the first encoding result; and a first fusion module for fusing the target luminance channel image with an initial chroma channel image of the original image to obtain a target image.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided according to this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of a deep learning model according to an embodiment of the present disclosure;

[0015] Figure 3 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram of a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 5 This is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0019] Figure 7 This is a block diagram of an electronic device according to an embodiment of the present disclosure, to which image processing methods and / or training methods for deep learning models can be applied. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In facial image processing, the lighting conditions can be adjusted to achieve balanced illumination. Poor exposure or backlighting during image acquisition can lead to suboptimal lighting. Consequently, under poor lighting conditions, the accuracy of processing results after facial image detection, key point localization, recognition, or tracking will be low.

[0022] In some embodiments, three-dimensional facial image reconstruction can be performed using two-dimensional facial images. During the reconstruction process, texture maps can be generated based on information from the two-dimensional image. During the rendering stage of the three-dimensional image, lighting parameters can be set to determine the lighting effects of the three-dimensional facial image.

[0023] In some embodiments, a two-dimensional image in the Blue Green Red (BGR) image space can be converted to the YUV or LAB image space. The luminance channel (e.g., Y channel) of the converted two-dimensional image is then equalized. However, equalizing the luminance channel of a two-dimensional image makes it difficult to deeply understand the underlying causes of poor exposure or backlighting. During the equalization process of the luminance channel, chroma-related channels (e.g., U and V channels) may also be processed, potentially altering skin tone. Furthermore, luminance information is difficult to extract from color, making it difficult to effectively improve the image's lighting effect after directly processing the luminance channel of a two-dimensional image.

[0024] Figure 1 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure.

[0025] like Figure 1 As shown, the method 100 may include operations S110 to S140.

[0026] In embodiments of this disclosure, the deep learning model may include an encoder network and a decoder network. For example, the encoder network may be an encoder for a style-based generative adversarial network (StyleGAN) model. The decoder network may be a decoder for a style-based generative adversarial network model.

[0027] In operation S110, the initial brightness channel sample image of the sample image is input into the encoding network to obtain the first sample encoding result of the initial brightness channel sample image.

[0028] In embodiments of this disclosure, the sample image may include a sample object. For example, the sample object may have a face, a head, etc. Alternatively, the sample object may be a person, an animal, a robot, or various other objects.

[0029] In this embodiment of the disclosure, the sample image can be an image in BGR image space or an image in YUV image space. For example, if the sample image is in YUV image space, the Y channel image of the sample image can be used as the initial luminance channel sample image. Alternatively, if the sample image is in BGR image space, the sample image can be converted to YUV image space, and then the Y channel image of the converted sample image can be used as the initial luminance channel sample image.

[0030] In this embodiment of the disclosure, the first sample encoding result may include first sample spherical harmonic coefficients. It is understood that the sample spherical harmonic coefficients can reflect the illumination conditions of the sample image. For example, the first sample spherical harmonic coefficients may be 9-dimensional.

[0031] In operation S120, the first sample encoding result is input into the decoding network to obtain the output luminance channel image.

[0032] For example, a decoding network can decode the encoded result of the first sample. The decoding result may include an output luminance channel image.

[0033] In operation S130, the output luminance channel image and the initial chroma channel sample image of the sample image are fused to obtain the output image.

[0034] For example, the initial chroma channel sample image of a sample image may include the U channel image and V channel image of the sample image. By fusing the initial chroma channel sample image and the output luminance channel image, the output image can be obtained.

[0035] In operation S140, a deep learning model is trained based on at least one of the output image, the output luminance channel image, and the first sample spherical harmonic coefficients.

[0036] In the embodiments disclosed herein, a deep learning model can be trained based on at least one of supervised training methods and unsupervised training methods.

[0037] For example, a sample image can correspond to a label image. The label image can also include the sample objects mentioned above. The sample objects in the sample image and the label image can be the same. The lighting conditions of the sample objects in the sample image can be different from those in the label image. As another example, taking training a deep learning model based on the output image as an example, a loss can be determined based on the output image and the label image. This loss can then be used to adjust the parameters of the deep learning model until the loss converges.

[0038] In this embodiment, the spherical harmonic coefficients of the luminance channel image of a sample image are obtained using an encoding network. These spherical harmonic coefficients are primarily related to the luminance channel image of the sample image, with very low correlation to the chrominance channel images (e.g., U-channel and V-channel images). Therefore, the luminance information of the sample image can be fully extracted from its color information. Consequently, in subsequent processing, the luminance information of the sample image can be effectively processed, reducing its impact on the chrominance information of the sample image.

[0039] The embodiments of this disclosure allow for efficient adjustment of the lighting conditions of sample images. Furthermore, during the adjustment process, the skin tone of the sample objects in the sample image can be kept stable at least, significantly reducing the difference between the skin tone of the sample objects in the output image and the skin tone of the sample objects in the sample image.

[0040] As can be understood, the training method of the deep learning model disclosed above has been explained. The following will combine... Figure 2 The training method disclosed herein will be further explained.

[0041] Figure 2 This is a schematic diagram of a deep learning model according to an embodiment of the present disclosure.

[0042] In some embodiments, the deep learning model may also include a convolutional network. For example, such as... Figure 2 As shown, a deep learning model may include an encoding network 210, a convolutional network 220, and a decoding network 230. For example, the convolutional network may be a convolutional neural network (CNN).

[0043] In some embodiments, the sample image includes a sample object associated with first sample illumination information, and the label image includes a sample object associated with second sample illumination information. For example, the first sample illumination information may include a first sample illumination angle. The second sample illumination information may include a second sample illumination angle. The first sample illumination angle and the second sample illumination angle may be different. For another example, for a sample object, multiple images can be acquired according to multiple preset illumination angles. Two images are randomly selected from these images as the sample image and the label image, respectively. Through the embodiments of this disclosure, the illumination conditions of the sample object are different in the label image and the sample image. After training with the sample image and the label image, the illumination conditions of the object in the image output by the deep learning model may be different from the input image. Therefore, the trained deep learning model can efficiently adjust the illumination effect of the object in the input image.

[0044] In some embodiments, the sample image may be an image in the BGR image space. For example, the sample image 201 may be converted to the YUV image space. Next, the Y channel image of the sample image 201 may be obtained as the initial luminance channel sample image 2011, or the U channel and V channel images of the sample image 201 may be obtained as the initial chroma channel sample image 2012.

[0045] In some embodiments, in some implementations of the above-described operation S110, the initial luminance channel sample image 2011 can be input into the encoding network 110 to obtain a first sample encoding result. The first sample encoding result may include the first sample spherical harmonic coefficients 211 and the first sample luminance image features.

[0046] In some embodiments of operation S120 described above, inputting the first sample encoding result into the decoding network to obtain the output luminance channel image includes: inputting the first sample encoding result into a convolutional network to obtain a second sample encoding result; and inputting the second sample encoding result into the decoding network to obtain the output luminance channel image.

[0047] In this embodiment, the second sample encoding result may include second sample spherical harmonic coefficients. For example, the first encoding result can be input into the convolutional network 220 to obtain the second sample encoding result. The second sample encoding result may include second sample spherical harmonic coefficients 221 and second sample luminance image features. Next, the second sample encoding result can be input into the decoding network 230 to obtain the output luminance channel image 231. Through this embodiment, by using a convolutional network to perform convolution processing on the first sample encoding result, the luminance information of the sample image can be further extracted, making the second sample encoding result more correlated with the luminance information of the sample image, so as to more flexibly adjust the illumination of the image without affecting the chromaticity of the image.

[0048] In some embodiments of operation S130 described above, fusing the output luminance channel image and the initial chroma channel sample image of the sample image to obtain the output image may include: fusing the output luminance channel image and the initial chroma channel sample image to obtain a sample fused image; converting the sample fused image to the blue-green-red image space to obtain the output image. For example, the output luminance channel image 231 and the initial chroma channel sample image 2012 can be fused to obtain the sample fused image. It is understood that the sample fused image can be an image in the YUV image space. The sample fused image can be converted to the BGR image space to obtain the output image 244.

[0049] In some embodiments of operation S140 described above, training a deep learning model based on at least one of the output image, the output luminance channel image, and the first sample spherical harmonic coefficients includes: training a deep learning model based on at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficients, and the second sample spherical harmonic coefficients.

[0050] In this embodiment, a first loss can be obtained based on the first sample spherical harmonic coefficients. For example, a sample image is processed using a preset function to obtain a first spherical harmonic coefficient label. The first loss is obtained based on the first spherical harmonic coefficient label and the first sample spherical harmonic coefficients. For another example, the preset function can be a spherical harmonic function. Processing the sample image 201 using the spherical harmonic function yields a 9-dimensional spherical harmonic coefficient, which serves as the first spherical harmonic coefficient label 241. Next, based on various loss functions, a first loss 251 can be obtained based on the first spherical harmonic coefficient label 241 and the first sample spherical harmonic coefficient 211. Through this embodiment, the spherical harmonic function processing result of the sample image is used as a label to train a deep learning model. Therefore, the encoding network of the trained deep learning model can accurately output spherical harmonic coefficients that are highly correlated with image brightness information.

[0051] In this embodiment, a second loss can be obtained based on the second sample spherical harmonic coefficients. For example, the label image of the sample image is processed using a preset function to obtain a second spherical harmonic coefficient label. The second loss is obtained based on the second spherical harmonic coefficient label and the second sample spherical harmonic coefficients. For another example, the label image 202 can be processed using a spherical harmonic function to obtain another 9-dimensional spherical harmonic coefficient, which serves as the second spherical harmonic coefficient label 242. Next, based on various loss functions, a second loss 252 can be obtained based on the second spherical harmonic coefficient label 242 and the second sample spherical harmonic coefficient 221. Through this embodiment, the result of the spherical harmonic function processing of the label image is used as a label to train a deep learning model. Thus, the trained deep learning model can transfer some or all of the illumination information from the label image to the sample image.

[0052] In this embodiment, a third loss can be obtained based on the output luminance channel image. For example, a luminance channel label image can be obtained based on the label image of the sample image. The third loss is obtained based on the luminance channel label image and the output luminance channel image. For another example, the label image 202 can also be an image in the BGR image space. The label image 202 can be converted to the YUV image space. Next, the Y channel image of the label image 202 can be obtained as the luminance channel label image 243. Next, a third loss 253 can be obtained based on various loss functions, using the luminance channel label image 243 and the output luminance channel image 231. Through this embodiment, the luminance channel image result of the label image is used as a label to train a deep learning model. Therefore, the trained deep learning model can more accurately transfer the illumination information of the label image to the sample image.

[0053] In this embodiment of the disclosure, a fourth loss can be obtained based on the output image. For example, the fourth loss can be obtained based on the output image and the label image of the sample image. Alternatively, a fourth loss 254 can be obtained based on various loss functions, using the output image 244 and the label image 202. Through this embodiment of the disclosure, the label image is used as a label to train the deep learning model. The sample objects in the label image and the sample image can be the same, but their illumination information differs. Therefore, the trained deep learning model can more accurately transfer the illumination information of the label image to the sample image.

[0054] In this embodiment of the disclosure, the total loss can be obtained based on the first loss, the second loss, the third loss, and the fourth loss. For example, based on the first loss 251, the second loss 252, the third loss 253, and the fourth loss 254, various calculations (such as summation) can be performed to obtain the total loss 255.

[0055] In this embodiment, the parameters of the deep learning model can be adjusted to bring the total loss to converge. For example, the parameters of the encoding network 210, convolutional network 220, and decoding network 230 in the deep learning model 200 can be adjusted. Then, I rounds of training are performed, yielding I total losses. I is an integer greater than 1. If the total loss in the i-th round is less than or equal to the total loss in the (i-1)-th round and the total loss in the i-th round is less than or equal to the total loss in the (i+1)-th round, it can be determined that the total loss has converged after i rounds of training. i can be an integer greater than 1 and less than 1. Next, the sample images can be changed, and the above process can be repeated. It can be understood that after training the deep learning model using a preset number of sample images, the training can be terminated.

[0056] It is understood that the total loss was determined above based on the first to fourth losses. However, this disclosure is not limited thereto. In embodiments of this disclosure, a deep learning model can be trained based on at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficients, and the second sample spherical harmonic coefficients. For example, the total loss can be obtained based on at least one of the first, second, third, and fourth losses. The parameters of the deep learning model can be adjusted based on the total loss.

[0057] As can be understood, the training method of the deep learning model disclosed herein has been described above, and the image processing method of this disclosure will be described below.

[0058] Figure 3 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0059] like Figure 3 As shown, the method 300 may include operations S310 to S330.

[0060] In operation S310, the initial brightness channel image of the original image is encoded to obtain the first encoding result of the initial brightness channel image.

[0061] In embodiments of this disclosure, the original image may include a target object. For example, the target object may have a face, a head, etc. Alternatively, the target object may be a person, an animal, a robot, or various other objects.

[0062] In this embodiment, the original image can be an image in BGR image space or an image in YUV image space. For example, if the original image is in YUV image space, the Y channel image of the original image can be used as the initial luminance channel image. Alternatively, if the original image is in BGR image space, the original image can be converted to YUV image space, and then the Y channel image of the converted original image can be used as the luminance channel image.

[0063] In this embodiment of the disclosure, the first encoding result includes a first luminance image feature. It is understood that encoding can be performed using various methods. The first luminance image feature can be correlated with information in the initial luminance channel image.

[0064] In operation S320, the target brightness channel image is obtained based on the target spherical harmonic coefficients and the first encoding result.

[0065] In embodiments of this disclosure, the target spherical harmonic coefficients can be user-inputted. For example, the target spherical harmonic coefficients can be related to the user's desired image lighting conditions. As another example, the target spherical harmonic coefficients can be 9-dimensional.

[0066] In this embodiment of the disclosure, the target spherical harmonic coefficients and the first luminance image features in the first encoding result can be decoded to obtain a decoding result. The decoding result may include the target luminance channel image. It is understood that decoding can be performed using various methods.

[0067] In operation S330, the target luminance channel image is fused with the initial chroma channel image of the original image to obtain the target image.

[0068] In this embodiment of the disclosure, the initial chroma channel image of the original image may include the U channel image and the V channel image of the original image. The target image can be obtained by fusing the initial chroma channel image and the target luminance channel image.

[0069] In this embodiment, the spherical harmonic coefficients of the luminance channel image of the original image are obtained using an encoding network. These spherical harmonic coefficients are primarily related to the luminance channel image of the original image, with very low correlation to the chrominance channel images (e.g., U-channel and V-channel images). Therefore, the luminance information of the sample image can be fully extracted from the color information of the original image. Consequently, in subsequent processing, the luminance information of the original image can be effectively processed, reducing the impact on the chrominance information of the original image.

[0070] The embodiments disclosed herein can efficiently adjust the lighting effects of the original image. Furthermore, during the adjustment process, the skin tone of the target object in the original image can be kept stable at least, significantly reducing the difference between the skin tone of the target object in the original image and the skin tone of the target object in the target image.

[0071] Furthermore, through the embodiments of this disclosure, the illumination of the original image can be altered using a target spherical harmonic coefficient provided by the user. This lowers the barrier to entry for adjusting image illumination, improves user experience, and enhances the competitiveness of related products.

[0072] It is understood that various methods can be used for encoding and decoding to obtain the encoding and decoding results. In this embodiment of the disclosure, a deep learning model can be used for encoding and decoding, which will be described in detail below.

[0073] Figure 4 This is a schematic diagram of a deep learning model according to an embodiment of the present disclosure.

[0074] In some embodiments, a deep learning model may include an encoder network, a convolutional network, and a decoder network. For example, such as Figure 4 As shown, a deep learning model may include an encoder network 410, a convolutional network 420, and a decoder network 430. For example, the encoder network 410 may be an encoder for a style-based generative adversarial network (GAN) model. The decoder network may be a decoder for a style-based GAN model.

[0075] In some embodiments, the original image may be an image in the BGR image space. For example, the original image 401 may be converted to the YUV image space. Next, the Y channel image of the original image 401 may be obtained as the initial luminance channel image 4011, or the U channel and V channel images of the original image 401 may be obtained as the initial chrominance channel image 4012.

[0076] In some embodiments of operation S310 described above, inputting the initial luminance channel image into the encoding network can yield a first encoding result. For example, the initial luminance channel image 4011 can be input into the encoding network 410 to obtain the first encoding result. The first encoding result may include a first spherical harmonic coefficient 411 and a first luminance image feature 412.

[0077] In some embodiments of operation S320 described above, obtaining the target luminance channel image based on the target spherical harmonic coefficients and the first encoding result may include: convolving the first encoding result to obtain a second encoding result of the initial luminance channel image. The target luminance channel image is then obtained based on the target spherical harmonic coefficients and the second encoding result.

[0078] In this embodiment, inputting the first encoding result into a convolutional network yields a second encoding result. For example, the first encoding result can be input into a convolutional network 420 to obtain the second encoding result. The second encoding result may include a second brightness image feature 422 and a second spherical harmonic coefficient 421. Through this embodiment, by using a convolutional network to perform convolution processing on the first encoding result, the brightness information of the original image can be further extracted, making the second spherical harmonic coefficient more correlated with the brightness information of the original image, thus facilitating subsequent processing.

[0079] In this embodiment of the disclosure, obtaining the target luminance channel image based on the target spherical harmonic coefficients and the second encoding result may include: replacing the second spherical harmonic coefficients in the second encoding result with the target spherical harmonic coefficients to obtain a third encoding result. For example, the second spherical harmonic coefficients 421 in the second encoding result may be replaced with the target spherical harmonic coefficients 403 to obtain the third encoding result. The third encoding result may include the second luminance image feature 422 and the target spherical harmonic coefficients 403.

[0080] In this embodiment of the disclosure, obtaining the target luminance channel image based on the target spherical harmonic coefficients and the second encoding result may further include: decoding the third encoding result to obtain the target luminance channel image. For example, inputting the third encoding result into the decoding network 430 can obtain the target luminance channel image 431. The luminance information of the original image has a high correlation with the second spherical harmonic coefficients. Through this embodiment of the disclosure, by replacing the second spherical harmonic coefficients with the target spherical harmonic coefficients and using the decoding network to decode the encoding result containing the target spherical harmonic coefficients and the features of the second luminance image, the correlation between the target luminance channel image and the target spherical harmonic coefficients can be improved, and the target spherical harmonic coefficients can be effectively used to adjust the illumination of the target object in the original image.

[0081] In some embodiments of operation S330 described above, fusing the target luminance channel image with the initial chroma channel image of the original image to obtain the target image may include: fusing the target luminance channel image and the initial chroma channel image to obtain a fused image. The fused image is then converted to a blue-green-red image space to obtain the target image. For example, the target luminance channel image 431 and the initial chroma channel image 4012 can be fused to obtain the fused image. It is understood that the fused image can be an image in the YUV image space. The fused image can be converted to the BGR image space to obtain the target image 444.

[0082] In some embodiments, the original image includes a target object associated with initial illumination information, and the target image includes a target object associated with target illumination information. For example, the initial illumination information may include an initial illumination angle. The target illumination information may include a target illumination angle. The initial illumination angle and the target illumination angle may be different.

[0083] In some embodiments, at least one of the above-described encoding network, convolutional network, and decoding network may be trained using method 100. For example, the above-described encoding network 410, convolutional network 420, and decoding network 430 may be trained encoding network 210, convolutional network 220, and decoding network 230.

[0084] It is understood that the target spherical harmonic coefficients were used to replace the second spherical harmonic coefficients in the second encoding result. However, this disclosure is not limited to this. In the embodiments of this disclosure, the target spherical harmonic coefficients can also be used to replace the first spherical harmonic coefficients in the first encoding result, which will be described in detail below.

[0085] In other embodiments, the first encoding result further includes first spherical harmonic coefficients. Obtaining the target luminance channel image based on the target spherical harmonic coefficients and the first encoding result may include: replacing the first spherical harmonic coefficients in the first encoding result with the target spherical harmonic coefficients to obtain a fourth encoding result; convolving the fourth encoding result to obtain a convolved fourth encoding result; and decoding the convolved fourth encoding result to obtain the target luminance channel image. For example, the fourth encoding result may include the target spherical harmonic coefficients and first luminance image features. Inputting the fourth encoding result into a convolutional network yields a convolved fourth encoding result. Inputting the convolved fourth encoding result into a decoding network yields a decoding result. This decoding result may include the target luminance channel image. Fusing the target luminance channel image with the initial chroma channel image and then performing image space transformation yields the target image. Through the embodiments of this disclosure, replacing the first spherical harmonic coefficients with target spherical harmonic coefficients and further convolving and decoding the encoding result containing the first spherical harmonic coefficients and first luminance image features can make the lighting effect of the target object in the target image more harmonious, thus helping to improve image quality.

[0086] Figure 5 This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0087] like Figure 5 As shown, the device 500 may include an encoding module 510, a first acquisition module 520, and a first fusion module 530.

[0088] Encoding module 510 is used to encode the initial brightness channel image of the original image to obtain a first encoding result of the initial brightness channel image. For example, the first encoding result includes first brightness image features;

[0089] The first acquisition module 520 is used to obtain the target brightness channel image based on the target spherical harmonic coefficients and the first encoding result.

[0090] The first fusion module 530 is used to fuse the target luminance channel image with the initial chrominance channel image of the original image to obtain the target image.

[0091] In some embodiments, the first obtaining module includes: a first convolution submodule, configured to convolve the first encoding result to obtain a second encoding result of the initial luminance channel image; and a first obtaining submodule configured to obtain a target luminance channel image based on the target spherical harmonic coefficients and the second encoding result.

[0092] In some embodiments, the first fusion module includes: a first fusion submodule, configured to fuse the target luminance channel image and the initial chroma channel image to obtain a fused image; and a first conversion submodule, configured to convert the fused image to a blue-green-red image space to obtain a target image.

[0093] In some embodiments, the second encoding result includes second spherical harmonic coefficients and second luminance image features. The first obtaining submodule includes: a replacement unit, configured to replace the second spherical harmonic coefficients in the second encoding result with target spherical harmonic coefficients to obtain a third encoding result; and a decoding unit, configured to decode the third encoding result to obtain a target luminance channel image.

[0094] In some embodiments, the first encoding result further includes first spherical harmonic coefficients, and the first obtaining module includes: a replacement submodule, used to replace the first spherical harmonic coefficients in the first encoding result with target spherical harmonic coefficients to obtain a fourth encoding result; a second convolution submodule, used to convolve the fourth encoding result to obtain a convolved fourth encoding result; and a decoding submodule, used to decode the convolved fourth encoding result to obtain a target luminance channel image.

[0095] In some embodiments, the encoding module includes a second obtaining submodule, configured to input the initial luminance channel image into the encoding network to obtain a first encoding result.

[0096] In some embodiments, the first convolutional submodule includes: a first obtaining unit, configured to input the first encoding result into the convolutional network to obtain a second encoding result.

[0097] In some embodiments, the decoding submodule includes: a second obtaining unit, configured to input the fourth encoding result into the decoding network to obtain a target luminance channel image.

[0098] In some embodiments, the original image includes a target object associated with initial illumination information, and the target image includes a target object associated with target illumination information.

[0099] Figure 6 This is a block diagram of a training apparatus for a deep learning model according to another embodiment of the present disclosure.

[0100] like Figure 6 As shown, the device 600 may include a second acquisition module 610, a third acquisition module 620, a second fusion module 630, and a training module 640. The deep learning model includes an encoding network and a decoding network.

[0101] The second obtaining module 610 is used to input the initial brightness channel sample image of the sample image into the encoding network to obtain the first sample encoding result of the initial brightness channel sample image. For example, the first sample encoding result includes the first sample spherical harmonic coefficients;

[0102] The third acquisition module 620 is used to input the encoding result of the first sample into the decoding network to obtain the output brightness channel image.

[0103] The second fusion module 630 is used to fuse the output luminance channel image and the initial chroma channel sample image of the sample image to obtain the output image.

[0104] Training module 640 is used to train a deep learning model based on at least one of the output image, the output luminance channel image, and the first sample spherical harmonic coefficients.

[0105] In some embodiments, the deep learning model further includes a convolutional network. The third obtaining module includes a third obtaining submodule, used to input the first sample encoding result into the convolutional network to obtain a second sample encoding result. For example, the second sample encoding result includes second sample spherical harmonic coefficients. A fourth obtaining submodule is used to input the second sample encoding result into a decoding network to obtain an output luminance channel image.

[0106] In some embodiments, the second fusion module includes: a second fusion submodule, configured to fuse the output luminance channel image and the initial chroma channel sample image to obtain a sample fused image; and a second conversion submodule, configured to convert the sample fused image to a blue-green-red image space to obtain an output image.

[0107] In some embodiments, the training module includes a training submodule for training a deep learning model based on at least one of an output image, an output luminance channel image, first sample spherical harmonic coefficients, and second sample spherical harmonic coefficients.

[0108] In some embodiments, the training submodule includes at least one of the following units: a third obtaining unit for obtaining a first loss based on the first sample spherical harmonic coefficients; a fourth obtaining unit for obtaining a second loss based on the second sample spherical harmonic coefficients; a fifth obtaining unit for obtaining a third loss based on the output luminance channel image; and a sixth obtaining unit for obtaining a fourth loss based on the output image.

[0109] In some embodiments, the training submodule further includes: a seventh obtaining unit, configured to obtain a total loss based on at least one of a first loss, a second loss, a third loss, and a fourth loss; and an adjusting unit, configured to adjust the parameters of the deep learning model such that the total loss converges to train the deep learning model.

[0110] In some embodiments, the third obtaining unit includes: a first processing subunit, configured to process the sample image using a preset function to obtain a first spherical harmonic coefficient label; and a first obtaining subunit, configured to obtain a first loss based on the first spherical harmonic coefficient label and the first sample spherical harmonic coefficients.

[0111] In some embodiments, the fourth obtaining unit includes: a second processing subunit, configured to process the label image of the sample image using a preset function to obtain a second spherical harmonic coefficient label; and a second obtaining subunit, configured to obtain a second loss based on the second spherical harmonic coefficient label and the second sample spherical harmonic coefficient.

[0112] In some embodiments, the fifth obtaining unit includes: a third obtaining subunit, configured to obtain a luminance channel label image based on the label image of the sample image; and a fourth obtaining subunit, configured to obtain a third loss based on the luminance channel label image and the output luminance channel image.

[0113] In some embodiments, the sixth obtaining unit includes: a fifth obtaining subunit, used to obtain a fourth loss based on the label images of the output image and the sample image.

[0114] In some embodiments, the sample image includes sample objects associated with first sample illumination information, and the label image includes sample objects associated with second sample illumination information.

[0115] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0117] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0119] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as image processing methods and / or deep learning model training methods. For example, in some embodiments, the image processing methods and / or deep learning model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the image processing methods and / or deep learning model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform image processing methods and / or training methods for deep learning models.

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) monitor or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0127] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method comprising: encoding an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result comprises first luminance image features; obtaining a target luminance channel image according to target spherical harmonic coefficients and the first encoding result, comprising: convolving the first encoding result to obtain a second encoding result of the initial luminance channel image; replacing second spherical harmonic coefficients in the second encoding result with target spherical harmonic coefficients input by a user to adjust a lighting condition of a target object in the original image using the target spherical harmonic coefficients, the target spherical harmonic coefficients being related to an image lighting condition expected by the user, to obtain a third encoding result; and decoding the third encoding result to obtain the target luminance channel image; and fusing the target luminance channel image with an initial chrominance channel image of the original image to obtain a target image.

2. The method of claim 1, wherein, The fusing the target luminance channel image with the initial chrominance channel image to obtain a target image comprises: fusing the target luminance channel image and the initial chrominance channel image to obtain a fused image; converting the fused image to a blue-green-red image space to obtain the target image.

3. The method of claim 2, wherein, The second encoding result further comprises second luminance image features.

4. The method of claim 1, wherein, The first encoding result further comprises first spherical harmonic coefficients, The obtaining a target luminance channel image according to target spherical harmonic coefficients and the first encoding result further comprises: replacing the first spherical harmonic coefficients in the first encoding result with the target spherical harmonic coefficients to obtain a fourth encoding result; convolving the fourth encoding result to obtain a convolved fourth encoding result; and decoding the convolved fourth encoding result to obtain the target luminance channel image.

5. The method of claim 1, wherein, The encoding an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image comprises: inputting the initial luminance channel image into an encoding network to obtain the first encoding result.

6. The method of claim 1, wherein, The convolving the first encoding result to obtain a second encoding result of the initial luminance channel image comprises: inputting the first encoding result into a convolution network to obtain the second encoding result.

7. The method of claim 4, wherein, The decoding the fourth encoding result to obtain the target luminance channel image comprises: inputting the fourth encoding result into a decoding network to obtain the target luminance channel image.

8. The method of claim 1, wherein, The original image comprises a target object related to initial lighting information, and the target image comprises the target object related to target lighting information.

9. A training method of a deep learning model, the deep learning model comprising an encoding network, a convolution network and a decoding network, comprising: inputting an initial luminance channel sample image of a sample image into the encoding network to obtain a first sample encoding result of the initial luminance channel sample image, wherein the first sample encoding result comprises first sample spherical harmonic coefficients; inputting the first sample encoding result into the convolution network to obtain a second sample encoding result, wherein the second sample encoding result comprises second sample spherical harmonic coefficients; and inputting the second sample encoding result into the decoding network to obtain an output luminance channel image; fusing the output luminance channel image and an initial chrominance channel sample image of the sample image to obtain an output image; and training the deep learning model according to at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficient and the second sample spherical harmonic coefficient, comprising: processing a label image of the sample image by using a preset function to obtain a second spherical harmonic coefficient label; and obtaining a second loss according to the second spherical harmonic coefficient label and the second sample spherical harmonic coefficient, so as to train the deep learning model, so that the trained deep learning model is used to migrate part or all of illumination information of the label image to the sample image.

10. The method of claim 9, wherein, The fusing the output luminance channel image and the initial chrominance channel sample image of the sample image to obtain an output image comprises: fusing the output luminance channel image and the initial chrominance channel sample image to obtain a sample fusion image; and converting the sample fusion image to a blue-green-red image space to obtain the output image.

11. The method of claim 9, wherein, The training the deep learning model according to at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficient and the second sample spherical harmonic coefficient further comprises at least one of the following operations: obtaining a first loss according to the first sample spherical harmonic coefficient; obtaining a third loss according to the output luminance channel image; obtaining a fourth loss according to the output image.

12. The method of claim 11, wherein, The training the deep learning model according to at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficient and the second sample spherical harmonic coefficient further comprises: obtaining a total loss according to at least one of the first loss, the third loss and the fourth loss and the second loss; adjusting parameters of the deep learning model so that the total loss converges, so as to train the deep learning model.

13. The method of claim 11, wherein, The obtaining the first loss according to the first sample spherical harmonic coefficient comprises: processing the sample image by using a preset function to obtain a first spherical harmonic coefficient label; and obtaining the first loss according to the first spherical harmonic coefficient label and the first sample spherical harmonic coefficient.

14. The method of claim 11, wherein, The obtaining the third loss according to the output luminance channel image comprises: obtaining a luminance channel label image according to a label image of the sample image; and obtaining the third loss according to the luminance channel label image and the output luminance channel image.

15. The method of claim 11, wherein, The obtaining the fourth loss according to the output image comprises: obtaining the fourth loss according to the output image and a label image of the sample image.

16. The method of claim 14 or 15, wherein, The sample image comprises a sample object related to first sample illumination information, and the label image comprises the sample object related to second sample illumination information.

17. An image processing apparatus, comprising: an encoding module configured to encode an initial luminance channel image of an original image to obtain a first encoding result of the initial luminance channel image, wherein the first encoding result comprises first luminance image features; The first obtaining module is configured to obtain a target luminance channel image according to the target spherical harmonic coefficient and the first encoding result; and The first fusion module is configured to fuse the target luminance channel image and an initial chrominance channel image of the original image to obtain a target image. The first obtaining module comprises: The first convolution submodule is configured to convolve the first encoding result to obtain a second encoding result of an initial luminance channel image. The replacement unit is configured to replace a second spherical harmonic coefficient in the second encoding result with a target spherical harmonic coefficient input by a user to adjust an illumination condition of a target object in the original image by using the target spherical harmonic coefficient, to obtain a third encoding result, the target spherical harmonic coefficient being related to an image illumination condition expected by the user. The decoding unit is configured to decode the third encoding result to obtain the target luminance channel image.

18. A training apparatus of a deep learning model, the deep learning model comprising an encoding network, a convolution network and a decoding network, comprising: The second obtaining module is configured to input an initial luminance channel sample image of a sample image into the encoding network to obtain a first sample encoding result of the initial luminance channel sample image, wherein the first sample encoding result comprises a first sample spherical harmonic coefficient. The third obtaining submodule is configured to input the first sample encoding result into the convolution network to obtain a second sample encoding result, the second sample encoding result comprising a second sample spherical harmonic coefficient. The fourth obtaining submodule is configured to input the second sample encoding result into the decoding network to obtain an output luminance channel image. The second fusion module is configured to fuse the output luminance channel image and an initial chrominance channel sample image of the sample image to obtain an output image. The training submodule is configured to train the deep learning model according to at least one of the output image, the output luminance channel image, the first sample spherical harmonic coefficient and the second sample spherical harmonic coefficient. The training submodule comprises: The second processing subunit is configured to process a label image of the sample image by using a preset function to obtain a second spherical harmonic coefficient label. The second obtaining subunit is configured to obtain a second loss according to the second spherical harmonic coefficient label and the second sample spherical harmonic coefficient, to train the deep learning model, so that the trained deep learning model is used to migrate part or all illumination information of the label image to the sample image.

19. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 16.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1 to 16.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 16.

Images