Image generation method and device and computer readable storage medium

Abstract

The invention provides a method for generating an image. The method comprises the following steps: acquiring a first image; based on the first image, obtaining texture prediction information of the target image corresponding to the first image through a texture prediction model, and obtaining color prediction information of the target image through a color prediction model; the target image is generated based on the first image according to the texture prediction information and the color prediction information, and the format of the target image is different from that of the first image. A target image which can better simulate clear-fuzzy distribution and predict color distribution closer to single-lens reflex equipment can be obtained. At least one of the plurality of modules may be implemented by an AI model. Functions associated with the AI may be performed by a non-volatile memory, a volatile memory, and a processor.

Description

Technical Field

[0001] The present invention relates to the field of image signal processing, and in particular to a technology for generating images of SLR quality. Background Art

[0002] The stringent size requirements of portable mobile devices usually lead to the use of compact sensors. Compared with mainstream SLR devices, the images captured by mobile devices have lower image quality. The image signal processing task from mobile devices to SLRs replaces the traditional image signal processing strategy by designing a learnable model, thereby narrowing the image quality gap caused by hardware. The mapping process aims to improve the image quality while keeping the content of the image unchanged. The image signal processing task from mobile devices to SLRs can be defined as a mapping problem from Raw images to sRGB images. Considering the various image processing operations it contains, it can also be regarded as an image restoration task or an image enhancement task.

[0003] At present, the image signal processing task from mobile devices to SLRs still faces many challenges. Summary of the invention

[0004] According to an embodiment of the present invention, a method for generating an image is provided, comprising: acquiring a first image; based on the first image, obtaining texture prediction information of a target image corresponding to the first image through a texture prediction model, and obtaining color prediction information of the target image through a color prediction model; generating the target image based on the first image according to the texture prediction information and the color prediction information, wherein a format of the target image is different from a format of the first image.

[0005] In one example, based on the first image, texture prediction information of a target image corresponding to the first image is obtained through a texture prediction model, including: encoding the first image through a first encoder of the texture prediction model to obtain image coding features; decoding the image coding features through a first decoder of the texture prediction model to obtain a first predicted image; and performing a texture extraction operation on the first predicted image through a texture extraction module to obtain the texture prediction information.

[0006] In one example, the texture prediction model is trained in the following manner: inputting training data into the first encoder to encode the training data to obtain image coding features, and the training data has the same format as the first image; inputting the image coding features into the first decoder and the second decoder respectively, obtaining texture prediction information through the first decoder, and obtaining depth prediction information through the second decoder; training the first encoder and the first decoder through the obtained texture prediction information and the reference image corresponding to the training data, and training the first encoder and the second decoder based on the obtained depth prediction information and the depth map obtained by the depth model, wherein the training data and the reference image are images from different sensors.

[0007] In one example, the depth map is obtained in the following manner: relative depth estimation is performed in a complete scene corresponding to the training data using the depth model to obtain a depth map corresponding to the training data.

[0008] In one example, based on the first image, color prediction information of the target image is obtained through a color prediction model, including: based on the first image, extracting features of the first image through a second encoder of the color prediction model; matching the features of the first image with a discrete code table of a reference image including prior information through the color prediction model; reconstructing the matched features through a third decoder of the color prediction model to generate a second predicted image; and performing color space transformation on the second predicted image, and determining the color component in the result obtained by the color space transformation as the color prediction information.

[0009] In one example, the color prediction model is trained in the following manner: based on a reference image, a third decoder and a discrete code table are trained; and based on training data, the second encoder is trained through the trained discrete code table and the third decoder, wherein the reference image and the training data are images from different sensors.

[0010] In one example, based on a reference image, a third decoder and a discrete code table are trained, including: inputting the reference image into a third encoder to extract features of the reference image; matching the features of the reference image with the previous discrete code table; reconstructing the matched features using the third decoder, and training the discrete code table and the third decoder based on the reference image and the reconstructed image.

[0011] In one example, the second encoder is trained based on training data through a trained discrete code table and a third decoder, including: inputting the training data into the second encoder to extract features of the training data; matching the features of the training data with the trained discrete code table; reconstructing the matched features using the trained third decoder, and training the third decoder based on a reference image corresponding to the training data and the reconstructed image.

[0012] In one example, it also includes: obtaining semantic information of the reference image; semantically matching features of the reference image that match the previous discrete code table with the semantic information, wherein reconstructing the matched features using a third decoder includes reconstructing the features after semantic matching using a third decoder.

[0013] In one example, the target image is generated based on the first image according to the texture prediction information and the color prediction information, including: fusing the texture prediction information and the color prediction information to obtain a first fused image; obtaining a first exposure parameter through an exposure estimation model based on the first fused image; and performing exposure adjustment on the first fused image based on the first exposure parameter to obtain the target image.

[0014] In one example, before performing texture prediction and color prediction, exposure normalization processing is performed on the first image for each color channel to obtain an exposure-normalized first image.

[0015] In one example, it also includes: estimating a second exposure parameter through an exposure estimation model based on the first image; and generating the target image based on the first image based on the texture prediction information and the color prediction information, including: generating an exposure-normalized third image based on the exposure-normalized first image based on the texture prediction information and the color prediction information; and performing exposure adjustment on the exposure-normalized third image using the second exposure parameter to obtain the target image.

[0016] According to an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the aforementioned methods.

[0017] According to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements any of the aforementioned audio signal processing methods when executed by a processor.

[0018] Therefore, according to an embodiment of the present invention, the depth of field difference between mobile devices (such as mobile phones, etc.) and SLR devices is taken into account, and the color distribution of the SLR device is taken into account. By predicting the texture information and color information of the target image, the clear and blurred distribution of the target image can be better simulated, and the color distribution that is closer to the target device can be predicted. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be more readily understood through the following detailed description with the aid of the accompanying drawings, in which like reference numerals designate like structural elements, and in which:

[0020] Figure 1 Shows the sRGB image of the mobile device and the sRGB image of the SLR device;

[0021] Figure 2 Shows the offset between the sRGB image of the mobile device and the sRGB image of the SLR device;

[0022] Figure 3 A schematic flow chart showing a method for generating an image according to an embodiment of the present invention;

[0023] Figure 4 A conceptual diagram showing an overall process according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a specific structure of a framework for processing an image signal from a mobile terminal device to generate a SLR quality image based on depth and SLR image prior guidance according to an embodiment of the present invention is shown;

[0025] Figure 6 A schematic diagram showing different depths of field for a mobile device and a SLR device;

[0026] Figure 7 is a schematic diagram of a texture prediction process according to an embodiment of the present invention;

[0027] Figure 8 Showing a result example and a comparative example of texture prediction according to an embodiment of the present invention;

[0028] Fig. 9 A schematic diagram showing a color prediction process according to an embodiment of the present invention;

[0029] Fig.10 A schematic diagram showing a curve-based exposure adjustment process for an RGB image;

[0030] Fig.11 The adjustment results corresponding to different exposure prediction parameters are shown;

[0031] Fig.12 A schematic diagram showing a curve-based exposure adjustment process for an input Raw domain image;

[0032] Fig.13 The color prediction result according to the embodiment of the present invention is shown;

[0033] Fig.14 A schematic diagram showing an electronic device according to an embodiment of the present invention;

[0034] Fig.15 A schematic diagram showing an electronic system according to an embodiment of the present invention; and

[0035] Fig.16 A schematic diagram showing another electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In the process of generating sRGB images, it is necessary to consider the different depths of field brought by different hardware devices or the exposure differences brought by different imaging scenes. Compared with mainstream SLR devices, mobile camera devices have smaller blur circles and correspondingly have a larger depth of field. Therefore, in most images, the images captured by mobile devices are clear. This obviously does not conform to the imaging law of SLR devices, resulting in a large difference in texture distribution between the generated SLR quality images and the real SLR images. In addition, different hardware devices and imaging environments have different exposure results for the same scene. In addition, the imaging color of SLR devices is also affected by unknown factors such as the device's own properties and the environment. Directly predicting the color may lead to inaccurate mapping results.

[0038] Therefore, there is a need to better simulate the sharp-blur distribution while predicting the color distribution that is closer to that of SLR devices.

[0039] Generally speaking, the signal processing of an image includes a series of operations, from image de-mosaicing and image denoising in low-level vision to color correction in high-level vision. A common strategy is to achieve the conversion of Raw images to sRGB images by completing each operation independently. It is already very challenging to complete image signal processing through an end-to-end model, and the image signal processing process of the Raw image collected by the mobile terminal to the camera-quality sRGB image involved in the present invention also needs to solve the imaging quality gap caused by hardware limitations.

[0040] Figure 1The sRGB image of the mobile device and the sRGB image of the SLR device are shown.

[0041] like Figure 1 As shown in the figure, the imaging results of sRGB images collected by SLR devices are affected by the image signal processing algorithm of the device itself, and are also affected by the environment when shooting. These factors are unknown in the mapping process. On the one hand, there is an unknown color mapping relationship between images collected by different devices. In addition, mobile devices and mainstream SLR devices are affected by their hardware properties and have different depths of field (DOF).

[0042] Figure 2 Shows the offset between the sRGB image of the mobile device and the sRGB image of the SLR device.

[0043] In the actual design process of the model, we also face the problem of spatial misalignment in the existing data. The existing relevant data is usually obtained by using different devices to collect the same scene in pairs of image data, but this process cannot guarantee that the collection results are spatially aligned. Using inaccurately aligned data for supervised training will cause problems such as pixel offset and blur in the results. Misalignment problems in data such as Figure 2 As shown, there is an obvious offset problem in the enlarged area of ​​the rectangular box.

[0044] Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

[0045] Although the description here is made by taking mobile Raw images and SLR images as examples, according to an embodiment of the present invention, various images with different image formats and / or image domains from other sensors may also be used, and are not limited to mobile Raw images and SLR images.

[0046] In the description of the embodiments of the present invention, the terms "first", "second", "third", etc. are merely used to distinguish corresponding features, parts, components, elements, etc., but not to limit the importance and quantity of the corresponding features, parts, components, elements, etc.

[0047] Figure 3 A schematic flow chart of a method 300 for generating an image according to an embodiment of the present invention is shown.

[0048] like Figure 3 As shown, in method 300 , at 310 , a first image is acquired.

[0049] At 320 , based on the first image, texture prediction information of the target image corresponding to the first image is obtained through a texture prediction model, and color prediction information of the target image is obtained through a color prediction model.

[0050] At 330 , the target image is generated based on the first image according to the texture prediction information and the color prediction information.

[0051] The format of the target image is different from the format of the first image.

[0052] According to one embodiment, the first image may be a Raw domain image of a mobile terminal, and the target image may be a SLR image, for example, in RGB format, but the present invention is not limited thereto. That is, the first image and the target image may be images with different formats, different spatial domains, and / or different color gamuts, etc.

[0053] According to the embodiment of the present invention, by predicting the texture information and color information of the target image, the clear and blurry distribution of the target image can be better simulated, and the color distribution that is closer to the target device can be predicted.

[0054] According to the embodiment of the present invention, texture prediction and color prediction may be executed in parallel or in sequence, and the execution order of texture prediction and color prediction may be determined as required.

[0055] In the embodiments of the present invention, the terms "depth of field" and "depth" may sometimes be used interchangeably.

[0056] Figure 4 A conceptual view showing an overall process according to an embodiment of the present invention.

[0057] like Figure 4 As shown, the process processes the image signal from the mobile device based on depth and SLR image prior guidance to generate SLR quality images. Figure 4 The process may include: (1) a color prediction module, which obtains a color estimation result of a target SLR image based on an input mobile Raw domain image; and (2) a texture prediction module, which obtains a texture detail distribution of the target SLR image based on an input mobile Raw domain image.

[0058] Figure 4 The texture prediction module and color prediction module in Figure 3 320 was executed.

[0059] In addition, if Figure 4 As shown, in order to ensure that the generated image is closer to the target SLR image and eliminate the exposure differences caused by different imaging scenes, an exposure prediction module is introduced to simulate the exposure of the target image. The process can also include: (3) an exposure prediction module, which estimates the exposure of the target SLR image based on the initial results of color prediction and texture prediction or the input mobile Raw domain image.

[0060] According to an embodiment of the present invention, various neural networks may be used to implement one or more of the color prediction module, the texture prediction module, and the exposure prediction module.

[0061] Figure 4 The process can be realized by deep neural network.

[0062] Figure 5 A specific structural schematic diagram of a framework for processing image signals from a mobile device to generate SLR quality images based on depth and SLR image prior guidance according to an embodiment of the present invention is shown. The training phase may include Raw image preprocessing, optical flow estimation and alignment, and SLR prior pre-training, while the inference phase may include a color prediction encoding and texture prediction module and an exposure prediction module.

[0063] During the training phase, the process may include: preprocessing the raw domain images on the mobile terminal and aligning the SLR images using optical flow; learning a discrete code table including the prior information of the SLR images from the SLR images for color prediction; using the aligned SLR images for training the color prediction model; and using the aligned SLR images and the reference depth map for training the texture prediction model.

[0064] In the inference stage, it can include: using the mobile Raw domain image as input to predict color; using the mobile Raw domain image as input to predict texture; combining the predicted color and texture to obtain a sRGB image of SLR quality; and using the mobile Raw domain image or the color prediction result and the texture prediction result as input to predict exposure to obtain an sRGB image with image quality closer to that of a SLR device.

[0065] Hereinafter, texture prediction, color prediction, and exposure prediction according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0066] The texture prediction according to the embodiment of the present invention is depth-based texture prediction. In the depth-based texture prediction module, the texture distribution of the target image is estimated based on the current mobile terminal Raw domain image, and a single image depth estimation method is used.

[0067] Affected by different hardware conditions, mobile devices and mainstream SLR devices usually have different depth of field. Mobile devices are limited by size constraints, such as smaller sensor size and compact lens, and usually have smaller blur circles, so they have a larger depth of field range. Correspondingly, mainstream SLR devices have larger blur circles and correspondingly smaller depth of field ranges, such as Figure 6 shown.

[0068] Figure 6 Schematic diagram showing the different depths of field of a mobile device and a SLR device.

[0069] The depth of field of an image is related to the depth of the scene. When the depth of the current area is within the depth of field, the device captures a clear image. Therefore, in most scenes, the imaging results of the mobile phone are clearer than those of a SLR camera. In the imaging results of the SLR device, affected by its depth of field range, the main target object is clear and the background is blurred, which makes it more in line with the visual habits of the human eye. Therefore, accurately imitating its blur effect during the image mapping process from the mobile terminal to the SLR device can further improve the image quality of the mapping result.

[0070] Figure 7 FIG. 1 is a schematic diagram of a texture prediction process according to an embodiment of the present invention. The process may be performed by a texture prediction model.

[0071] According to an embodiment of the present invention, a first image such as a mobile terminal Raw domain image is encoded by a first encoder (e.g., a shared encoder) of a texture prediction model to obtain image coding features. The image coding features are decoded by a first decoder (e.g., a texture decoder) of the texture prediction model to obtain a first predicted image, such as an sRGB predicted image. Then, a texture extraction operation is performed on the first predicted image by a texture extraction module to obtain texture prediction information.

[0072] The following reference Figure 7 Describe in detail.

[0073] Specifically, for the input Raw domain image (i.e., the first image), the encoder (e.g., the shared encoder) encodes it to obtain the corresponding image coding features. The depth decoder (i.e., the second decoder) takes the image coding features output by the shared encoder as input, restores the corresponding scene depth through the image coding features, and obtains the depth prediction result. In addition, the texture decoder (i.e., the first decoder) is based on the image coding features output by the shared encoder, takes the features obtained after passing through the intermediate layer as input, and decodes to obtain the corresponding first predicted image, such as the sRGB predicted image.

[0074] Finally, the color space of the sRGB prediction image is converted and the corresponding texture information is extracted to obtain the corresponding texture prediction information.

[0075] For example, a texture extraction module may be used to perform a texture extraction operation on the sRGB prediction image to obtain the texture prediction information.

[0076] In the embodiment of the present invention, the texture extraction module may be included in the texture prediction model, and may also be connected to the output of the texture prediction model, that is, based on the result output by the texture prediction model.

[0077] In the training stage of the texture prediction model, the training data is input into a first encoder to encode the training data to obtain image coding features, and the format of the training data is the same as that of the first image; the image coding features are respectively input into a first decoder (e.g., a texture decoder) and a second decoder (e.g., a depth decoder), texture prediction information is obtained through the first decoder, and depth prediction information is obtained through the second decoder; the first encoder and the first decoder are trained according to the obtained texture prediction information and a reference image corresponding to the training data, and the first encoder and the second decoder are trained according to the obtained depth prediction information and a depth map obtained by a depth model, wherein the training data and the reference image are images from different sensors.

[0078] Relative depth estimation is performed in a complete scene corresponding to the training data using the depth model to obtain a depth map corresponding to the training data.

[0079] For example, the training data may be Raw domain image data, and the reference image may be a SLR image.

[0080] In addition, during the training stage, a depth map corresponding to the input Raw domain image is obtained by performing image relative depth estimation in the complete scene corresponding to the Raw domain image, and the depth map is used as a pseudo label for depth prediction in the subsequent training process for training the first encoder and the third decoder.

[0081] Among them, relative depth estimation can obtain the depth relationship between objects in the image, for example, whether two objects are "close" or "far away".

[0082] It can be seen that in this process, the main task is to use the mobile Raw domain image as input to predict the sRGB image, and to learn depth-related features by using the depth estimation of a single image as an auxiliary task to imitate the texture clarity-blur distribution in the second image, such as the SLR image. The auxiliary task is jointly trained with the main task to constrain the texture prediction model to learn depth-related features in the encoding stage, thereby better predicting different depths of field and their corresponding clarity-blur distribution. The auxiliary task is only used in the training stage.

[0083] In the embodiment of the present invention, the texture decoder and the depth decoder share the image coding features output from the shared encoder, so that the texture prediction can learn the depth information, and the depth prediction result can further affect the texture prediction result.

[0084] Figure 8 Results examples and comparative examples of texture prediction according to an embodiment of the present invention are shown.

[0085] It can be seen from the corresponding depth map that the scene in the reference SLR image has a large depth, which is outside the depth of field range, so the final imaging result is relatively blurred. The mobile image has a large depth of field range, and the image signal processing result obtained without considering the depth of field is clearer than the reference image, as shown in the Lite camera image processing result (LiteISP) of the comparison example. The method proposed in the present invention effectively estimates the depth of field range of the scene by combining depth prediction, and the final image signal processing result is closer to the reference SLR image.

[0086] For the texture prediction model, the loss includes depth prediction loss and texture prediction loss.

[0087] The depth prediction loss is the L1 loss L between the predicted depth and the reference depth l1 .

[0088] The texture prediction loss is the L1 loss between the generated image and the SLR image, and the perceptual loss L perceptual , SSIM (Structural Similarity Index) loss L SSIM and generate adversarial loss L adv .

[0089] Therefore, the loss function of the texture prediction model is:

[0090] Loss texture =L l1 (Depth predict , Depth reference )+α 1 L l1 (t out ,t)+β 1 L perceptual (t out ,y)+γ 1 L SSIM (t out ,y)+ε 1 L adv (t out )

[0091] Among them, Depthpredict is the depth prediction result obtained by the depth decoder, Depth refenrece is the reference depth image, t out is the generated image of the texture prediction model, y is the SLR image, t represents the time, α 1 , β 1 , γ 1 and ε 1 They are L1 loss and perceptual loss L perceptual , SSIM loss L SSIMand generate adversarial loss L adv The coefficient of .

[0092] Optionally, α 1 , β 1 , γ 1 and ε 1 It can be obtained based on experience or through simulation, and its value is not limited in the embodiment of the present invention.

[0093] The color prediction according to the embodiment of the present invention is based on a priori color prediction, for example, it can be based on a second image, for example, a SLR image that is a reference image of the target image, and a priori color prediction. After a stage of pre-training in the color prediction model, the trained discrete code table and the corresponding decoder are called the prior information of the SLR image. It can be considered that such prior information covers the SLR image distribution information, including the color component information of the SLR image.

[0094] In the color prediction module based on SLR image prior, the color of the target image is estimated based on the current mobile Raw domain image.

[0095] There is an unknown color mapping relationship between the input mobile Raw domain image and the target SLR sRGB image, which is affected by many unknown factors, such as camera parameters and the shooting environment. Therefore, it is difficult to accurately predict the target color directly based on the input Raw domain image.

[0096] In order to estimate the color distribution of the target SLR image, the embodiment of the present invention constructs a discrete code table based on the SLR image through pre-training, and the discrete code table may include the prior information of the SLR image, which is used to store the feature matrix of the SLR image. At the same time, the color decoder is obtained by pre-training, and the corresponding features of the code table are decoded to obtain the sRGB image.

[0097] In addition, features are extracted from the input Raw domain image and matched with the discrete code table of the SLR image to obtain a color prediction result with the raw domain graphic content and the color distribution of the SLR image.

[0098] Fig. 9 A schematic diagram of a color prediction process according to an embodiment of the present invention is shown. The process shows a training phase. Fig. 9 Provide a detailed description.

[0099] like Fig. 9 As shown, the training process of the color prediction model according to the embodiment of the present invention mainly includes two stages.

[0100] In the first stage of training, a discrete code table and a corresponding third decoder are pre-trained based on a reference image, and the reference image may be a SLR image. In the pre-training stage, the reference image (SLR image) is used as input and matched to a discrete code table to obtain a discrete code table of the SLR image. For example, the SLR image may be encoded by a third encoder, such as a SLR image encoder, to obtain features of the SLR image, and then the features of the reference image may be matched with the previous discrete code table. The matching result is decoded by a third decoder (such as a color decoder) to reconstruct the SLR image. The training process of the discrete code table and the decoder is implemented based on the reference image and the image obtained after reconstruction. The weights of the decoder obtained can be used for a decoder in the color prediction stage, such as a third decoder.

[0101] In the first stage of training, the initial discrete code table may be obtained based on experience or may be randomly initialized. The initial discrete code table is not limited in the embodiment of the present invention.

[0102] In the second stage of training, the second encoder is trained based on the training data (such as images captured by the mobile device) through the discrete code table and the third decoder after the first stage of training. In one example, the training data is input into the second encoder to extract the features of the training data; the features of the training data are matched with the trained discrete code table; the matched features are reconstructed using the trained third decoder, and the third decoder is trained based on the reference image corresponding to the training data and the reconstructed image.

[0103] Among them, the reference image and training data are images from different sensors.

[0104] In addition, in order to further improve the correlation between color prediction results and image semantic information, semantic information is introduced in the first stage of training as a guide for the first stage of training. The semantic guidance process is achieved by narrowing the distance between the semantic information of the reference image extracted by the pre-trained model and the SLR image features obtained after feature matching.

[0105] like Fig. 9 As shown in , the semantic information of the reference image can be obtained through the pre-trained VGG (Visual Geometry Group) network, and semantically matched with the features of the reference image that match the previous discrete code table, and then the semantically matched features are input into the decoder for reconstruction.

[0106] Then, according to an embodiment of the present invention, in the inference stage of color prediction, based on the first image, features of the first image are extracted through a second encoder of a color prediction model, such as a color prediction encoder. Thereafter, the extracted features of the first image are matched with a discrete code table including prior information of a reference image through the color prediction model, and the matched features are reconstructed through a third decoder of the color prediction model to generate a second predicted image, such as a predicted sRGB image, and a color space transformation is performed on the second predicted image, and the color components in the result of the color space transformation are determined as color prediction information.

[0107] Specifically, the input Raw domain image is feature extracted by the encoder, for example, the Raw domain image is encoded by the color prediction encoder to obtain its features / information, and the extracted features / information are matched to the discrete code table of the SLR image. The matched features are reconstructed by a decoder (such as a color decoder) to generate a second predicted image, such as an sRGB predicted image. The decoder in the pre-training stage shares parameters (such as weights) with the decoder in the prediction stage. In the color prediction stage, the pre-trained discrete code table and the corresponding decoder weights are fixed.

[0108] Finally, the obtained sRGB predicted image is converted to a color space, and the color information corresponding to the color is extracted to retain its color component, thereby obtaining the final color prediction information. For example, if the image is converted to a YCrCb space, the Cr component and the Cb component are retained. However, the example of the color space is not limited thereto.

[0109] According to an embodiment of the present invention, the color space conversion may be performed in the color prediction model, or may be performed outside the color prediction model by outputting the color prediction model.

[0110] In one example, the color prediction module based on SLR image priors may include the following four steps:

[0111] (1) Pre-training discrete code tables and corresponding decoders based on SLR images;

[0112] (2) Feature extraction and feature matching of Raw domain images;

[0113] (3) The decoder reconstructs the matched features and generates the predicted sRGB results;

[0114] (4) Perform color space conversion on the predicted sRGB image and retain its color components.

[0115] For the color prediction model, in the pre-training stage, the loss function is:

[0116] Loss pretrain =L l1(y recon ,y)+α 2 L l2 (sg(ze),zc)+β 2 L l2 (sg(zc),ze)+γ 2 L l2 (zc, vgg(y)).

[0117] Among them, L l2 is the L2 loss, sg() is the gradient stop operation, y recon is the SLR image reconstructed by the decoder, ze is the encoder output feature, zc is the matched feature, and vgg(y) is the semantic feature α extracted from the input SLR image through the pre-trained VGG model 2 , β 2 and γ 2 are the coefficients of the L1 loss and the corresponding L2 loss, respectively, and can be obtained empirically.

[0118] In the prediction stage, the loss function is:

[0119] Loss color =L l2 (z mobile , z dslr )+α 3 L l1 (c out ,y)+β 3 L perceptual (c out ,y)+γ 3 L adv (c out ).

[0120] Among them, z mobile， z dslr is the feature after matching between the mobile image and the SLR image, c out Generate an image for the color prediction model, y is the SLR image, α 3 , β 3 and γ 3 They are L1 loss and perceptual loss L perceptual and generate adversarial loss L adv The coefficient of .

[0121] Optionally, α 3 , β 3 and γ 3 It can be obtained based on experience or through simulation, and its value is not limited in the embodiment of the present invention.

[0122] As mentioned above, the exposure result of the image captured by the device is affected by many aspects, such as the exposure method, the scene lighting conditions during shooting, etc. Therefore, even if different devices shoot the same scene, there may still be large exposure differences in their imaging results. Therefore, in order to further improve the image quality generated by the embodiment of the present invention and make it closer to the SLR image, the embodiment of the present invention also performs exposure prediction before obtaining the final target image.

[0123] The embodiment of the present invention proposes two different exposure strategies: exposure adjustment for RGB images; and exposure adjustment for input Raw domain images.

[0124] In the exposure adjustment for the RGB image, the exposure adjustment is performed on the initial ISP result (ie, the fusion result of the color prediction and the texture prediction without the exposure adjustment).

[0125] Specifically, in order to adjust the exposure, the embodiment of the present invention predicts the exposure adjustment curve, and the prediction process only depends on the input initial ISP result, that is, the fusion result of the color prediction and texture prediction that have not been adjusted by the exposure. In the exposure adjustment process, on the one hand, it is necessary to keep the pixel value of the image from overflowing and causing information loss, and on the other hand, it is necessary to keep the original content in the image unchanged.

[0126] Fig.10 A schematic diagram of a curve-based exposure adjustment process for an RGB image is shown. The process can be performed by an exposure adjustment module.

[0127] like Fig.10 As shown, by fusing the initial color prediction result and the texture prediction result, an initial fused image, namely the initial ISP result, is obtained as input. Then, the exposure prediction model estimates the first exposure parameter α according to the input image (such as the input initial ISP result). 4 , and according to the first exposure parameter α 4 The initial ISP result is subjected to exposure adjustment to obtain the target image. In this process, the exposure adjustment is defined as the following equation (1).

[0128] I o =I i +α 4 I i (1-I i ) α 4 ∈[-1,+1] (1)

[0129] Among them, I o is the output exposure adjusted image, α 4 is the first exposure coefficient predicted by the model, I i This is the initial fusion result of color prediction and texture prediction.

[0130] Fig.11 The adjustment results corresponding to different exposure prediction parameters are shown, wherein the exposure results in two extreme cases are shown.

[0131] In addition, in some application scenarios, there is an unclear exposure relationship between the image data collected by the mobile terminal and the image data collected by the camera device. Therefore, in order to predict a more accurate image, the embodiment of the present invention proposes a curve-based exposure estimation model, that is, exposure adjustment for the input Raw domain image.

[0132] Fig.12 A schematic diagram of a curve-based exposure adjustment process for an input Raw domain image is shown. The process can be performed by an exposure adjustment module.

[0133] and Fig.10 Compared with the exposure adjustment for the RGB image shown in , the exposure adjustment model for the input Raw domain image proposed in the embodiment of the present invention takes the raw domain image of the mobile terminal, that is, the first image, as input, and directly estimates the exposure of the raw domain image of the mobile terminal to perform exposure adjustment, thereby avoiding the influence of the initial ISP result.

[0134] like Fig.12 As shown, before performing texture prediction and color prediction, the mobile Raw domain image, i.e., the first image, is subjected to exposure normalization for each color channel to obtain an exposure-normalized first image (a Raw domain image with unchanged exposure) and used as input for color prediction and texture prediction. Exposure normalization is defined as the following equation (2).

[0135] I n =Norm(I n ) n∈(R,G,B) (2)

[0136] For example, exposure normalization normalizes the three color channels (R, G, B) of an image to reduce the difference between images with different exposures.

[0137] Furthermore, after texture prediction and color prediction, a first image after texture prediction and color prediction can be obtained, that is, an exposure-normalized third image, which is an RGB domain image.

[0138] On the other hand, the second exposure parameter can be estimated by directly targeting the first image (mobile Raw domain image) through the exposure estimation model. Here, the second exposure parameter α can be estimated from the first image 5 , similarly, α 5 is the exposure coefficient predicted by the model, α 5 ∈[-1,+1].

[0139] Then, the exposure-normalized third image is adjusted according to equation (1) above to generate an output image having an exposure close to that of the target image.

[0140] Fig.13 The color prediction result according to the embodiment of the present invention is shown.

[0141] like Fig.13 As shown, for the input mobile terminal Raw domain image, the embodiment of the present invention performs color prediction and texture prediction respectively, and finally combines them to obtain the output image. It can be seen that compared with the ISP algorithm result of the mobile terminal, as the final result of the present invention, it is compared with the ISP result built into the mobile phone. The color and texture are closer to the reference SLR image.

[0142] Therefore, according to an embodiment of the present invention, a method for processing an image signal of a mobile device based on depth and SLR image prior guidance to generate a SLR quality image is proposed, including a color prediction module based on SLR image prior and a texture prediction module based on depth. The embodiment of the present invention can be applied to tasks such as mobile camera image signal processing, and generates an sRGB image with SLR device quality and close to the color and texture distribution of the target device under the premise of only a single mobile Raw domain image as input.

[0143] Furthermore, by using single image depth estimation as an auxiliary task, the model can learn depth-related feature information during the encoding process, and then learn the depth of field of the target device in the current scene. At the same time, taking into account the exposure differences of the scene, the exposure of the target device is predicted. In addition, in terms of color prediction, the prior information of the target device is learned by constructing a SLR image prior code table, and then more accurate color prediction is achieved through feature matching. Based on the above solution, the embodiment of the present invention can better simulate the clear and blurred distribution of the target image in the process of mapping the mobile terminal image to the SLR device image, and at the same time predict the color distribution that is closer to the target device, and generate an image that is more in line with the visual laws of the human eye.

[0144] Fig.14 A schematic diagram of an electronic device 1400 according to an embodiment of the present invention is shown. The electronic device may be a terminal, a server or any other device.

[0145] like Fig.14 As shown in , the electronic device 1400 includes an image acquisition unit 1410 , a prediction information acquisition unit 1420 , and an image generation unit 1430 .

[0146] The image acquiring unit 1410 is configured to acquire a first image.

[0147] The prediction information acquisition unit 1420 is configured to obtain, based on the first image, texture prediction information of the target image corresponding to the first image through a texture prediction model, and obtain color prediction information of the target image through a color prediction model.

[0148] The image generating unit 1430 is configured to generate the target image based on the first image according to the texture prediction information and the color prediction information.

[0149] The format of the target image is different from the format of the first image.

[0150] In an example, the prediction information acquisition unit 1420 is configured to: encode the first image through the first encoder of the texture prediction model to obtain image coding features; decode the image coding features through the first decoder of the texture prediction model to obtain a first predicted image; and perform a texture extraction operation on the first predicted image through a texture extraction module to obtain the texture prediction information.

[0151] In one example, the texture prediction model is trained in the following manner: inputting training data into the first encoder to encode the training data to obtain image coding features, and the training data has the same format as the first image; inputting the image coding features into the first decoder and the second decoder respectively, obtaining texture prediction information through the first decoder, and obtaining depth prediction information through the second decoder; training the first encoder and the first decoder through the obtained texture prediction information and the reference image corresponding to the training data, and training the first encoder and the second decoder based on the obtained depth prediction information and the depth map obtained by the depth model, wherein the training data and the reference image are images from different sensors.

[0152] In one example, the depth map is obtained in the following manner: relative depth estimation is performed in a complete scene corresponding to the training data using the depth model to obtain a depth map corresponding to the training data.

[0153] In one example, the prediction information acquisition unit 1420 is configured to: based on the first image, extract features of the first image through a second encoder of a color prediction model; match the features of the first image with a discrete code table of a reference image including prior information through the color prediction model; reconstruct the matched features through a third decoder of the color prediction model to generate a second predicted image; and perform color space transformation on the second predicted image, and determine the color component in the result obtained by the color space transformation as the color prediction information.

[0154] In one example, the color prediction model is trained in the following manner: based on a reference image, a third decoder and a discrete code table are trained; and based on training data, the second encoder is trained through the trained discrete code table and the third decoder, wherein the reference image and the training data are images from different sensors.

[0155] In one example, in one example, the prediction information acquisition unit 1420 is also configured to: input the reference image into a third encoder to extract features of the reference image; match the features of the reference image with the previous discrete code table; use a third decoder to reconstruct the matched features, and train the discrete code table and the third decoder based on the reference image and the reconstructed image.

[0156] In one example, the prediction information acquisition unit 1420 is also configured to: input the training data into a second encoder to extract features of the training data; match the features of the training data with the trained discrete code table; reconstruct the matched features using the trained third decoder, and train the third decoder based on the reference image corresponding to the training data and the reconstructed image.

[0157] In one example, the prediction information acquisition unit 1420 is further configured to: obtain semantic information of the reference image; semantically match features of the reference image that match the previous discrete code table with the semantic information, wherein reconstructing the matched features using a third decoder includes reconstructing the features after semantic matching using a third decoder.

[0158] In one example, the electronic device 1400 also includes an exposure adjustment unit (not shown), which is configured to fuse the texture prediction information and the color prediction information to obtain a first fused image; obtain a first exposure parameter based on the first fused image through an exposure estimation model; and perform exposure adjustment on the first fused image based on the first exposure parameter to obtain the target image.

[0159] In one example, before performing texture prediction and color prediction, for each color channel, the exposure adjustment unit performs exposure normalization processing on the first image to obtain an exposure-normalized first image.

[0160] In one example, the exposure adjustment unit estimates a second exposure parameter based on the first image through an exposure estimation model; and generates the target image based on the first image according to the texture prediction information and the color prediction information, including: generating an exposure-normalized third image based on the exposure-normalized first image based on the texture prediction information and the color prediction information; and performing exposure adjustment on the exposure-normalized third image using the second exposure parameter to obtain the target image.

[0161] Fig.15 A schematic diagram of an electronic system 1500 according to an embodiment of the present invention is shown.

[0162] like Fig.15 As shown, the electronic system 1500 includes a memory 1510 and a processor 1520 .

[0163] The memory 1510 is configured to store instructions.

[0164] The processor 1520 is coupled to the memory 1510 and is configured to execute the instructions so that the electronic system 1500 performs any of the aforementioned methods.

[0165] Fig.16 A schematic diagram of another electronic device 1600 according to an embodiment of the present invention is shown. The electronic device may be a terminal, a server or any other device.

[0166] like Fig.16 As shown in , the electronic device 1600 includes a memory 1610, a processor 1620, and a computer program 1630 stored in the memory 1610, wherein the processor 1620 executes the computer program 1630 to implement any of the aforementioned methods.

[0167] According to an embodiment of the present invention, the present invention further includes a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the audio signal processing method described in any one of the preceding claims is implemented.

[0168] At least one of the multiple modules may be implemented by an AI model. Functions associated with the AI ​​may be performed by a non-volatile memory, a volatile memory, and a processor.

[0169] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors such as a central processing unit (CPU), an application processor (AP), etc., graphics processing units such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor such as a neural processing unit (NPU).

[0170] One or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0171] Here, providing by learning means making a predefined operating rule or AI model of desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system.

[0172] The AI ​​model can be composed of multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0173] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating an image, comprising: acquiring a first image; Based on the first image, obtaining texture prediction information of the target image corresponding to the first image through a texture prediction model, and obtaining color prediction information of the target image through a color prediction model; generating the target image based on the first image according to the texture prediction information and the color prediction information, The format of the target image is different from the format of the first image.

2. The method according to claim 1, wherein: Based on the first image, obtaining texture prediction information of a target image corresponding to the first image through a texture prediction model includes: Encoding the first image by using a first encoder of the texture prediction model to obtain image coding features; Decoding the image coding feature by a first decoder of the texture prediction model to obtain a first predicted image; and A texture extraction operation is performed on the first prediction image through a texture extraction module to obtain the texture prediction information.

3. The method according to claim 2, wherein: The texture prediction model is trained in the following way: Inputting training data into the first encoder to encode the training data to obtain image coding features, wherein the training data has the same format as the first image; Inputting the image coding features into a first decoder and a second decoder respectively, obtaining texture prediction information through the first decoder, and obtaining depth prediction information through the second decoder; The first encoder and the first decoder are trained using the obtained texture prediction information and a reference image corresponding to the training data, and the first encoder and the second decoder are trained based on the obtained depth prediction information and a depth map obtained through a depth model, wherein the training data and the reference image are images from different sensors.

4. The method according to claim 3, wherein: The depth map is obtained by: Relative depth estimation is performed in a complete scene corresponding to the training data using the depth model to obtain a depth map corresponding to the training data.

5. The method according to any one of claims 1 to 4, wherein: Based on the first image, obtaining color prediction information of the target image by using a color prediction model includes: Based on the first image, extracting features of the first image through a second encoder of a color prediction model; Matching the features of the first image with a discrete code table including prior information of a reference image through a color prediction model; Reconstructing the matched features through a third decoder of the color prediction model to generate a second predicted image; and A color space transformation is performed on the second predicted image, and a color component in a result obtained by the color space transformation is determined as the color prediction information.

6. The method according to claim 5, wherein: The color prediction model is trained in the following way: Based on the reference image, training the third decoder and the discrete code table; and Based on the training data, the second encoder is trained by using the trained discrete code table and the third decoder, The reference image and the training data are images from different sensors.

7. The method according to claim 6, wherein: Based on the reference image, the third decoder and the discrete code table are trained, including: Inputting the reference image into a third encoder to extract features of the reference image; Matching the features of the reference image with the previous discrete code table; The matched features are reconstructed using a third decoder, and the discrete code table and the third decoder are trained according to the reference image and the reconstructed image.

8. The method according to claim 6, wherein: Based on the training data, the second encoder is trained by using the trained discrete code table and the third decoder, including: Inputting the training data into a second encoder to extract features of the training data; Matching the features of the training data with the discrete code table obtained by training; The matched features are reconstructed using the trained third decoder, and the third decoder is trained according to the reference image corresponding to the training data and the reconstructed image.

9. The method according to claim 6, wherein: Also includes: Obtaining semantic information of the reference image; Perform semantic matching between the features of the reference image that match the previous discrete code table and the semantic information, The reconstructing the matched features using the third decoder includes reconstructing the semantically matched features using the third decoder.

10. The method according to any one of claims 1 to 9, wherein: Generating the target image based on the first image according to the texture prediction information and the color prediction information includes: Fusing the texture prediction information and the color prediction information to obtain a first fused image; Obtaining a first exposure parameter through an exposure estimation model according to the first fused image; and Based on the first exposure parameter, exposure adjustment is performed on the first fused image to obtain the target image.

11. The method according to any one of claims 1 to 9, wherein: Before performing texture prediction and color prediction, for each color channel, an exposure normalization process is performed on the first image to obtain an exposure-normalized first image.

12. The method according to claim 11, further comprising: estimating a second exposure parameter according to the first image by using an exposure estimation model; and Generating the target image based on the first image according to the texture prediction information and the color prediction information includes: generating an exposure-normalized third image from the exposure-normalized first image based on the texture prediction information and the color prediction information; as well as The second exposure parameter is used to perform exposure adjustment on the exposure-normalized third image to obtain the target image.

13. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method of any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio signal processing method according to any one of claims 1 to 12 is implemented.

Images