Training method for image signal processing, and corresponding storage medium

Abstract

Disclosed in embodiments of the present invention are a training method for image signal processing (ISP), a system, and a storage medium, applied to the technical field of image processing. In the embodiments of the present invention, in the process of training an ISP model, ISP for original captured sample images needs to be divided, wherein one portion, i.e., first ISP related to non-shared parameters of an image capture device, is performed using a conventional method, and the other portion, i.e., second ISP, is performed using an initial ISP model.

Description

Training method of image signal processing and corresponding storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a training method of image signal processing and a storage medium. BACKGROUND

[0002] The optical image of an object in nature is projected onto the surface of an image sensor with a color filter array through a camera lens, is first converted into an analog electrical signal through photoelectric conversion, is then converted into a digital image signal, i.e., a Bayer format image, through an analog to digital converter (A / D), and is then sent to a digital signal processing chip (DSP) for image signal processing (ISP) to form an RGB format image.

[0003] General ISP processing can include black level compensation, lens shading correction, bad pixel correction, color interpolation, Bayer noise removal, white balance correction, color correction, Gamma correction, color space conversion, color noise removal and edge enhancement in the YUV color space, color and contrast enhancement, etc. With the increasing complexity of image processing scenarios and the increasing special requirements for image quality, general ISP faces challenges such as a large parameter library, difficulty in debugging, and a long development cycle.

[0004] SUMMARY

[0005] The embodiments of the present application provide a training method of image signal processing and a storage medium, so that the quality of the image processed by the trained image signal processing model is improved.

[0006] In an aspect, the embodiments of the present application provide a training method of image signal processing, comprising:

[0007] determining an initial image signal processing model;

[0008] determining training samples, wherein the training samples include a plurality of groups of sample images, and each group of sample images includes an original sample image and a corresponding display sample image;

[0009] The first image signal processing is processing related to non-shared parameters of a photographing device, and the second image signal processing is other image signal processing than the first image signal processing.

[0010] The image signal processing initial model is adjusted according to a display format image obtained according to the image signal processing initial model and a display sample image in the training sample, so as to obtain an image signal processing model.

[0011] The embodiment of the present application also provides a computer readable storage medium which stores a plurality of computer programs, and the computer programs are suitable for being loaded and executed by a processor to implement the training method of the image signal processing.

[0012] It can be seen that, in the training process of the image signal processing model, the ISP processing on the original photographing sample image is divided into two parts in the embodiment of the present application, one part is the first image signal processing related to non-shared parameters of a photographing device, and the other part is the second image signal processing using the image signal processing initial model, so that the display format image obtained by using the image signal processing model obtained by training is compared with the display format image obtained by using the traditional ISP processing, and the quality of the display format image obtained by using the method of the embodiment can be greatly improved, especially in the peak signal-to-noise ratio, structural similarity index and chroma channel. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0014] Fig. 1 is a schematic diagram of a training method of image signal processing provided by an embodiment of the present application;

[0015] Fig. 2 is a flow chart of a training method of image signal processing provided by an embodiment of the present application;

[0016] Fig. 3 is a schematic diagram of an image signal processing initial model in an embodiment of the present application;

[0017] Fig. 4 is a schematic diagram of a VGG16 network in an embodiment of the present application;

[0018] FIG. 5 is a schematic diagram of calculating an adversarial loss function in an embodiment of the present application;

[0019] FIG. 6 is a flowchart of an image signal processing method in an embodiment of the present application;

[0020] FIG. 7 is a flowchart of training an image signal processing model in a specific application embodiment of the present application;

[0021] FIG. 8 is a flowchart of an image signal processing method provided in a specific application embodiment of the present application;

[0022] FIG. 9 is a schematic diagram of the logical structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0024] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0025] The embodiments of the present application provide a training method for image signal processing, which can mainly train an image signal processing model to be applied to a process of obtaining a display format image by processing an original shooting format image after an image is shot by a shooting device, as shown in FIG. 1. The training system for image signal processing can perform the following training for image signal processing:

[0026] Determine an image signal processing initial model; determine a training sample, the training sample including a plurality of groups of sample images, each group of sample images including an original photographed sample image and a corresponding display sample image; perform first image signal processing on each of the original photographed sample images to obtain a first-processed image, and then perform second image signal processing on the first-processed image by using the image signal processing initial model to obtain a corresponding display format image; the first image signal processing is processing related to non-shared parameters of a photographing device, and the second image signal processing is other image signal processing than the first image signal processing; and adjust the image signal processing initial model according to the display format image obtained by using the image signal processing initial model and the display sample image in the training sample, to obtain an image signal processing model.

[0027] The preset image signal processing model is a machine learning model based on artificial intelligence, which can be obtained by training in a certain method. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0028] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, machine learning and deep learning, etc.

[0029] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0030] When the image signal processing model trained in this way is actually applied to the field of image processing, the quality of the display format image obtained by using the method of the embodiment can be greatly improved compared with the quality of the display format image obtained by using the traditional ISP processing, and is excellent in peak signal-to-noise ratio, structural similarity index, and chroma channel.

[0031] An embodiment of the present application provides a training method of image signal processing, mainly a method performed by a training system of image signal processing, a flowchart of which is shown in FIG. 2, comprising:

[0032] Step 101, determining an initial model of image signal processing.

[0033] It can be understood that when the training system of image signal processing determines the initial model of image signal processing, the initial model of image signal processing is determined to include a multi-layer structure and initial values of parameters in each layer structure. The parameters of the initial model of image signal processing refer to fixed parameters used in the calculation process of each layer structure in the initial model of image signal processing, which do not need to be assigned at any time, such as parameter size, weight value, user vector length and the like.

[0034] Specifically, the initial model of image signal processing is specifically used for performing second image signal processing on the image processed for the first time (obtained by the first image signal processing) and outputting a display format image. In the specific implementation process, as shown in FIG. 3, the initial model of image signal processing can include a multi-level wavelet transform image signal processing network based on a convolutional network for biomedical image segmentation (U-Net), can include a multi-layer U-Net (2-layer U-Net is taken as an example for illustration in FIG. 3), and each layer of U-Net can include a wavelet transform down-sampling layer, a wavelet transform up-sampling layer, a convolution and activation layer and a residual module. In the multi-layer U-Net, any layer of U-Net is nested in another layer of U-Net.

[0035] Specifically, the residual module can be composed of a residual channel attention module (RCAB), which can minimize the loss of information in these layers.

[0036] Step 102, determining training samples, the training samples including a plurality of groups of sample images, each group of sample images including an original sample image and a corresponding display sample image.

[0037] Specifically, in determining the training samples, the corresponding display sample images can be obtained directly after traditional ISP processing of each original shooting sample image in the shooting device, or the corresponding display sample images can be obtained after processing the original shooting sample images by using an image editing program (such as PS software, etc.).

[0038] It can be understood that when the shooting device shoots an image, an optical image generated by an object passing through a camera lens is projected onto the surface of an image sensor using a color filter array, is first converted into an analog electrical signal through photoelectric conversion, and is then converted into a digital image signal (i.e., a Bayer format image) through an analog-to-digital converter. In this embodiment, the Bayer format image can be taken as an original shooting sample image. In order to meet the special scene requirements and special quality requirements of different users for shooting images, the original shooting sample image needs to be processed by image signal processing (i.e., ISP processing) to obtain a display format image, such as a joint photographic experts group (JPEG) image.

[0039] Specifically, the traditional ISP processing of the original shooting sample image can include but is not limited to the following processing: black level compensation, lens shading correction, bad pixel correction, color interpolation (demosaic), Bayer noise removal, auto white balance (AWB) correction, color correction matrix (CCM), Gamma correction, demosaicing, color space conversion (conversion from RGB to YUV), removal of color noise and edge enhancement in the YUV color space, color and contrast enhancement, and automatic exposure control.

[0040] Step 103: performing first image signal processing on each original shooting sample image to obtain a first-processed image, and then performing second image signal processing on the first-processed image by using an image signal processing initial model to obtain a corresponding display format image, wherein the first image signal processing is processing related to non-shared parameters of the shooting device, and the second image signal processing is other image signal processing than the first image signal processing.

[0041] The first image signal processing herein mainly refers to processing related to non-shared parameters of the photographing device, the non-shared parameters herein refer to parameters that are not shared (i.e. same) by all photographing devices, and the non-shared parameters of different photographing devices are different, wherein the photographing device refers to a device used to photograph the original sample image. For example, black level correction, normalization, automatic white balance, color correction, demosaicing, etc., mainly involve basic color correction, color space conversion and color detail restoration, etc., which require high accuracy and stability, and the parameters based on these processes of different photographing devices are different, which are related to the non-shared parameters of the photographing device. In the embodiment, the original sample image is processed by a traditional method to provide accurate and stable processing results, which is based on long-term accumulated experience, strict mathematical models and optimized algorithms.

[0042] The second image signal processing is other image signal processing in addition to the first image signal processing, which is generally related to shared parameters of the photographing device, the shared parameters herein refer to parameters that can be shared (i.e. same) by all photographing devices, for example, parameters focusing on color restoration, detail texture recovery, brightness adjustment, contrast adjustment and noise removal, etc. These processes include color, texture, brightness, contrast, noise and other complex data characteristics, and the parameters based on these processes of different photographing devices can be the same.

[0043] In step 104, the image signal processing initial model is adjusted according to the display format image obtained by the image signal processing initial model and the display sample image in the training sample, to obtain the image signal processing model.

[0044] Specifically, the training system of the image signal processing can first calculate the loss function related to the image signal processing initial model according to each display format image obtained by the image signal processing initial model and the corresponding display sample image in the training sample, and then adjust the parameter value in the image signal processing initial model according to the calculated loss function.

[0045] Further, the training process of the image signal processing model is to minimize the value of the loss function, which is to continuously optimize the parameter value of the parameters in the image signal processing initial model determined in step 101 by a series of mathematical optimization methods such as back propagation and gradient descent, and to minimize the calculated value of the loss function.

[0046] It should be noted that the above steps 102 to 104 are the adjustment of the parameter values in the image signal processing initial model based on the display format images obtained by the image signal processing initial model, and in actual application, the above steps 102 to 104 need to be executed in a loop until the adjustment of the parameter values meets a certain stop condition.

[0047] Therefore, after the execution of the above steps 101 to 104 of the embodiment, the training system of the image signal processing also needs to judge whether the current adjustment of the parameter values meets the preset stop condition, and when it meets, the process is ended, and the parameter values of the image signal processing initial model adjusted in the above step 104 are taken as the parameter values in the finally trained image signal processing model; when it does not meet, the above steps 102 to 104 are returned to be executed for the image signal processing initial model after the adjustment of the parameter values, that is, a new batch of training samples is replaced, and the parameter values in the image signal processing initial model are adjusted according to the replaced training samples. The preset stop condition includes but is not limited to any one of the following conditions: the difference between the current adjusted parameter values and the last adjusted parameter values is less than a threshold value, that is, the adjusted parameter values reach convergence; and the number of adjustments of the parameter values is equal to the preset number of times, etc.

[0048] The loss function calculated by the training system of the image signal processing in the process of adjusting the parameter values of the image signal processing initial model can be used to indicate the difference between the display format images obtained by the image signal processing initial model and the display sample images with better quality actually obtained based on the corresponding original shooting sample images (i.e. the display sample images in the training samples), such as cross-entropy loss function, etc.

[0049] In a specific embodiment, when the training system of the image signal processing calculates the loss function related to the image signal processing initial model according to the display format images obtained by the image signal processing initial model and the display sample images in the training samples, one or more loss functions can be calculated, and a first loss function is calculated as the loss function related to the image signal processing initial model according to these loss functions, such as the weighted sum value of these loss functions, and the parameter values in the image signal processing initial model are adjusted according to the first loss function, and specifically these loss functions can include:

[0050] (1) Pixel loss function Loss L1

[0051] The pixel loss function L1 is a loss function that measures the difference between the display format image obtained by the image signal processing initial model and the corresponding display sample image in the training sample at the pixel level. The pixel loss function helps to improve the stability and detail retention capability of the image signal processing initial model for images. Through the characteristics of noise resistance, maintaining image details, sparsity, etc., the generated display format image is more consistent with human eye perception, clearer and more stable. For an original shooting sample image, the calculation method of the pixel loss function is the average of the absolute difference between the display format image obtained by the image signal processing initial model and the corresponding display sample image y i , as shown in the following formula 1, where n is the number of original shooting sample images in the training sample:

[0052] (2) perceptual loss function Loss vgg

[0053] The perceptual loss function, also known as the mean square error loss function (MSE), is a commonly used index to measure the difference between two numerical values. In this embodiment, it is the square of the difference between the high-order features of the display format image obtained by the image signal processing initial model and the high-order features of the corresponding display sample image in the training sample. Specifically, it can be represented by the following formula 2:

[0054] In this embodiment, X and Y in the above formula 2 can be the high-order features feature y and Specifically, a pre-trained Visual Geometry Group (VGG) 16 network can be used to extract features from the display format image obtained by the image signal processing initial model and the corresponding display sample image in the training sample, respectively.

[0055] As shown in FIG. 4, the VGG16 network is a deep neural network that can extract high-order features and can include 16 hidden layers (13 convolutional layers and 3 fully connected layers). Using the perceptual loss function based on the VGG16 network helps the image signal processing initial model to learn the detailed information of the image.

[0056] (3) structure loss function Loss ssim

[0057] ​The structural loss function is also called a structural similarity index (SSIM) loss function, which is an index for measuring the similarity between two images. In this embodiment, the two images are specifically the display format images obtained by the image signal processing initial model and the corresponding display sample image y in the training sample. The SSIM of the two images comprehensively considers various information such as brightness, contrast, and structure. The value range of the SSIM is usually between [0, 1], and the closer to 1 indicates that the two images are more similar. Through the structural loss function, the image signal processing initial model can capture the structure and texture in the image, thereby generating a high-quality image that more comprehensively considers the image features. The SSIM of the two images can be represented by the following public

[0058] formula 3:

[0059] wherein C1=(K1L) 2 , C2=(K2L) 2 , L=2 B -1, u y , respectively represent the mean of the two images, respectively represent the variance of the two images, represents the covariance of the two images, C1 and C2 are stability coefficients, K1 and K2 are both default parameters 0.01 and 0.03, and B is the bit depth of the image. Therefore, the SSIM between the display format image obtained by the image signal processing initial model and the corresponding display sample image y in the training sample is included in the structural loss function, which can be specifically represented by the following formula 4: ssim =1-SSIM (4)

[0060] Further, in order to make the trained image signal processing model more accurately process the image signal, after adjusting the parameter values in the image signal processing initial model according to the first loss function calculated in steps 101 to 104, a preliminary image signal processing initial model is obtained, and in other embodiments, the parameter values in the preliminary image signal processing initial model can be further fine-tuned. The fine-tuning method is similar to the method of adjusting the parameter values in the image signal processing initial model according to the first loss function described above, except that in the fine-tuning process, one or more loss functions are first calculated, and then a second loss function is calculated as a loss function related to the image signal processing initial model according to these loss functions, such as the weighted sum of these loss functions; then the parameter values in the image signal processing initial model are fine-tuned according to the second loss function, such as calculating another overall loss function according to the first loss function and the second loss function, and fine-tuning the parameter values in the image signal processing initial model according to the overall loss function. The loss function used to obtain the second loss function can include:

[0061] (1) Chroma channel loss function Loss UV

[0062] The chroma channel (UV) loss function is a loss function that measures the difference between the generated image and the target image in the UV color space. Compared with the RGB channel, the UV channel is more sensitive to color changes, which can more effectively affect the human eye's perception of color. In this embodiment, the UV loss function is used to measure the difference between the display format image obtained by the image signal processing initial model and the corresponding display sample image in the training sample. The introduction of the UV loss function can enable the image signal processing model to learn and adjust colors more carefully, thereby reducing the perceived color error.

[0063] Specifically, a Gaussian blur operator G(x) with a mean of 0 and a variance σ of 20 is used to blur the display format image obtained by the image signal processing initial model and the display sample image in the training sample, respectively, as shown in the following formula 5: and G(y) i ; Finally, the loss function between the UV channels of the display format image obtained by the image signal processing initial model is calculated, as shown in the following formula 6:

[0064] (2) Adversarial loss function Loss GAN

[0065] The adversarial loss function L1 is based on a loss function obtained by a generative adversarial network (GAN). The GAN network is a deep learning framework composed of a generator and a discriminator. The generator and the discriminator are trained in an adversarial manner in the GAN network.

[0066] In this embodiment, to calculate the adversarial loss function, a discriminator can be determined, and the image signal processing initial model is used as the generator. The discriminator is used to distinguish whether the display sample image in the training sample and the display format image obtained by the image signal processing initial model are display sample images. Thus, the adversarial loss function of the generator can be calculated according to the discrimination result of the discriminator. The purpose of the generator is to make the display format image generated by the image signal processing initial model approximate the display sample image in the training sample, and the discriminator distinguishes the display format image as the display sample image. The purpose of the discriminator is to accurately distinguish whether the input image is the display sample image or the display format image generated by the generator. Usually, a scalar real = 1 is used to represent that the discriminator is very sure that the input image is the display sample image, and a scalar fake = 0 is used to represent that the discriminator is very sure that the input image is not the display sample image.

[0067] Specifically, in this embodiment, the PatchGAN structure is mainly used for the selection of the discriminator. The PatchGAN discriminator distinguishes the local region of the image, and the output is an N*N matrix X. Each element X[i][j] of the matrix X is also called a local region (Patch), which represents the discrimination output of the discriminator on a local receptive field of the input image. Finally, the mean value of each Patch is taken as the discrimination result output of the PatchGAN discriminator on the input image. This local region judgment helps the image signal processing initial model to better generate realistic image local details and textures in the adversarial learning process, and has great significance for improving the overall quality and authenticity of the display format image obtained by the image signal processing model.

[0068] In the training process of this embodiment, the generator (i.e., the image signal processing initial model) and the discriminator are alternately trained, so that they are mutually antagonistic and learn from each other. Finally, the display format image obtained by the image signal processing initial model approximates the display sample image in the training sample, and the discriminator cannot accurately distinguish the difference between the display format image and the display sample image.

[0069] Specifically, as shown in FIG. 5, the generator and the discriminator are adversarially trained. In this process, the display format image obtained by the image signal processing initial model is first input into the discriminator, and the discriminator distinguishes whether the input image is the display sample image. The input is processed by the discriminator, and the output is the probability of determining whether the sample image is displayed. The generator's adversarial loss function, Loss, is calculated using this probability. GAN The initial model (i.e., the generator) for image signal processing is updated using optimization algorithms such as backpropagation and gradient descent. The adversarial loss function includes the probability that the display format image obtained from the initial image signal processing model is identified as the corresponding display sample image by the discriminator. It aims to minimize the probability that the display format image is identified as a non-display sample image by the discriminator, guiding the generator to approximate the display sample image with the display format image. Specifically, this can be expressed by equations 7 and 8, where real = 1 indicates that the discriminator is highly confident that the display format image... This displays a sample image:

[0070] Furthermore, after updating the generator parameters, the display format images obtained from the initial image signal processing model are then processed. The displayed sample image y from the training samples is input into the discriminator to obtain the corresponding probabilities. and P y The loss function L2 is calculated separately for real and fake data, resulting in two loss functions. real and loss fake Finally, the average is used to obtain the adversarial loss function (Loss) of the discriminator. PatchGAN The goal is to minimize the probability that a displayed sample image is misidentified as a non-display sample image, and to minimize the probability that a displayed format image is misidentified as a display sample image. Specifically, this can be expressed by the following formulas 9 to 11: loss real =L2(PatchGAN(y),real) (9)

[0071] After calculating the adversarial loss function of the discriminator, the discriminator's parameters are updated using optimization algorithms such as backpropagation and gradient descent. The updates to the initial image signal model (i.e., the generator) and the discriminator are performed alternately. This alternating training method helps balance the learning processes of the generator and discriminator, preventing one from becoming too powerful and causing training instability. Simultaneously, this strategy allows the generator and discriminator to compete and learn from each other, ultimately achieving the goal of generating images that approximate good quality.

[0072] When the image signal processing model is trained by the method described above, the running logic of the image signal processing model can be preset in the system to realize ISP processing of any original shooting format image to obtain a display format image. Specifically, as shown in FIG. 6, the image signal processing method based on the training in the embodiment can include:

[0073] Step 201, obtaining an original shooting format image.

[0074] Step 202, performing first image signal processing on the original shooting format image to obtain a first processed image, where the first image signal processing is processing related to the non-shared parameters of the shooting device, and the specific first image signal processing method is described in the above embodiment and will not be repeated here.

[0075] Step 203, performing second image signal processing on the first processed image according to the preset image signal processing model to obtain a display format image, where the image signal processing model is used for the second image signal processing on the first processed image, and the second image signal processing is other image signal processing than the first image signal processing.

[0076] In this way, in the training process of the image signal processing model, the ISP processing on the original shooting sample image is divided into two parts, one part is the first image signal processing related to the non-shared parameters of the shooting device, which is performed by using the traditional method, and the other part is the second image signal processing, which is performed by using the image signal processing initial model. When the image signal processing model trained in this way is actually applied to the field of image processing, the quality of the display format image obtained by using the method of the embodiment and the traditional ISP processing is compared, and the quality of the display format image obtained by using the method of the embodiment can be greatly improved, especially in the peak signal-to-noise ratio, structural similarity index, and chroma channel.

[0077] The training method of the image signal processing in the embodiment of the application is described below with a specific application example. As shown in FIG. 7, the method can include the following steps:

[0078] Step 301, determining the structure of each layer included in the image signal processing initial model and the initial value of the parameters in each layer, specifically, the structure of the image signal processing initial model can be as shown in FIG. 3, and will not be repeated here.

[0079] Step 302, determining training samples, where the training samples include multiple groups of sample images, and each group of sample images includes an original shooting sample image and a corresponding display sample image.

[0080] Step 303, after the first image signal processing of each original shooting sample image is performed to obtain a first processed image, the second image signal processing is performed on the first processed image by using the image signal processing initial model to obtain a corresponding display format image.

[0081] Step 304, according to each display format image obtained by the image signal processing initial model and the corresponding display sample image in the training sample, the pixel loss function Loss L1 , the perceptual loss function Loss vgg , and the structural loss function Loss ssim are calculated, and the first loss function is calculated according to the weighted sum of the pixel loss function, the perceptual loss function, and the structural loss function, which can be specifically shown in the following formula 12: Loss pretrain =loss L1 +loss vgg +loss ssim *0.15 (12)

[0082] Among them, the pixel loss function focuses on the accurate matching at the bottom pixel level, the pixel loss function is an important part to ensure the quality of the generated image, and the perceptual loss function considers the high-level perceptual features of the image, so that the generated image is more consistent with the human eye perception. In the research of the second image signal processing based on deep learning, setting the weight of them to be 1 is a common practice. Such weight setting comprehensively considers different levels of information, and ensures that the generated image achieves a good balance in the bottom and high-level features.

[0083] The structural loss function further improves the structural integrity. The overall structure of the image involves the relationship between different elements, for example, the relative position between objects, the coherence of texture, etc. In this embodiment, on the basis of maintaining the pixel level and perceptual similarity, the structural loss function is introduced to participate in the training, and the generated image is further pushed to better maintain the structural similarity with the display sample image as a whole. In the training process, it is found that the value of the structural loss function is large (the order of magnitude is 1e-1). In order to avoid the training process from excessively focusing on the structural loss function and ignoring the pixel loss function and the perceptual loss function (the order of magnitude is 1e-2), the weight of the structural loss function is adjusted to 0.15, so that the numerical magnitude is consistent with the other two, which helps to ensure that the model considers the detail accuracy, the human eye perceptual similarity, and the overall structural similarity when generating the image.

[0084] Step 305, according to the first loss function, the initial value of the parameter in the image signal processing initial model is adjusted, and the purpose of the adjustment is to minimize the first loss function.

[0085] Step 306, judging whether the adjustment of the parameter value in the initial model of image signal processing satisfies the preset stop condition, if yes, taking the parameter value of the signal processing initial model adjusted in the above step 305 as the parameter value in the image signal processing model obtained by training; if not, returning to execute the above step 302 for the image signal processing initial model after adjusting the parameter value, i.e. changing a batch of training samples, and adjusting the parameter value in the image signal processing initial model according to the changed training samples.

[0086] It should be noted that the image signal processing model can be trained by the above steps 301 to 306, and the image signal processing model can be tested on the test set, and the peak signal to noise ratio (PSNR), the structural similarity coefficient (SSIM) and the UV value calculated are 23.49, 0.8865 and 0.0125 respectively, i.e. there is still color error between the display format image obtained by the trained image signal processing model and the display sample image.

[0087] Thus in a specific embodiment, the trained image signal processing model needs to be further fine-tuned, specifically, the image signal processing model can be fine-tuned according to the steps of the above steps 302 to 306, and in this process, a second loss function can be introduced, specifically a chroma channel loss function, i.e. when the parameter value in the image signal processing model is fine-tuned according to the first loss function in the above step 305, an overall loss function can be calculated based on the first loss function and the second loss function according to the following formula 13, and the parameter value in the above trained image signal processing model is adjusted according to the calculated overall loss function: Loss finetrain = Loss pretrain + α × loss UV (13)

[0088] Wherein, α represents the weight value of the chroma channel loss function, since the chroma channel loss function and the pixel loss function, the perception loss function are in the same order of magnitude, in order to make the image signal processing model more focused on color restoration and reducing color error in the fine-tuning stage, a relatively large weight value is allocated to the chroma loss function in this embodiment. When the weight value of the chroma channel loss function is 1, 4, 10, 15 and 20 respectively, the results on the test set are shown in Table 1 below, and it is found that when the weight value is 10, not only the UV color error can be further reduced, but also the PSNR and SSIM values can be further improved.

[0089] Table 1

[0090] In another specific embodiment, after the image signal processing model is trained by the above steps 301 to 306, in order to further promote the image signal processing model to obtain a more realistic display format image and have a certain stylization effect. In this embodiment, an adversarial loss function Loss GAN Because the training of the GAN network is unstable, it is a reasonable choice to set a small weight value of the adversarial loss function, especially in the fine-tuning stage. Because in fine-tuning, the main goal is to maintain the accuracy and reality of the image, and not to emphasize the reality generated by the GAN network too much.

[0091] In this case, when further fine-tuning the trained image signal processing model, specifically, the image signal processing model can be fine-tuned according to the steps of steps 302 to 306, and in this process, another second loss function, specifically an adversarial loss function, can be introduced. That is, when fine-tuning the parameter values in the image signal processing model according to the first loss function in step 305, a loss function can be calculated according to the following formula 14, and the parameter values in the trained image signal processing model can be adjusted according to the calculated loss function: Loss finetrain = Loss pretrain + α × loss UV + β × loss GAN (14)

[0092] In the fine-tuning stage, it is found that the numerical magnitude of the adversarial loss function is 1e-1, in order to make its numerical magnitude less than that of other loss functions, in this embodiment, the weight value of the adversarial loss function is set to 0.0001, 0.001, 0.01 and 0.1 respectively. The PSNR, SSIM and UV values calculated on the test set are shown in Table 2 below, it is found that when the weight value is set to 0.0001, the adversarial loss function has less effect on the image signal processing model (i.e. the second row of data is close to the first row), and when the weight value is set to 0.001, not only can the UV color error be further reduced, but also the PSNR and SSIM values can be further improved.

[0093] Table 2

[0094] It should be noted that in the training process of the image signal processing model, the original shooting sample image can be first processed by the traditional method to complete the first image signal processing, which can provide more accurate reference for the training of the image signal processing model, so that the image signal processing model can better adapt to the characteristics and hardware differences of different cameras, and improve the generalization ability of the model to different cameras. After the image signal processing model is trained by the method in steps 301 to 306, the running logic of the image signal processing model is preset to the system, and then the image signal processing can be performed according to the following method as shown in FIG. 8:

[0095] In step 401, an original shooting format image shot by a shooting device is obtained, which can be a Bayer format image.

[0096] In step 402, the original shooting format image is subjected to black level correction and normalization processing to obtain a normalized image, specifically:

[0097] The sensors and hardware configurations of different shooting devices can have great differences, resulting in different black level and white level values. In this embodiment, the traditional black level correction and normalization processing is adopted. Specifically, first, the data type of the original shooting format image Image unit16 is converted from uint16 to float32 to ensure that no overflow occurs during numerical calculation, then each pixel of the converted image is subtracted by the black level value, and the value less than 0 is truncated to 0, that is, the BLC process; finally, the corrected image value is divided by the difference between the white level white_level and the black level black_level, and the dynamic range of the image is normalized to [0, 1], thereby obtaining the normalized image. This can facilitate subsequent numerical operations and also make the image present appropriate contrast and details under different lighting conditions. Specifically, it can be represented by the following formula 15: the above process is represented by the following formula:

[0098] In step 403, the normalized image is subjected to demosaicing processing to obtain a demosaiced image. Specifically:

[0099] The original shooting format, i.e. the Bayer format image, is a color filter array image supporting different format arrangements. Each pixel only contains red, green and blue (R, G, B) single color light, instead of complete RGB information. Therefore, a single-channel image needs to be converted into a complete RGB color image through a demosaicing algorithm to obtain a demosaicing processed image. Demosaicing mainly interpolates the original shooting format image into a complete RGB image. Taking a bilinear interpolation algorithm as an example, interpolation is performed in the horizontal and vertical directions, and the missing color channel is estimated by weighted average of surrounding pixels. Specifically, interpolation can be performed in the horizontal and vertical directions in the manner of formula 16 to estimate the G channel pixel:

[0100] As shown in formula 17, interpolation is performed in the diagonal direction to estimate the R and B channel pixels:

[0101] As shown in formula 18, the interpolated G, R and B channel values are used to obtain a complete RGB image:

[0102] Since the Bayer format is not fixed, there are many different formats, and different cameras can select different Bayer formats. The demosaicing process for each Bayer format involves complex image features, nonlinear relationships, etc. The traditional demosaicing processing has unique advantages because it can directly restore complete RGB information through interpolation, supports different Bayer formats, has good anti-aliasing effect, can preserve image details and edges, and accurately restores color information.

[0103] In step 404, the demosaicing processed image is subjected to white balance processing to obtain a white balance processed image. Specifically:

[0104] Cameras usually use red gain (R Gain ), green gain (G Gain ) and blue gain (B Gain ) to adjust the white balance so that the white point in the image looks neutral white under various lighting conditions. Specifically, for the original RGB value of each pixel, the corresponding color gain is multiplied to achieve the white balance adjustment effect. Different camera systems use different white balance algorithms, so they have different color gain parameters. In this embodiment, the color gain parameters in the camera are directly read, and the R, G and B values of each pixel in the demosaicing processed image are multiplied by the corresponding color gain to obtain a white balance processed image Image wb .

[0105] Step 405, color correction is performed on the white balance processed image to obtain a corrected image, which is the first processed image mentioned above. Specifically:

[0106] The camera color space is designed and implemented by the camera manufacturer to represent and process the color information captured by the camera. Different camera manufacturers may use different sensors, image processing procedures and color models, so their camera color spaces will be different. In order to ensure the color consistency of the image on different hardware devices, it is necessary to map the image from the camera color space to the standard color space through color correction. The color correction matrix of the camera is obtained by calibration and calibration of the camera manufacturer. By multiplying the original color vector of the image by the color correction matrix, the image can be mapped from the camera color space to the standard color space, so that the image presents standard colors on different devices.

[0107] Since different cameras have different color correction matrices, in this embodiment, the white balance processed image Image wb is multiplied by the color correction matrix M to obtain the corrected image Image corrected , which is specifically shown in the following formula 19:

[0108] Step 406, the corrected image is processed again according to the preset image signal processing model to obtain a display format image.

[0109] The server provided in the embodiment of the present application can have great differences due to different configurations or performances, and can include one or more central processing units (CPU) 20 (for example, one or more processors) and a memory 21, one or more storage media 22 (for example, one or more mass storage devices) for storing application programs 221 or data 222. Among them, the memory 21 and the storage medium 22 can be temporary storage or persistent storage. The programs stored in the storage medium 22 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the central processing unit 20 can be configured to communicate with the storage medium 22 and execute a series of instruction operations in the storage medium 22 on the server.

[0110] Specifically, the application 221 stored in the storage medium 22 comprises an application of training of image signal processing, and the application is adapted to be loaded by the processor and execute the above-mentioned training method of image signal processing, which is not described herein. Further, the central processor 20 can be configured to communicate with the storage medium 22, and perform a series of operations corresponding to the application of training of image signal processing stored in the storage medium 22 on the server.

[0111] The server can further comprise one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input / output interfaces 25, and / or one or more operating systems 223, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0112] The steps performed by the training system of image signal processing in the above-mentioned method embodiments can be based on the structure of the server shown in Fig. 9.

[0113] Further, the embodiments of the present application also provide a computer readable storage medium storing a plurality of computer programs, which are adapted to be loaded by the processor and execute the above-mentioned image signal processing method performed by the training system of image signal processing.

[0114] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by programs instructing the related hardware, and the programs can be stored in a computer readable storage medium, which can include read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0115] The above describes in detail the training method of image signal processing, the storage medium and the server provided by the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples in this paper. The above-mentioned embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In summary, the content of the present description should not be understood as the limitation of the present application.

Claims

1. A training method for image signal processing, comprising: An initial model for image signal processing is determined; wherein the initial model for image signal processing includes a multi-layer convolutional network for image segmentation, and each layer of the convolutional network for image segmentation includes a wavelet transform downsampling layer, a wavelet transform upsampling layer, a convolutional and activation layer, and a residual module; wherein any layer of the convolutional network for image segmentation is nested within another layer of the convolutional network for image segmentation. The training samples are determined, which include multiple sets of sample images, each set of sample images including the original captured sample image and its corresponding display sample image; Each of the original captured sample images is subjected to a first image signal processing to obtain a first-processed image. Then, the first-processed image is subjected to a second image signal processing through the image signal processing initial model to obtain a corresponding display format image. The first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing. Calculate the pixel loss function, which includes: the average of the absolute differences between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; Calculate the perceptual loss function, which includes: the mean square value of the difference between the high-order features of the display format image obtained by the initial image signal processing model and the high-order features of the corresponding display sample image in the training samples; Calculate the structural loss function, which includes: the structural similarity index between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; Calculate a first loss function based on the pixel loss function, the perceptual loss function, and the structural loss function, and adjust the parameter values ​​in the initial model of the image signal processing based on the first loss function; Calculate the chroma channel loss function, which is used to indicate the difference in chroma channels between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; Calculate the adversarial loss function, which includes the probability that the display format image obtained from the initial model of the image signal is identified by the discriminator as the corresponding display sample image; Calculate the second loss function based on the chroma channel loss function and / or the adversarial loss function; The parameter values ​​in the initial model of image signal processing are fine-tuned according to the second loss function to obtain the image signal processing model.

2. The training method for image signal processing according to claim 1, wherein calculating the adversarial loss function specifically includes: A discriminator is determined, and the initial image signal processing model is used as a generator. The discriminator is used to determine whether the display sample image and the display format image obtained by the initial image signal processing model are display sample images. The adversarial loss function of the generator is calculated based on the discrimination result of the discriminator.

3. The training method for image signal processing according to claim 1, wherein fine-tuning the parameter values ​​in the initial model of image signal processing according to the second loss function specifically includes: Calculate another overall loss function based on the first loss function and the second loss function, and fine-tune the parameter values ​​in the initial model of the image signal processing based on the overall loss function.

4. The image signal processing training method according to claim 1, wherein the method further comprises: Acquire images in their original shooting format; The original captured image is subjected to first image signal processing to obtain the first processed image; The first image signal processing is processing related to non-shared parameters of the imaging device; The image after the first processing is processed again according to a preset image signal processing model to obtain a display format image. The image signal processing model is used to perform a second image signal processing on the image after the first processing. The second image signal processing is other image signal processing besides the first image signal processing.

5. A training method for image signal processing, comprising: Determine the initial model for image signal processing; The training samples are determined, which include multiple sets of sample images, each set of sample images including the original captured sample image and its corresponding display sample image; Each of the original captured sample images is subjected to a first image signal processing to obtain a first-processed image. Then, the first-processed image is subjected to a second image signal processing through the image signal processing initial model to obtain a corresponding display format image. The first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing. Based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, the initial image signal processing model is adjusted to obtain the image signal processing model.

6. The image signal processing training method as described in claim 5, wherein determining the initial model for image signal processing specifically includes: The initial model for image signal processing is determined to include: a multi-layer convolutional network for image segmentation, wherein each layer of the convolutional network for image segmentation includes a wavelet transform downsampling layer, a wavelet transform upsampling layer, a convolution and activation layer, and a residual module; In the multi-layer image segmentation convolutional network, any layer of the image segmentation convolutional network is nested within... In another layer of the image segmentation convolutional network.

7. The image signal processing training method as described in claim 5, wherein adjusting the image signal processing initial model based on the display format image obtained from the image signal processing initial model and the display sample images in the training samples specifically includes: Calculate the pixel loss function, which includes: the average of the absolute differences between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; Calculate the perceptual loss function, which includes: the mean square value of the difference between the high-order features of the display format image obtained by the initial image signal processing model and the high-order features of the corresponding display sample image in the training samples; Calculate the structural loss function, which includes: the structural similarity index between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; A first loss function is calculated based on the pixel loss function, the perceptual loss function, and the structural loss function, and the parameter values ​​in the initial model of the image signal processing are adjusted based on the first loss function.

8. The image signal processing training method as described in claim 7, wherein the method further comprises: Calculate the chroma channel loss function, which is used to indicate the difference in chroma channels between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; Calculate the adversarial loss function, which includes the probability that the display format image obtained from the initial model of the image signal is identified by the discriminator as the corresponding display sample image; Calculate the second loss function based on the chroma channel loss function and / or the adversarial loss function; The parameter values ​​in the initial model of image signal processing are fine-tuned according to the second loss function to obtain the image signal processing model.

9. The image signal processing training method as described in claim 8, wherein calculating the adversarial loss function specifically includes: A discriminator is determined, and the initial image signal processing model is used as a generator. The discriminator is used to determine whether the display sample image and the display format image obtained by the initial image signal processing model are display sample images. The adversarial loss function of the generator is calculated based on the discrimination result of the discriminator.

10. The training method for image signal processing as described in claim 8, wherein fine-tuning the parameter values ​​in the initial model of image signal processing according to the second loss function specifically includes: Calculate another overall loss function based on the first loss function and the second loss function, and fine-tune the parameter values ​​in the initial model of the image signal processing based on the overall loss function.

11. The image signal processing training method as described in claim 5, wherein the method further comprises: Acquire images in their original shooting format; The original captured image is subjected to first image signal processing to obtain the first processed image; The first image signal processing is processing related to non-shared parameters of the imaging device; The image after the first processing is processed again according to a preset image signal processing model to obtain a display format image. The image signal processing model is used to perform a second image signal processing on the image after the first processing. The second image signal processing is other image signal processing besides the first image signal processing.

12. A computer-readable storage medium storing a plurality of computer programs adapted to be loaded by a processor and executed by the training method for image signal processing as claimed in claim 1.

Images