Training method, system and storage medium for image signal processing

Abstract

This invention discloses a training method, system, and storage medium for image signal processing, applicable to the field of image processing technology. In the training process of the image signal processing model, the ISP processing performed on the original captured sample images needs to be divided. A first part of the image signal processing, related to non-shared parameters of the capturing device, is performed using conventional methods. The second part of the image signal processing is performed using the initial image signal processing model. When the trained image signal processing model is actually applied to the field of image processing, its quality is significantly improved compared to display format images obtained using conventional ISP processing. The method described in this embodiment demonstrates particularly excellent performance in peak signal-to-noise ratio, structural similarity index, and chroma channels.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a training method, system and storage medium for image signal processing. Background Technology

[0002] Optical images of objects in nature, generated by a camera lens, are projected onto the surface of an image sensor using a color filter array. The images are first converted from photoelectric signals to analog electrical signals, then converted into digital image signals (Bayer format images) by an analog-to-digital converter (A / D), and finally sent to a digital signal processing chip (DSP) for image signal processing (ISP) to form an RGB format image.

[0003] Typical ISP processing can include: black level compensation, lens shading correction, bad pixel correction, color interpolation, Bayer noise removal, white balance correction, color correction, gamma correction, color space conversion, removal of color noise and edge enhancement in the YUV color space, color and contrast enhancement, etc. As the complexity of image processing scenarios increases and the special requirements for image quality become more stringent, typical ISPs face challenges such as a large parameter library, difficulty in debugging, and long development cycles. Summary of the Invention

[0004] This invention provides a training method, system, and storage medium for image signal processing, which improves the quality of images processed by the trained image signal processing model.

[0005] One embodiment of the present invention provides a training method for image signal processing, comprising:

[0006] Determine the initial model for image signal processing;

[0007] The training samples are determined, which include multiple sets of sample images, each set of sample images including the original captured sample image and its corresponding display sample image;

[0008] Each of the original captured sample images is subjected to a first image signal processing to obtain a first-processed image. Then, the first-processed image is subjected to a second image signal processing through the image signal processing initial model to obtain a corresponding display format image. The first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing.

[0009] Based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, the initial image signal processing model is adjusted to obtain the image signal processing model.

[0010] Another aspect of this invention provides a training system for image signal processing, comprising:

[0011] The model determination unit is used to determine the initial model for image signal processing;

[0012] A sample determination unit is used to determine training samples, wherein the training samples include multiple sets of sample images, and each set of sample images includes an original captured sample image and its corresponding display sample image;

[0013] The sample processing unit is used to perform a first image signal processing on each of the original captured sample images to obtain a first-processed image, and then perform a second image signal processing on the first-processed image through the image signal processing initial model to obtain a corresponding display format image; the first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing;

[0014] The adjustment unit is used to adjust the initial image signal processing model based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, so as to obtain the image signal processing model.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the image signal processing training method as described in one aspect of the present invention.

[0016] As can be seen, in the training process of the image signal processing model in this embodiment of the invention, it is necessary to divide the ISP processing of the original captured sample image. The first image signal processing, which is related to the non-shared parameters of the capturing device, is performed using the traditional method, while the second image signal processing is performed using the initial image signal processing model. When the image signal processing model trained in this way is actually applied to the field of image processing, the quality of the display format image obtained by the method of this embodiment can be greatly improved compared with that obtained by the traditional ISP processing. In particular, it performs well in terms of peak signal-to-noise ratio, structural similarity index, and chroma channel. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an image signal processing training method provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of a training method for image signal processing provided in one embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of an initial model for image signal processing in one embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of a VGG16 network in one embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram illustrating the calculation of the adversarial loss function in one embodiment of the present invention;

[0023] Figure 6 This is a flowchart of an image signal processing method in one embodiment of the present invention;

[0024] Figure 7 This is a flowchart of training an image signal processing model in a specific application embodiment of the present invention;

[0025] Figure 8 This is a flowchart of an image signal processing method provided in a specific application embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram of the logical structure of an image signal processing training system provided in an embodiment of the present invention;

[0027] Figure 10 This is a schematic diagram of the logical structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] This invention provides a training method for image signal processing, which mainly trains an image signal processing model for use in the process of processing the original captured image to obtain a display format image after the image is captured by an imaging device. Figure 1 As shown, the image signal processing training system can perform image signal processing training as follows:

[0031] An initial image signal processing model is determined; training samples are determined, including multiple sets of sample images, each set including an original captured sample image and its corresponding display sample image; a first image signal processing is performed on each of the original captured sample images to obtain an image after initial processing, and then a second image signal processing is performed on the image after initial processing using the initial image signal processing model to obtain a corresponding display format image; the first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing; based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, the initial image signal processing model is adjusted to obtain an image signal processing model.

[0032] The training system for image signal processing can be applied to terminal devices or servers. Terminal devices can include, but are not limited to, the following electronic devices that require image processing: mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, etc.

[0033] The aforementioned pre-set image signal processing model is a machine learning model based on artificial intelligence (AI). It can be trained using certain methods. AI is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, machine learning, and deep learning.

[0035] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0036] When the image signal processing model trained in this way is applied to the field of image processing, the quality of the display format image obtained by the method of this embodiment can be greatly improved when compared with the display format image obtained by traditional ISP processing. In particular, it performs well in terms of peak signal-to-noise ratio, structural similarity index and chroma channel.

[0037] One embodiment of the present invention provides a training method for image signal processing, mainly a method executed by an image signal processing training system, the flowchart of which is shown below. Figure 2 As shown, it includes:

[0038] Step 101: Determine the initial model for image signal processing.

[0039] It is understandable that when determining the initial model for image signal processing, the training system will determine the multi-layer structure included in the initial model and the initial values ​​of the parameters in each layer. The parameters of the initial model refer to the fixed parameters used in the computation of each layer during the initial model, which do not require constant reassignment, such as parameter size, weight values, and user vector length.

[0040] Specifically, the initial image signal processing model is used to perform a second image signal processing on the image after the first processing (obtained through the first image signal processing), and output a display format image. In the specific implementation process, such as... Figure 3 As shown, the initial model for image signal processing can include a multi-level wavelet transform image signal processing network based on image segmentation convolutional networks (U-Net), and can include multi-layer U-Nets (…). Figure 3 (Taking a 2-layer U-Net as an example) Each U-Net layer can include: wavelet transform downsampling layer, wavelet transform upsampling layer, convolution and activation layers, and residual module. In a multi-layer U-Net, any U-Net layer is nested in another U-Net layer.

[0041] Specifically, the residual module can be composed of residual channel attention modules (RCAB), which can minimize information loss in these layers.

[0042] Step 102: Determine the training samples. The training samples include multiple sets of sample images. Each set of sample images includes the original captured sample image and its corresponding display sample image.

[0043] Specifically, when determining training samples, each original captured sample image can be directly processed by traditional ISP in the imaging device to obtain the corresponding display sample image; or the original captured sample image can be processed by an image editing program (such as Photoshop) to obtain the corresponding display sample image.

[0044] It is understood that when the imaging device captures an image, the optical image of the object generated by the camera lens is projected onto the surface of an image sensor using a color filter array. This image is first converted from photoelectric signal to analog electrical signal, and then converted into a digital image signal (Bayer format image) after passing through an analog-to-digital converter. In this embodiment, this Bayer format image can be used as the original captured sample image. To meet the specific scene and quality requirements of different users, image signal processing (ISP) is required on the original captured sample image to obtain a display format image, such as a lossy compression format (Joint Photographic Experts Group, IPEG) image.

[0045] Specifically, traditional ISP processing of original captured sample images may include, but is not limited to, the following: Black Level Compensation, Lens Shading Correction, Bad Pixel Correction, Demosaic, Bayer noise removal, Autowhite Balance (AWB) correction, Color Correction Matrix (CCM), Gamma correction, Demosaicing, Color Space Conversion (RGB to YUV), noise removal and edge enhancement in the YUV color space, color and contrast enhancement, and automatic exposure control in between.

[0046] Step 103: Perform first image signal processing on each original captured sample image to obtain the first processed image, and then perform second image signal processing on the first processed image through the initial image signal processing model to obtain the corresponding display format image. The first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing.

[0047] The first image signal processing here mainly refers to processing related to the non-shared parameters of the capturing device. These non-shared parameters are not parameters shared (i.e., identical) by all capturing devices; they differ between different capturing devices. The capturing device refers to the device used to capture the aforementioned original sample image. Processes such as black level correction, normalization, automatic white balance, color correction, and de-mosaic mainly involve basic color correction, color space conversion, and color detail restoration. These processes require high accuracy and stability, and the parameters used for these processes differ between different capturing devices, thus relating to the non-shared parameters of the capturing device. In this embodiment, the first image signal processing of the original sample image using traditional methods can provide accurate and stable processing results, stemming from its foundation of long-term accumulated experience, rigorous mathematical models, and optimized algorithms.

[0048] The second image signal processing is other image signal processing besides the first image signal processing mentioned above. It is generally related to the shared parameters of the shooting device. Here, shared parameters refer to parameters that can be shared (i.e., the same) by all shooting devices. For example, parameters that focus on color reproduction, detail and texture restoration, brightness adjustment, contrast adjustment, and noise removal. These processing methods include complex data features such as color, texture, brightness, contrast, and noise. Different shooting devices can have the same parameters based on these processing methods.

[0049] Step 104: Adjust the initial image signal processing model based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples to obtain the image signal processing model.

[0050] Specifically, the image signal processing training system can first calculate the loss function related to the image signal processing initial model based on the images of various display formats obtained from the image signal processing initial model and the corresponding display sample images in the training samples, and then adjust the parameter values ​​in the image signal processing initial model according to the calculated loss function.

[0051] Furthermore, the training process of the image signal processing model aims to minimize the value of the aforementioned loss function. This training process continuously optimizes the parameter values ​​of the initial image signal processing model determined in step 101 through a series of mathematical optimization techniques such as backpropagation differentiation and gradient descent, thereby minimizing the calculated value of the aforementioned loss function.

[0052] It should be noted that steps 102 to 104 above are adjustments to the parameter values ​​in the initial image signal processing model for each display format image obtained from the initial image signal processing model. In practical applications, steps 102 to 104 above need to be executed repeatedly until the adjustment of the parameter values ​​meets certain stopping conditions.

[0053] Therefore, after executing steps 101 to 104 of the above embodiment, the image signal processing training system also needs to determine whether the current adjustment of the parameter values ​​meets the preset stopping conditions. If it does, the process ends, and the parameter values ​​of the initial image signal processing model adjusted in step 104 are used as the parameter values ​​in the finally trained image signal processing model. If it does not meet the conditions, for the initial image signal processing model with adjusted parameter values, steps 102 to 104 are executed again, that is, a new batch of training samples is used, and the parameter values ​​in the initial image signal processing model are adjusted according to the new training samples. The preset stopping conditions include, but are not limited to, any one of the following conditions: the difference between the currently adjusted parameter value and the previously adjusted parameter value is less than a threshold, that is, the adjusted parameter value has converged; and the number of parameter value adjustments is equal to the preset number, etc.

[0054] In particular, the loss function calculated by the training system of image signal processing during the process of adjusting the parameter values ​​of the initial model of image signal processing can be used to indicate the difference between the various display format images obtained by the initial model of image signal processing and the better quality display sample images (i.e., display sample images in the training samples) obtained based on the corresponding original captured sample images. For example, it can be represented by the cross-entropy loss function.

[0055] In a specific embodiment, when the image signal processing training system calculates the loss function related to the initial image signal processing model based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, it may first calculate, but is not limited to, one or more of the following loss functions, and then calculate a first loss function as the loss function related to the initial image signal processing model based on these loss functions. For example, the first loss function may be a weighted sum of these loss functions. The parameter values ​​in the initial image signal processing model may then be adjusted based on the first loss function. Specifically, these loss functions may include:

[0056] (1) Pixel loss function Loss L1

[0057] The L1 pixel loss function is a loss function that measures the difference between the display format image obtained from the initial image signal processing model and the corresponding display sample image in the training samples at the pixel level. This pixel loss function helps improve the stability and detail preservation ability of the initial image signal processing model. Through its noise resistance, preservation of image details, and sparsity properties, it makes the generated display format image more consistent with human visual perception, clearer, and more stable. For an original captured sample image, the pixel loss function is calculated based on the display format image obtained from the initial image signal processing model. With the corresponding display sample image y i The average of the absolute differences between them is shown in Formula 1 below, where n is the number of original captured sample images in the training samples:

[0058]

[0059] (2) Perceptual Loss Function vgg

[0060] The perceptual loss function, also known as the mean squared error (MSE) loss function, is a commonly used metric to measure the difference between two numerical values. In this embodiment, it specifically refers to the mean squared value of the difference between the high-order features of the display format image obtained from the initial model of image signal processing and the high-order features of the corresponding display sample images in the training samples. This can be expressed by the following formula 2:

[0061]

[0062] In this embodiment, X and Y in Formula 2 above can be the high-order features of the display format image obtained from the initial model of image signal processing and the corresponding display sample image in the training samples, respectively. y and Specifically, features can be extracted from the display format image obtained from the initial model of image signal processing and the corresponding display sample image in the training samples using a pre-trained Visual Geometry Group (VGG)16 network.

[0063] Among them, such as Figure 4 As shown, the VGG16 network is a deep neural network that can extract high-order features. It can include 16 hidden layers (13 convolutional layers and 3 fully connected layers). Using a perceptual loss function based on the VGG16 network helps the initial model of image signal processing learn detailed information about the image.

[0064] (3) Structural loss function Loss ssim

[0065] The structural loss function, also known as the structural similarity index (SSIM) loss function, is an indicator that measures the degree of similarity between two images. In this embodiment, the two images are specifically the display format images obtained from the initial image signal processing model. The corresponding display sample image y in the training samples. The SSIM of the two images comprehensively considers information such as brightness, contrast, and structure. The value of SSIM is usually between [0,1]. The closer it is to 1, the more similar the two images are. This structural loss function helps the initial model of image signal processing to capture the structure and texture in the image, thereby generating a high-quality image that more comprehensively considers image features. The SSIM of the two images can be obtained through the following formula.

[0066] Equation 3 represents:

[0067]

[0068] Where C1 = (K1L) 2 C2 = (K2L) 2 L = 2 B -1, u y , Let represent the means of the two images respectively. Let represent the variances of the two images respectively. Let C1 and C2 represent the covariance of the two images, K1 and K2 be the stability coefficients, and both be the default parameters of 0.01 and 0.03, respectively. Let B be the image bit depth. Then, the structural loss function includes the display format image obtained from the initial image signal processing model. The SSIM between the training sample and the corresponding displayed sample image y can be specifically expressed by the following formula 4:

[0069] Loss ssim =1-SSIM (4)

[0070] Furthermore, to enable the trained image signal processing model to process image signals more accurately, after adjusting the parameter values ​​in the initial image signal processing model using the first loss function calculated in steps 101 to 104 above, a preliminary initial image signal processing model is obtained. In other embodiments, the parameter values ​​in the preliminary initial image signal processing model can be further fine-tuned. The fine-tuning method is similar to the method described above for adjusting the parameter values ​​in the initial image signal processing model using the first loss function. The difference is that, in the fine-tuning process, one or more loss functions are calculated first, and then a second loss function is calculated based on these loss functions as a loss function related to the initial image signal processing model. For example, the second loss function is a weighted sum of these loss functions. Then, the parameter values ​​in the initial image signal processing model are fine-tuned based on the second loss function. For example, another overall loss function is calculated based on the first and second loss functions, and the parameter values ​​in the initial image signal processing model are fine-tuned based on the overall loss function. The loss function used to obtain the second loss function may include:

[0071] (1) Color channel loss function Loss UV

[0072] The chroma channel (UV) loss function measures the difference between the generated and target images in the UV color space. Compared to the RGB channels, the UV channels are more sensitive to color changes, which have a greater impact on human color perception. In this embodiment, the UV loss function is used to measure the difference in the chroma channels between the initial display format image of the image signal processing model and the corresponding display sample images in the training samples. Introducing the UV loss function allows the image signal processing model to learn and adjust colors more meticulously, thereby reducing perceptual color errors.

[0073] Specifically, the Gaussian blur operator G(x) with a mean of 0 and a variance σ of 20, as shown in Formula 5, is first used to blur the display format image obtained from the initial model of image signal processing and the display sample image in the training samples, respectively, to obtain the blurred image; then, a linear transformation is used to map the blurred image from the RGB color space to the YUV color space to obtain... and G(y) i Finally, the loss function between the UV channels of the display format image obtained from the initial image signal processing model is calculated, as shown in Formula 6 below:

[0074]

[0075] (2) Adversarial loss function Loss GAN

[0076] The adversarial loss function L1 is based on the loss function of Generative Adversarial Network (GAN), a deep learning framework consisting of a generator and a discriminator. The generator and discriminator are trained in the GAN network through adversarial training.

[0077] In this embodiment, to calculate the adversarial loss function, a discriminator can be determined. The aforementioned image signal processing initial model is used as the generator. The discriminator is used to distinguish whether the display sample image in the training samples and the display format image obtained by the image signal processing initial model are display sample images. Thus, the adversarial loss function of the generator can be calculated based on the discriminator's judgment result. The generator's purpose is to make the display format image generated by the image signal processing initial model approximate the display sample image in the training samples, and the discriminator classifies the display format image as a display sample image. The discriminator's purpose is to accurately determine whether the input image is a display sample image or a display format image generated by the generator. Typically, a scalar `real = 1` indicates that the discriminator is very confident that the input image is a display sample image, and a scalar `fake = 0` indicates that the discriminator is very confident that the input image is a non-display sample image.

[0078] Specifically, this embodiment primarily employs the PatchGAN structure for the discriminator. The PatchGAN discriminator discriminates by classifying local regions of an image, and its output is an N*N matrix X. Each element X[i][j] of matrix X is also called a local region (Patch), representing the discriminator's output for a local receptive field of the input image. Finally, the average of all Patches is taken as the discriminant result of the PatchGAN discriminator on the input image. This local region judgment helps the initial image signal processing model generate more realistic local image details and textures during adversarial learning, which is of great significance for improving the overall quality and realism of the display format image obtained by the image signal processing model.

[0079] In the training process of this embodiment, the generator (i.e., the initial model of image signal processing) and the discriminator are trained alternately, so that they compete against each other and learn from each other. In the end, the display format image obtained by the initial model of image signal processing approximates the display sample image in the training samples, and the discriminator is unable to accurately distinguish the difference between the display format image and the display sample image.

[0080] Specifically, such as Figure 5 The adversarial training of the generator and discriminator is shown. In this process, the display format image obtained from the initial model of image signal processing is first processed. The input is processed by the discriminator, and the output is the probability of determining whether the sample image is displayed. The generator's adversarial loss function, Loss, is calculated using this probability. GAN The initial model (i.e., the generator) for image signal processing is updated using optimization algorithms such as backpropagation and gradient descent. The adversarial loss function includes the probability that the display format image obtained from the initial image signal processing model is identified as the corresponding display sample image by the discriminator. It aims to minimize the probability that the display format image is identified as a non-display sample image by the discriminator, guiding the generator to approximate the display sample image with the display format image. Specifically, this can be expressed by equations 7 and 8, where real = 1 indicates that the discriminator is highly confident that the display format image... This displays a sample image:

[0081]

[0082] Furthermore, after updating the generator parameters, the display format images obtained from the initial image signal processing model are then processed. The displayed sample image y from the training samples is input into the discriminator to obtain the corresponding probabilities. and P y The loss function L2 is calculated separately for real and fake data, resulting in two loss functions. real and loss fake Finally, the average is used to obtain the adversarial loss function (Loss) of the discriminator. PatchGAN The aim is to minimize the probability that a displayed sample image is misidentified as a non-display sample image, and to minimize the probability that a displayed format image is misidentified as a display sample image. This can be specifically expressed by the following formulas 9 to 11:

[0083] loss real =L2(PatchGAN(y),real) (9)

[0084]

[0085] After calculating the adversarial loss function of the discriminator, the discriminator's parameters are updated using optimization algorithms such as backpropagation and gradient descent. The updates to the initial image signal model (i.e., the generator) and the discriminator are performed alternately. This alternating training method helps balance the learning processes of the generator and discriminator, preventing one from becoming too powerful and causing training instability. Simultaneously, this strategy allows the generator and discriminator to compete and learn from each other, ultimately achieving the goal of generating images that approximate good quality.

[0086] Once the image signal processing model is trained using the methods described above, its operational logic can be pre-programmed into the system to perform ISP processing on any original captured image format to obtain a display format image. Specifically, for example... Figure 6 As shown, the image signal processing method based on training in this embodiment may include:

[0087] Step 201: Obtain the original captured image format.

[0088] Step 202: Perform first image signal processing on the original captured image to obtain the first processed image. Here, the first image signal processing is related to the non-shared parameters of the capturing device. The specific first image signal processing method is described in the above embodiments and will not be repeated here.

[0089] Step 203: The image after the first processing is processed again according to the preset image signal processing model to obtain the display format image. The image signal processing model is used to perform a second image signal processing on the image after the first processing. The second image signal processing is other image signal processing besides the first image signal processing.

[0090] Thus, during the training of the image signal processing model, the ISP processing performed on the original captured sample images needs to be divided. The first image signal processing, which is related to the non-shared parameters of the capturing device, is performed using the traditional method, while the second image signal processing is performed using the initial image signal processing model. When the trained image signal processing model is actually applied to the field of image processing, the quality of the display format images obtained by the method of this embodiment and by the traditional ISP processing will be compared. The quality of the display format images obtained by the method of this embodiment can be greatly improved, especially in terms of peak signal-to-noise ratio, structural similarity index, and chroma channels.

[0091] The following example illustrates the training method for image signal processing in this embodiment of the invention. Figure 7 As shown, the specific process may include the following:

[0092] Step 301: Determine the layer structure of the initial image signal processing model and the initial values ​​of the parameters in each layer. Specifically, the structure of the initial image signal processing model can be as follows: Figure 3 As shown, it will not be elaborated upon here.

[0093] Step 302: Determine the training samples. The training samples include multiple sets of sample images. Each set of sample images includes the original captured sample image and its corresponding display sample image.

[0094] Step 303: After performing the first image signal processing on each original captured sample image to obtain the first processed image, the second image signal processing is then performed on the first processed image through the initial image signal processing model to obtain the corresponding display format image.

[0095] Step 304: Based on the display format images obtained from the initial image signal processing model and the corresponding display sample images from the training samples, calculate the pixel loss function Loss mentioned above. L1 Perceptual loss function Loss vgg Structural loss function Loss ssim The first loss function is calculated based on the weighted sum of the pixel loss function, the perceptual loss function, and the structural loss function, as shown in Formula 12 below:

[0096] Loss pretrain =loss L1 +loss vgg +loss ssim *0.15 (12)

[0097] Among them, the pixel loss function focuses on precise matching at the low-level pixel level and is an important component in ensuring the quality of the generated image, while the perceptual loss function considers the high-level perceptual features of the image, making the generated image more consistent with human visual perception. In research on deep learning-based second image signal processing, setting both to a weight of 1 is a common practice. This weight setting comprehensively considers information at different levels, ensuring a good balance between low-level and high-level features in the generated image.

[0098] The structural loss function further improves structural integrity. The overall structure of an image involves the relationships between different elements, such as the relative positions of objects and the coherence of textures. In this embodiment, while maintaining pixel-level and perceptual similarity, a structural loss function is introduced into the training process to further promote the generated image to better maintain the structural similarity with the displayed sample image. During training, it was found that the value of the structural loss function was relatively large (on the order of 1e-1). To avoid overemphasizing the structural loss function during training and neglecting the pixel loss function and the perceptual loss function (both on the order of 1e-2), the weight of the structural loss function was adjusted to 0.15, making its value consistent with the other two. This helps ensure that the model takes into account detail accuracy, human visual perception similarity, and overall structural similarity when generating images.

[0099] Step 305: Adjust the initial values ​​of the parameters in the initial model of image signal processing according to the first loss function. The purpose of the adjustment is to minimize the first loss function.

[0100] Step 306: Determine whether the adjustment of the parameter values ​​in the initial image signal processing model meets the preset stopping condition. If it does, use the parameter values ​​of the initial signal processing model adjusted in step 305 as the parameter values ​​in the trained image signal processing model. If it does not meet the condition, return to step 302 for the initial image signal processing model after adjusting the parameter values, that is, change a batch of training samples and adjust the parameter values ​​in the initial image signal processing model according to the changed training samples.

[0101] It should be noted that the image signal processing model can be trained through steps 301 to 306 above. The image signal processing model can be tested on the test set. The calculated peak signal-to-noise ratio (PSNR), structural similarity coefficient (SSIM), and UV value are 23.49, 0.8865, and 0.0125, respectively. That is, there is still a color error between the display format image obtained by the trained image signal processing model and the display sample image.

[0102] In a specific embodiment, the trained image signal processing model needs further fine-tuning. Specifically, the image signal processing model can be fine-tuned according to steps 302 to 306 above. In this process, a second loss function can be introduced, specifically a chroma channel loss function. That is, when fine-tuning the parameter values ​​in the image signal processing model according to the first loss function in step 305 above, an overall loss function can be calculated based on the first and second loss functions according to the following formula 13, and the parameter values ​​in the trained image signal processing model can be adjusted according to the calculated overall loss function:

[0103] Loss finetrain =Loss pretrain +α×loss UV (13)

[0104] Here, α represents the weight value of the chroma channel loss function. Since the chroma channel loss function, pixel loss function, and perceptual loss function are of the same order of magnitude, in order to allow the image signal processing model to focus more on color restoration and reducing color errors during the fine-tuning stage, a relatively large weight value is assigned to the chroma loss function in this embodiment. The results on the test set when the weight values ​​of the chroma channel loss function are set to 1, 4, 10, 15, and 20 are shown in Table 1 below. It is found that when the weight value is 10, it can not only further reduce UV color errors, but also further improve PSNR and SSIM values.

[0105] Weights of the chroma channel loss function PSNR SSIM UV 0 23.49 0.8865 0.0125 1 23.26 0.8885 0.0119 5 23.31 0.8897 0.0122 10 24.23 0.8971 0.0113 15 24.12 0.891 0.0118 20 23.6 0.88 0.012

[0106] Table 1

[0107] In another specific embodiment, after training the image signal processing model through steps 301 to 306 above, in order to further improve the image signal processing model to obtain a more realistic display format image and have a certain stylization effect, this embodiment further introduces an adversarial loss function, Loss, during the fine-tuning stage. GAN Since the training of GAN networks is unstable, setting a small weight value for the adversarial loss function is a reasonable choice, especially during the fine-tuning stage. This is because the main goal during fine-tuning is to maintain the accurate reproduction and realism of the image, rather than overemphasizing the realism generated by the GAN network.

[0108] In this case, when further fine-tuning the trained image signal processing model, specifically, the model can be fine-tuned according to steps 302 to 306 above. During this process, a second loss function, specifically an adversarial loss function, can be introduced. That is, when fine-tuning the parameter values ​​in the image signal processing model according to the first loss function in step 305 above, a loss function can be calculated according to formula 14 below, and the parameter values ​​in the trained image signal processing model can be adjusted based on the calculated loss function.

[0109] Loss finetrain =Loss pretrain +α×loss UV +β×loss GAN (14)

[0110] During the fine-tuning phase, it was found that the magnitude of the adversarial loss function was 1e-1. To make its magnitude smaller than that of other loss functions, in this embodiment, the weight values ​​of the adversarial loss function were set to 0.0001, 0.001, 0.01, and 0.1, respectively. The PSNR, SSIM, and UV values ​​calculated on the test set are shown in Table 2 below. It was found that when the weight value was set to 0.0001, the adversarial loss function had a smaller impact on the image signal processing model (i.e., the data in the second row was close to that in the first row). Setting the weight value to 0.001 not only further reduced the UV color error but also further improved the PSNR and SSIM values.

[0111]

[0112] Table 2

[0113] It should be noted that during the training process of the image signal processing model, the original captured sample images can first undergo initial image signal processing using traditional methods. This provides a more accurate reference for the training of the image signal processing model, enabling it to better adapt to the characteristics and hardware differences of different cameras and improve its generalization ability across different cameras. After the image signal processing model is trained using the methods described in steps 301 to 306 above, its operating logic is pre-programmed into the system, as follows: Figure 8 As shown, image signal processing can be performed using the following method:

[0114] Step 401: Obtain the original image captured by the shooting device in the original shooting format, specifically a Bayer format image.

[0115] Step 402: Perform black level correction and normalization on the original captured image to obtain a normalized image. Specifically:

[0116] Different shooting devices may have significantly different sensor and hardware configurations, resulting in different black and white level values. In this embodiment, traditional black level correction and normalization processing is used. Specifically, the original shooting format image is first processed... unit16 The data type is converted from uint16 to float32 to prevent overflow during numerical calculations. Then, the black level value is subtracted from each pixel of the converted image, and values ​​less than 0 are truncated to 0 (BLC process). Finally, the corrected image value is divided by the difference between the white level (white_level) and the black level (black_level) to normalize the image's dynamic range to [0,1]. This results in a normalized image, which facilitates subsequent numerical calculations and ensures the image maintains appropriate contrast and detail under different lighting conditions. Specifically, this can be represented by the following formula 15: The above process is represented by the following formula:

[0117]

[0118] Step 403: Perform de-mosaic processing on the normalized image to obtain the de-mosaiced image. Specifically:

[0119] The original image format, Bayer format, is a color filter array image that supports different format arrangements. Each pixel contains only red, green, and blue (R, G, B) monochromatic light, not complete RGB information. Therefore, a demosaicing algorithm is needed to convert the single-channel image into a complete RGB color image, resulting in the demosaiced image. Demosaicing primarily involves interpolating and reconstructing the original image into a complete RGB image. Taking bilinear interpolation as an example, it interpolates in both the horizontal and vertical directions, estimating the missing color channels by weighted averaging of surrounding pixels. Specifically, interpolation can be performed in the horizontal and vertical directions using the following formula (16) to estimate the G channel pixels:

[0120]

[0121] As shown in Equation 17 below, interpolation is performed in the diagonal direction to estimate the R and B channel pixels:

[0122]

[0123] As shown in Formula 18 below, the complete RGB image is obtained using the interpolated G, R, and B channel values:

[0124]

[0125] Since the Bayer format is not fixed and there are many different formats, different cameras will choose different Bayer formats. The demosaicing process for each Bayer format involves complex image features and non-linear relationships. Traditional demosaicing has unique advantages because it can directly restore the complete RGB information through interpolation, supports different Bayer formats, has good anti-aliasing effect, can preserve image details and edges, and accurately restore color information.

[0126] Step 404: Perform white balance processing on the de-mosaiced image to obtain the white balance-processed image. Specifically:

[0127] Cameras typically use red gain (R). Gain ), Green gain (G) Gain Blue gain (B) GainThese three parameters are used to adjust the white balance, ensuring that white points in the image appear neutral white under various lighting conditions. Specifically, the original RGB value of each pixel is multiplied by the corresponding color gain to achieve the white balance adjustment. Since different camera systems use different white balance algorithms, they have different color gain parameters. In this embodiment, the color gain parameters are directly read from the camera's internal database. For the de-mosaiced image, the R, G, and B values ​​of each pixel in the de-mosaiced image are multiplied by the corresponding color gain to obtain the white balance-adjusted image. wb .

[0128] Step 405: Perform color correction on the white balance processed image to obtain the corrected image, which is the image after the initial processing described above. Specifically:

[0129] Camera color spaces are designed and implemented by camera manufacturers to represent and process color information captured by the camera. Different camera manufacturers may use different sensors, image processing workflows, and color models, resulting in different camera color spaces. To ensure color consistency across different hardware devices, color correction is needed to map the image from the camera color space to a standard color space. The camera's color correction matrix is ​​obtained by the camera manufacturer through calibration and adjustment. By multiplying the original color vector of the image by the color correction matrix, it is mapped from the camera color space to a standard color space, ensuring that the image displays standard colors on different devices.

[0130] Since different cameras have different color correction matrices, in this embodiment, the white balance processed image is obtained by directly reading the camera's color correction matrix. wb Multiplying by the color correction matrix M yields the corrected image. corrected Specifically, as shown in Formula 19:

[0131]

[0132] Step 406: The corrected image is processed again according to the preset image signal processing model to obtain the display format image.

[0133] This invention also provides a training system for image signal processing, the schematic diagram of which is shown below. Figure 9 As shown, it can specifically include:

[0134] Model determination unit 10 is used to determine the initial model for image signal processing.

[0135] Specifically, the model determination unit 10 is specifically used to determine the initial model of the image signal processing, which includes: a multi-layer convolutional network for image segmentation, wherein each layer of the image segmentation convolutional network includes a wavelet transform downsampling layer, a wavelet transform upsampling layer, a convolution and activation layer, and a residual module; wherein any layer of the multi-layer convolutional network for image segmentation is nested in another layer of the image segmentation convolutional network.

[0136] The sample determination unit 11 is used to determine training samples, which include multiple sets of sample images, each set of sample images including the original captured sample image and its corresponding display sample image.

[0137] The sample processing unit 12 is used to perform a first image signal processing on the original captured sample images determined by each of the sample determination units 11 to obtain a first-processed image, and then perform a second image signal processing on the first-processed image through the image signal processing initial model determined by the model determination unit 10 to obtain a corresponding display format image; the first image signal processing is a processing related to non-shared parameters of the shooting device, and the second image signal processing is other image signal processing besides the first image signal processing.

[0138] The adjustment unit 13 is used to adjust the initial image signal processing model according to the display format image obtained from the initial image signal processing model in the sample processing unit 12 and the display sample image in the training samples, so as to obtain the image signal processing model.

[0139] The adjustment unit 13 is specifically used to calculate a pixel loss function, which includes: the average absolute difference between the display format image obtained by the image signal processing initial model and the corresponding display sample image in the training samples; calculate a perceptual loss function, which includes: the mean square value of the difference between the high-order features of the display format image obtained by the image signal processing initial model and the high-order features of the corresponding display sample image in the training samples; calculate a structural loss function, which includes: the structural similarity index between the display format image and the corresponding display sample image in the training samples obtained by the image signal processing initial model; calculate a first loss function based on the pixel loss function, the perceptual loss function, and the structural loss function; and adjust the parameter values ​​in the image signal processing initial model based on the first loss function.

[0140] When adjusting the parameter values ​​in the initial image signal processing model according to the first loss function, the adjustment unit 13 is specifically used to calculate the chroma channel loss function, which indicates the difference between the chroma channels of the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; calculate the adversarial loss function, which includes the probability that the display format image obtained by the initial image signal processing model is identified as the corresponding display sample image by the discriminator; calculate the second loss function according to the chroma channel loss function and / or the adversarial loss function; and fine-tune the parameter values ​​in the initial image signal processing model according to the second loss function to obtain the image signal processing model.

[0141] Specifically, when calculating the adversarial loss function, the adjustment unit 13 is used to determine a discriminator, using the image signal processing initial model as a generator. The discriminator is used to determine whether the display sample image and the display format image obtained by the image signal processing initial model are display sample images.

[0142] The adversarial loss function of the generator is calculated based on the discrimination result of the discriminator. When fine-tuning the parameter values ​​in the initial image signal processing model based on the second loss function, specifically, another overall loss function is calculated based on the first and second loss functions, and the parameter values ​​in the initial image signal processing model are fine-tuned based on the overall loss function.

[0143] Furthermore, the image signal processing training system in this embodiment may also include:

[0144] Image processing unit 14 is used to acquire an original captured image; perform first image signal processing on the original captured image to obtain a first-processed image; the first image signal processing is related to non-shared parameters of the capturing device; and process the first-processed image again according to a preset image signal processing model to obtain a display format image. The image signal processing model is used to perform second image signal processing on the first-processed image, and the second image signal processing is other image signal processing besides the first image signal processing. The preset image signal processing model is the image signal processing model trained by the adjustment unit 13 mentioned above.

[0145] In the training process of the image signal processing model in this embodiment, the ISP processing of the original captured sample images needs to be divided. The first image signal processing, which is related to the non-shared parameters of the capturing device, is performed using traditional methods. The second image signal processing is performed using the initial image signal processing model. When the image signal processing model trained in this way is actually applied to the field of image processing, the quality of the display format images obtained by the method of this embodiment can be greatly improved compared with that obtained by traditional ISP processing. In particular, it performs well in terms of peak signal-to-noise ratio, structural similarity index, and chroma channels.

[0146] This invention also provides a server, the structural diagram of which is shown below. Figure 10 As shown, the server can vary considerably depending on its configuration or performance, and may include one or more central processing units (CPUs) 20 (e.g., one or more processors) and memory 21, and one or more storage media 22 (e.g., one or more mass storage devices) for storing application programs 221 or data 222. The memory 21 and storage media 22 can be temporary or persistent storage. The program stored in the storage media 22 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 20 may be configured to communicate with the storage media 22 and execute the series of instruction operations stored in the storage media 22 on the server.

[0147] Specifically, the application program 221 stored in the storage medium 22 includes an application program for training image signal processing, and this program may include the model determination unit 10, sample determination unit 11, sample processing unit 12, adjustment unit 13, and image processing unit 14 in the image signal processing training system described above, which will not be elaborated here. Furthermore, the central processing unit 20 may be configured to communicate with the storage medium 22 and execute a series of operations corresponding to the image signal processing training application program stored in the storage medium 22 on a server.

[0148] The server may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input / output interfaces 25, and / or one or more operating systems 223, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0149] The steps performed by the image signal processing training system in the above method embodiments can be based on this. Figure 10 The server structure is shown.

[0150] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of computer programs adapted to be loaded by a processor and executed by an image signal processing method performed by the training system for image signal processing described above.

[0151] This invention also provides a server, including a processor and a memory; the memory is used to store a plurality of computer programs, the computer programs being loaded by the processor and executed as in the image signal processing method performed by the image signal processing training system described above; the processor is used to implement each of the plurality of computer programs.

[0152] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0153] The foregoing has provided a detailed description of an image signal processing training method, system, storage medium, and server provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A training method of image signal processing, characterized by, include: Determine the initial model for image signal processing; The training samples are determined, which include multiple sets of sample images, each set of sample images including the original captured sample image and its corresponding display sample image; Each of the original captured sample images is subjected to a first image signal processing to obtain a first-processed image. Then, the first-processed image is subjected to a second image signal processing through the image signal processing initial model to obtain a corresponding display format image. The first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing. Based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, the initial image signal processing model is adjusted to obtain the image signal processing model. The step of adjusting the initial image signal processing model based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples specifically includes: Calculate the pixel loss function, which includes: the average of the absolute differences between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; wherein is a display format image obtained by an initial model of image signal processing, is a corresponding display sample image in the training sample, is a pixel loss function; Calculate the perceptual loss function, which includes the mean square value of the difference between the high-order features of the display format image obtained by the initial image signal processing model and the high-order features of the corresponding display sample image in the training samples. ; wherein X and Y are high-order features of the display format image obtained from the initial image signal processing model and the corresponding display sample image in the training sample, respectively; is a perceptual loss function; Calculate the structural loss function, which includes: the structural similarity index between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; ; ; , , , Let represent the means of the two images respectively. Let represent the variances of the two images respectively. This represents the covariance of two images. The stability coefficient, Let the default parameters be 0.01 and 0.03, respectively, and B be the image bit depth. Then the structural loss function... This includes the display format image obtained from the initial model of image signal processing. Display sample images corresponding to the training samples SSIM between; Calculate a first loss function based on the pixel loss function, the perceptual loss function, and the structural loss function, and adjust the parameter values ​​in the initial model of the image signal processing based on the first loss function; Calculate the chroma channel loss function, which is used to indicate the difference in chroma channels between the display format image obtained by the initial image signal processing model and the corresponding display sample image in the training samples; ； ； Using a mean of 0 and a variance Gaussian blur operator of 20 The display format image obtained from the initial model of image signal processing and the display sample image from the training samples are blurred respectively to obtain the blurred image; then, a linear transformation is used to map the blurred image from RGB to YUV color space to obtain... and Finally, the loss function between the UV channels of the display format image obtained from the initial image signal processing model is calculated. ; Calculate the adversarial loss function, which includes the probability that the display format image obtained from the initial model of the image signal is identified by the discriminator as the corresponding display sample image; ； ； The display format image obtained from the initial model of image signal processing. The input is processed by the discriminator, and the output is the probability of determining whether the sample image is displayed. The generator's adversarial loss function is calculated using this probability. A real value of 1 indicates that the discriminator is highly confident in the display format of the image. It displays sample images; Calculate the second loss function based on the chroma channel loss function and the adversarial loss function; Calculate another overall loss function based on the first loss function and the second loss function, and fine-tune the parameter values ​​in the initial model of the image signal processing based on the overall loss function to obtain the image signal processing model; ； Weights of the chroma channel loss function The weight value of the adversarial loss function is 10. It is 0.

001.

2. The method as described in claim 1, characterized in that, Determine the initial model for image signal processing, specifically including: The initial model for image signal processing is determined to include: a multi-layer convolutional network for image segmentation, wherein each layer of the convolutional network for image segmentation includes a wavelet transform downsampling layer, a wavelet transform upsampling layer, a convolution and activation layer, and a residual module; In this multi-layer image segmentation convolutional network, any layer of the image segmentation convolutional network is nested within another layer of the image segmentation convolutional network.

3. The method as described in claim 1, characterized in that, The calculation of the adversarial loss function specifically includes: A discriminator is determined, and the initial image signal processing model is used as a generator. The discriminator is used to determine whether the display sample image and the display format image obtained by the initial image signal processing model are display sample images. The adversarial loss function of the generator is calculated based on the discrimination result of the discriminator.

4. The method as described in claim 1 or 2, characterized in that, The method further includes: Obtain the original captured image format; The original captured image is subjected to a first image signal processing to obtain a first-processed image; the first image signal processing is related to non-shared parameters of the capturing device. The image after the first processing is processed again according to a preset image signal processing model to obtain a display format image. The image signal processing model is used to perform a second image signal processing on the image after the first processing. The second image signal processing is other image signal processing besides the first image signal processing.

5. A training system for image signal processing using any one of the training methods of claims 1-4, characterized in that, include: The model determination unit is used to determine the initial model for image signal processing; A sample determination unit is used to determine training samples, wherein the training samples include multiple sets of sample images, and each set of sample images includes an original captured sample image and its corresponding display sample image; The sample processing unit is used to perform a first image signal processing on each of the original captured sample images to obtain a first-processed image, and then perform a second image signal processing on the first-processed image through the image signal processing initial model to obtain a corresponding display format image; the first image signal processing is a processing related to non-shared parameters of the capturing device, and the second image signal processing is other image signal processing besides the first image signal processing; The adjustment unit is used to adjust the initial image signal processing model based on the display format image obtained from the initial image signal processing model and the display sample images in the training samples, so as to obtain the image signal processing model.

6. The system as described in claim 5, characterized in that, The system also includes: An image processing unit is configured to acquire an original captured image; perform a first image signal processing on the original captured image to obtain a first-processed image; the first image signal processing is a processing related to non-shared parameters of the capturing device; and process the first-processed image again according to a preset image signal processing model to obtain a display format image, wherein the image signal processing model is configured to perform a second image signal processing on the first-processed image, and the second image signal processing is other image signal processing besides the first image signal processing.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer programs adapted to be loaded by a processor and executed by the training method for image signal processing as described in any one of claims 1 to 4.

Images