AI-ISP generalization method based on unsupervised camera color space mapping
Through unsupervised camera color space mapping and loop adversarial generation network, the problem of AI-ISP cross-camera generalization between different cameras is solved, and efficient image processing is achieved under complex lighting conditions, avoiding data set reacquisition and model reconciliation train.
Patent Information
- Application Number
- CN202411097327.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-06
AI Technical Summary
The existing AI-ISP method cannot achieve cross-camera generalization of the entire end-to-end model, and cannot be applied to complex lighting scenarios, requires camera-related metadata and is difficult to obtain.
The unsupervised camera color space mapping method is adopted to build a camera color space block mapping network through a loop-to-coordinated generation network to realize image mapping between different cameras, avoiding the retraining of AI-ISP and the re-acquisition of data sets.
The AI-ISP is realized across cameras on different cameras, and can maintain performance in complex lighting scenarios without the need for pairing data or other prior information.
Smart Images

Figure CN119090742A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to an AI-ISP generalization method based on unsupervised camera color space mapping. Background Art
[0002] Image signal processor (ISP) refers to a series of image processing modules that convert the low-quality original RAW images captured by the camera into high-quality sRGB images. Traditional ISP systems include white balance modules, denoising modules, demosaicing modules, etc. In recent years, with the rapid development of deep learning technology, artificial intelligence-based image processors (AI-ISP) have become a more effective way to process raw RAW images. Current AI-ISP methods generally use end-to-end deep neural networks to replace all ISP modules and use carefully collected pairs of raw RAW images and sRGB images for training.
[0003] However, due to the different spectral sensitivities of camera filters, raw images captured by different cameras exist in different color spaces, which causes AI-ISP to perform poorly when deployed across different cameras. Generally speaking, each AI-ISP can only be deployed on one camera after training, and each camera needs to re-shoot the data set for training the corresponding AI-ISP. The process of re-shooting the paired data set and retraining the AI-ISP is very cumbersome. At present, although there are some generalization methods for white balance modules and denoising modules, such as the C5 (Cross Camera Convolutional Color Constancy) model proposed by Mahmoud Afifi et al., which can be used for cross-camera deployment of white balance modules, and the physical guided low-light RAW denoising algorithm proposed by Feng et al., which can be used for cross-camera deployment of denoising modules, there is currently no generalization method for the entire AI-ISP model.
[0004] The existing AI-ISP generalization methods have the following disadvantages: 1. They are not applicable to the generalization of the entire end-to-end AI-ISP and can only complete the incomplete generalization of some modules. 2. They cannot be applied to complex lighting scenes and are only applicable to simple, certain lighting conditions. 3. They require camera-related metadata, which is difficult to obtain. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an AI-ISP generalization method based on unsupervised camera color space mapping, which maps RAW images taken under complex and unknown lighting conditions to a specific camera color space and then processes them by AI-ISP, thereby avoiding the retraining of AI-ISP and the re-collection of data sets.
[0006] To achieve the above-mentioned object of the invention, the present invention provides an AI-ISP generalization method based on unsupervised camera color space mapping, characterized in that it includes the following steps:
[0007] (1) Image acquisition and preprocessing;
[0008] (1.1) Set the image acquisition format of camera A and camera B to RAW format, and arrange the acquired images in RGGB array;
[0009] (1.2) Under the same scene conditions, use camera A and camera B to collect several images respectively, where the images collected by camera A are saved in image set A, and the images collected by camera B are saved in image set B;
[0010] (1.3) For any image set, normalize each image in the image set first, then perform channel conversion so that each image is converted from the original one-channel to four-channel, and finally crop the channel-converted image into a block of size H*W;
[0011] Finally, each image in the image set A is processed and stored in the tile set Set(A); each image in the image set B is processed and stored in the tile set Set(B);
[0012] (2) Build a recurrent adversarial generative network;
[0013] The cyclic adversarial generation network consists of two generators and two discriminators, where both generators use the camera color space block mapping network, denoted by G and a→b and G b→a , G a→b It is used to map the image in the color space of camera A to the color space of camera B. Another network G b→a Used to map the image in the color space of camera B to the color space of camera A;
[0014] The camera color space block mapping network consists of a feature extraction network and a parameter matrix prediction network. The feature extraction network has two branches, a convolution branch and a Transformer branch, and performs feature fusion of the two branches through deformation and convolution operations. Specifically, the convolution branch consists of 8 convolution layers, and the Transformer branch consists of 4 Transformer layers. Every 2 convolution layers and 1 Transformer layer form a small block. For the input small block image I, it will first be sent to the first convolution layer and the Transformer layer at the same time. After the first convolution layer processes the image, the image is directly sent to the second convolution layer. The Transformer layer will divide the image into blocks, downsample it, and perform embedding operations to convert it into several visual vectors, and then perform self-attention operations and Feed Forward operation, then the visual vector will be reshaped and upsampled back to the same shape as the output of the second layer of convolution, and directly added, and used as the input of the next block after 2 times downsampling; the last small block will not be reshaped and upsampled, but directly used as the output of the feature extraction network, and the output shape is I'; the number of output channels of each convolution layer is 16, 32, 32, 64, 64, 128, 128, 256, using LeakeyRelu as the activation function; the embedding layer dimensions of the Transformer layer are 32, 64, 128, 256, using Gelu as the activation function;
[0015] The parameter matrix prediction network consists of two fully connected layers, with the output I' of the feature extraction network as input. The output dimensions of the fully connected layer are 128 and 56 respectively, and the output of the fully connected layer is the color space mapping matrix CCM; then, the color space mapping matrix will be interpolated and upsampled to obtain CCM', and then rearranged into shape CCM"; at the same time, the original input image I is expanded from the original four channels to fourteen channels, and arranged and combined into image I poly Finally, I poly Perform element-by-element matrix multiplication with "CCM" to obtain the output of the entire network
[0016] The two discriminators are denoted as D a , D b , each discriminator consists of a five-layer convolutional neural network and an average pooling layer; each layer of the five-layer convolutional neural network uses LeakyReLU as the activation function, the size of each convolution kernel is 4*4, the number of channels is 64, 128, 256, 512, and 1 respectively, and the average pooling layer is connected after the last layer of convolution to pool the feature map into the discrimination probability;
[0017] (3) Training cyclic adversarial generative network;
[0018] (3.1) Randomly sample a tile from Set(A), denoted as I A ; Randomly sample a tile from Set(B), denoted as I B ;
[0019] (3.2), image I A Input camera color space block mapping network G a→b , and get the output image I AB ; Similarly, image I B Input camera color space block mapping network G b→a , get image I BA ;
[0020] Then image I B Input camera color space block mapping network G a→b , get image I BB , image I A Input camera color space block mapping network G b→a , get image I AA ; Then calculate the consistency loss Loss iden ;
[0021] Loss iden =MSE(I BB ,I B )+MSE(I AA ,I A )
[0022] Among them, MSE() means calculating the mean square error of each pixel in the two images;
[0023] (3.3), image I AB Input Discriminator D a , and the discriminant probability is obtained Image I BA Input Discriminator D b , and the discriminant probability is obtained Then calculate the adversarial loss Loss of the generator gen-gan ;
[0024]
[0025] (3.4), image I AB Input camera color space block mapping network G b→a , get image I ABA , image I BA Input camera color space block mapping network G a→b , get image I BAB , and then calculate the cycle consistency loss Loss cycle ;
[0026] Loss cycle =MSE(I ABA ,I A )+MSE(I BAB ,I B )
[0027] (3.5) Calculate the total loss Loss of the generator in the cyclic adversarial generation network gen ;
[0028] Loss gen =Loss iden +Loss gen-gan +Loss cycle
[0029] (3.6), image I A Input Discriminator D b , and the discriminant probability is obtained Image I B Input Discriminator D a , and the discriminant probability is obtained Then calculate the total loss Loss of the discriminator in the cyclic adversarial generation network dis ;
[0030]
[0031] (3.7), according to the total loss Loss of the generator gen , using the gradient descent method to update the camera color space block mapping network G a→b , G b→a Parameters; according to the total loss Loss of the discriminator dis , update the discriminator D using the gradient descent method a , D b Parameters;
[0032] (3.8) After one round of training is completed, repeat steps (3.1)-(3.7) until the network converges, and obtain the cyclic adversarial generation network after training;
[0033] (4) Generalizable ISP deployment;
[0034] (4.1) Use the tile set Set(A) to train the AI-ISP network, and the trained network is denoted as ISP A ;
[0035] (4.2) Extract the camera color space block mapping network G from the trained cyclic adversarial generation network b→a , then the network ISP A Series in G b→aThe output terminal;
[0036] Prepare an AI-ISP network trained in any way on the A camera dataset, denoted as ISP A , and the camera color space block mapping network G b→a ;
[0037] (4.3) The image I captured by camera B B Input to the camera color space block mapping network G b→a Thus, the output image I BA , and then image I BA Input to network ISP A , and obtain a good quality RGB image F A .
[0038] The object of the invention of the present invention is achieved in this way:
[0039] The present invention is based on the AI-ISP generalization method of unsupervised camera color space mapping. First, images acquired by different cameras are preprocessed to construct a training data set. Then, an unsupervised training framework based on a recurrent adversarial generative network is built to train the color space block mapping network proposed in the present invention through the training data set. Finally, the same AI-ISP is deployed on different cameras by connecting the trained color space block mapping network and the AI-ISP network in series, thereby realizing the cross-camera generalization of AI-ISP.
[0040] At the same time, the AI-ISP generalization method based on unsupervised camera color space mapping of the present invention also has the following features:
[0041] Beneficial effects:
[0042] (1) The present invention uses RAW images acquired by two different cameras (without pairing) to train a camera color space block mapping network without any other prior information, and uses the network to assist the generalization of the end-to-end AI-ISP model;
[0043] (2) Since the images acquired by different cameras have significant domain differences due to different scenes, lighting and other factors, such as different noise levels, different blur levels, different scene styles, etc., in comparison, the domain differences caused by different camera spectral sensitivities are masked by the above domain differences and are difficult to be captured by the model; the present invention designs a camera color space block mapping network, which can accurately capture the domain differences caused by different camera spectral sensitivities and ignore the influence of other factors;
[0044] (3) The camera color space block mapping network of the present invention benefits from its block prediction strategy and can work in scenes with non-uniform lighting, so that the generalized AI-ISP can maintain its performance in complex lighting scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is an image processor based on artificial intelligence;
[0046] Figure 2 It is a flow chart of the AI-ISP generalization method based on unsupervised camera color space mapping of the present invention;
[0047] Figure 3 It is a schematic diagram of the camera color space block mapping network structure;
[0048] Figure 4 This is the cross-camera generalization flow chart of AI-ISP;
[0049] Figure 5 This is a comparison chart of the visualization results of different algorithms. DETAILED DESCRIPTION
[0050] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.
[0051] Example
[0052] Figure 1 It is a flow chart of the AI-ISP generalization method based on unsupervised camera color space mapping of the present invention.
[0053] In this embodiment, if Figure 1 As shown, the present invention provides an AI-ISP generalization method based on unsupervised camera color space mapping, comprising the following steps:
[0054] S1, image acquisition and preprocessing;
[0055] S1.1. Set the image acquisition format of camera A and camera B to RAW format, and arrange the acquired images in RGGB array form (i.e., the image is one channel, the upper left corner of a 2*2 small block is the red pixel value obtained by the camera, the upper right corner and the lower left corner are green pixel values, and the lower right corner is the blue pixel value);
[0056] S1.2. Under the same scene conditions, use camera A and camera B to collect 200 images respectively, wherein the images collected by camera A are saved in image set A, and the images collected by camera B are saved in image set B;
[0057] S1.3, for any image set, normalize each image in the image set first, then perform channel conversion, so that each image is converted from the original one channel to four channels, and finally crop the channel-converted image into a block of size H*W=256×256;
[0058] Finally, each image in the image set A is processed and stored in the tile set Set(A); each image in the image set B is processed and stored in the tile set Set(B);
[0059] S2. Build a cyclic adversarial generation network;
[0060] The cyclic adversarial generation network consists of two generators and two discriminators, where both generators use the camera color space block mapping network, denoted by G and a→b and G b→a , G a→b It is used to map the image in the color space of camera A to the color space of camera B. Another network G b→a Used to map the image in the color space of camera B to the color space of camera A;
[0061] In this embodiment, the images acquired by different cameras have significant domain differences due to different factors such as scenes and lighting, such as different noise levels, different blur levels, and different scene styles. In contrast, the domain differences caused by different camera spectral sensitivities are masked by the above domain differences and are difficult to be captured by the model. This patent innovatively designs a camera color space block mapping network, which can accurately capture the domain differences caused by different camera spectral sensitivities while ignoring the impact of other factors. At the same time, in order to give full play to the advantages of deep neural networks, the network predicts the color space mapping parameters of each block of the image in a block-by-block manner to adapt to the problem that different color space change operations are required for each position of the image in complex lighting scenes.
[0062] like Figure 3 As shown in Fig. 1, the camera color space block mapping network consists of a feature extraction network and a parameter matrix prediction network. The feature extraction network has two branches, a convolution branch and a Transformer branch, and performs feature fusion of the two branches through deformation and convolution operations. Specifically, the convolution branch consists of 8 convolution layers, and the Transformer branch consists of 4 Transformer layers. Every 2 convolution layers and 1 Transformer layer form a small block. For the input small block image I∈R H×W×4, it will first be sent to the first convolution layer and the Transformer layer at the same time. After the first convolution layer processes the image, the image will be directly sent to the second convolution layer. The Transformer layer will divide the image into blocks, downsample and embed it into several visual vectors, and then perform self-attention and Feed Forward operations. Then the visual vector will be reshaped and upsampled back to the same shape as the output of the second convolution layer, and directly added. After 2 times downsampling, it will be used as the input of the next block; the last small block will not be reshaped or upsampled, but will be directly used as the output of the feature extraction network. The output shape is PS represents the pre-set hyperparameters, the size of which is 16; the number of output channels of each convolutional layer is 16, 32, 32, 64, 64, 128, 128, 256, and LeakeyRelu is used as the activation function; the embedding layer dimensions of the Transformer layer are 32, 64, 128, 256, and Gelu is used as the activation function;
[0063] The parameter matrix prediction network consists of two fully connected layers, which are composed of the output of the feature extraction network As input, the output dimensions of the fully connected layer are 128 and 56 respectively, and the output of the fully connected layer is the color space mapping matrix Subsequently, the color space mapping matrix will be interpolated and upsampled to obtain CCM'∈R HW×56 , and then rearranged into the shape CCM”∈R H×W×14×4 ; At the same time, the original input image I∈R H×W×4 The channel level will be arranged and combined, and the original four channels [R, Gr, Gb, G] will be expanded to [R, Gr, Gb, G, R 2 ,Gr 2 ,Gb 2 ,G 2 ,RGr,RGb,RB,GrGb,GrB,GbB] a total of fourteen channels composed of image I poly ∈R H×W×14 Finally, I poly Perform element-by-element matrix multiplication with "CCM" to obtain the output of the entire network
[0064] The two discriminators are denoted as D a , D b , each discriminator consists of a five-layer convolutional neural network and an average pooling layer; each layer of the five-layer convolutional neural network uses LeakyReLU as the activation function, the size of each convolution kernel is 4*4, the number of channels is 64, 128, 256, 512, and 1 respectively, and the average pooling layer is connected after the last layer of convolution to pool the feature map into the discrimination probability;
[0065] S3, training cycle adversarial generation network;
[0066] S3.1. Randomly sample a tile from Set(A), denoted as I A ; Randomly sample a tile from Set(B), denoted as I B ;
[0067] S3.2, image I A Input camera color space block mapping network G a→b , and get the output image I AB ; Similarly, image I B Input camera color space block mapping network G b→a , get image I BA ;
[0068] Then image I B Input camera color space block mapping network G a→b , get image I BB , image I A Input camera color space block mapping network G b→a , get image I AA ; Then calculate the consistency loss Loss iden ;
[0069] Loss iden =MSE(I BB ,I B )+MSE(I AA ,I A )
[0070] Among them, MSE() means calculating the mean square error of each pixel in the two images;
[0071] S3.3, image I AB Input Discriminator D a , and the discriminant probability is obtained Image I BA Input Discriminator D b , and the discriminant probability is obtained Then calculate the adversarial loss Loss of the generator gen-gan ;
[0072]
[0073] S3.4, image I AB Input camera color space block mapping network G b→a , get image I ABA , image I BA Input camera color space block mapping network G a→b , get image IBAB , and then calculate the cycle consistency loss Loss cycle ;
[0074] Loss cycle =MSE(I ABA ,I A )+MSE(I BAB ,I B )
[0075] S3.5. Calculate the total loss of the generator in the cyclic adversarial generation network gen ;
[0076] Loss gen =Loss iden +Loss gen-gan +Loss cycle
[0077] S3.6, image I A Input Discriminator D b , and the discriminant probability is obtained Image I B Input Discriminator D a , and the discriminant probability is obtained Then calculate the total loss Loss of the discriminator in the cyclic adversarial generation network dis ;
[0078]
[0079] S3.7. According to the total loss Loss of the generator gen , using the gradient descent method to update the camera color space block mapping network G a→b , G b→a Parameters; according to the total loss Loss of the discriminator dis , update the discriminator D using the gradient descent method a , D b Parameters;
[0080] S3.8. After one round of training is completed, repeat steps S3.1-S3.7 until the network converges to obtain a trained cyclic adversarial generation network;
[0081] S4, generalizable ISP deployment;
[0082] S4.1. The cross-camera generalization process of AI-ISP is as follows Figure 4 As shown, we first use the tile set Set(A) to train the AI-ISP network, and the trained network is recorded as ISP A ;
[0083] In this embodiment, the structure of the AI-ISP network is as follows: Figure 1 As shown in the figure, the AI-ISP network has a traditional structure, and the specific training process will not be described here;
[0084] S4.2. Extract the camera color space block mapping network G from the trained cyclic adversarial generation network b→a , then the network ISP A Series in G b→a The output terminal;
[0085] Prepare an AI-ISP network trained in any way on the A camera dataset, denoted as ISP A , and the camera color space block mapping network G b→a ;
[0086] S4.3, the image I captured by camera B B Input to the camera color space block mapping network G b→a Thus, the output image I BA , and then image I BA Input to network ISP A , and obtain a good quality RGB image F A .
[0087] Simulation Verification
[0088] The present invention was trained and tested on the color space mapping dataset proposed by Mahmoud Afifi et al., where the AI-ISP model was supervised trained on the RAW images taken by iPhone-X and the corresponding RGB images, with a training amount of 200 rounds, a learning rate of 2e-4, and optimization using the Adam optimizer. The camera color space block mapping network was unsupervisedly trained on the RAW images taken by iPhone-X and Samsung-S9, with a training amount of 35 rounds, a generator learning rate of 1e-5, a discriminator learning rate of 2e-5, and optimization using the Adam optimizer. Comparison method The uvcgan algorithm and Afifi's algorithm were trained under the same settings. The quantitative comparison results are shown in Table 1. The Fréchet factor distance (FID) indicator and the Kulbeck-Leibler divergence (KL divergence) indicator are used for evaluation. The method proposed in the present invention is significantly better than the other two comparison methods in terms of the FID indicator and the KL divergence indicator, which shows that the AI-ISP generalization method proposed in the present invention can significantly improve the performance of the AI-ISP model when deployed across cameras:
[0089] Table 1 Quantitative evaluation results Evaluation results;
[0090] Model FID↓ KL divergence↓ uvcgan 133.62 1.15+ Afifi's 86.80 1.63 Ours 62.24 1.06
[0091] The visualization results are as follows Figure 5 As shown, from left to right are the results without using domain generalization means, using the uvcgan algorithm, Afifi's algorithm, and the generalization results of the present invention, as well as the reference image. Figure 5 It can be seen that the present invention successfully generalizes the AI-ISP model. Even if the AI-ISP model is trained under the camera A data set, it can successfully render the original RAW image taken by camera B.
[0092] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.
Claims
1. An AI-ISP generalization method based on unsupervised camera color space mapping, It is characterized in that The following steps are involved: (1) Image acquisition and preprocessing; (1.1) Set the image acquisition format of camera A and camera B to RAW format, and arrange the acquired images in RGGB array; (1.2) Under the same scene conditions, use camera A and camera B to collect several images respectively, where the images collected by camera A are saved in image set A, and the images collected by camera B are saved in image set B; (1.3) For any image set, normalize each image in the image set first, then perform channel conversion so that each image is converted from the original one-channel to four-channel, and finally crop the channel-converted image into a block of size H*W; Finally, each image in the image set A is processed and stored in the tile set Set(A); each image in the image set B is processed and stored in the tile set Set(B); (2) Build a recurrent adversarial generative network; The cyclic adversarial generation network consists of two generators and two discriminators, where both generators use the camera color space block mapping network, denoted by G and a→b and G b→a , G a→b It is used to map the image in the color space of camera A to the color space of camera B. Another network G b→a Used to map the image in the color space of camera B to the color space of camera A; The camera color space block mapping network consists of a feature extraction network and a parameter matrix prediction network. The feature extraction network has two branches, a convolution branch and a Transformer branch, and performs feature fusion of the two branches through deformation and convolution operations. Specifically, the convolution branch consists of 8 convolution layers, and the Transformer branch consists of 4 Transformer layers. Every 2 convolution layers and 1 Transformer layer form a small block. For the input small block image I, it will first be sent to the first convolution layer and the Transformer layer at the same time. After the first convolution layer processes the image, the image is directly sent to the second convolution layer. The Transformer layer will divide the image into blocks, downsample it, and perform embedding operations to convert it into several visual vectors, and then perform self-attention operations and Feed Forward operation, then the visual vector will be reshaped and upsampled back to the same shape as the output of the second layer of convolution, and directly added, and used as the input of the next block after 2 times downsampling; the last small block will not be reshaped and upsampled, but directly used as the output of the feature extraction network, and the output shape is I'; the number of output channels of each convolution layer is 16, 32, 32, 64, 64, 128, 128, 256, using LeakeyRelu as the activation function; the embedding layer dimensions of the Transformer layer are 32, 64, 128, 256, using Gelu as the activation function; The parameter matrix prediction network consists of two fully connected layers, with the output I' of the feature extraction network as input. The output dimensions of the fully connected layer are 128 and 56 respectively, and the output of the fully connected layer is the color space mapping matrix CCM; then, the color space mapping matrix will be interpolated and upsampled to obtain CCM', and then rearranged into shape CCM"; at the same time, the original input image I is expanded from the original four channels to fourteen channels, and arranged and combined into image I poly Finally, I poly Perform element-by-element matrix multiplication with "CCM" to obtain the output of the entire network The two discriminators are denoted as D a , D b , each discriminator consists of a five-layer convolutional neural network and an average pooling layer; each layer of the five-layer convolutional neural network uses LeakyReLU as the activation function, the size of each convolution kernel is 4*4, the number of channels is 64, 128, 256, 512, and 1 respectively, and the average pooling layer is connected after the last layer of convolution to pool the feature map into the discrimination probability; (3) Training cyclic adversarial generative network; (3.1) Randomly sample a tile from Set(A), denoted as I A ; Randomly sample a tile from Set(B), denoted as I B ; (3.2), image I A Input camera color space block mapping network G a→b , and get the output image I AB ; Similarly, image I B Input camera color space block mapping network G b→a , get image I BA ; Then image I B Input camera color space block mapping network G a→b , get image I BB , image I A Input camera color space block mapping network G b→a , get image I AA ; Then calculate the consistency loss Loss iden ; Loss iden =MSE(I BB ,I B )+MSE(I AA ,I A ) Among them, MSE() means calculating the mean square error of each pixel in the two images; (3.3), image I AB Input Discriminator D a , and the discriminant probability is obtained Image I BA Input Discriminator D b , and the discriminant probability is obtained Then calculate the adversarial loss Loss of the generator gen-gan ; (3.4), image I AB Input camera color space block mapping network G b→a , get image I ABA , image I BA Input camera color space block mapping network G a→b , get image I BAB , and then calculate the cycle consistency loss Loss cycle ; Loss cycle =MSE(I ABA ,I A )+MSE(I BAB ,I B ) (3.5) Calculate the total loss Loss of the generator in the cyclic adversarial generation network gen ; Loss gen =Loss iden +Loss gen-gan +Loss cycle (3.6), image I A Input Discriminator D b , and the discriminant probability is obtained Image I B Input Discriminator D a , and the discriminant probability is obtained Then calculate the total loss Loss of the discriminator in the cyclic adversarial generation network dis ; (3.7), according to the total loss Loss of the generator gen , using the gradient descent method to update the camera color space block mapping network G a→b , G b→a Parameters; according to the total loss Loss of the discriminator dis , update the discriminator D using the gradient descent method a , D b Parameters; (3.8) After one round of training is completed, repeat steps (3.1)-(3.7) until the network converges, and obtain the cyclic adversarial generation network after training; (4) Generalizable ISP deployment; (4.1) Use the tile set Set(A) to train the AI-ISP network, and the trained network is denoted as ISP A ; (4.2) Extract the camera color space block mapping network G from the trained cyclic adversarial generation network b→a , then the network ISP A Series in G b→a The output terminal; Prepare an AI-ISP network trained in any way on the A camera dataset, denoted as ISP A , and the camera color space block mapping network G b→a ; (4.3) The image I captured by camera B B Input to the camera color space block mapping network G b→a Thus, the output image I BA , and then image I BA Input to network ISP A , and obtain a good quality RGB image F A .
