Image demosaicing method, device and storage medium based on diffusion model
Through the image demosaic method based on diffusion model, combined with the mosaic encoder and conditional noise predictor, unified demosaic processing in different color filter array modes is achieved, and the poor image demosaic effect and data set problems in the existing technology are solved. , improves image quality and processing efficiency.
Patent Information
- Application Number
- CN202411215314.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing technology cannot uniformly process the mosaic images collected by different color filter arrays, resulting in poor image desailing effect. The deep learning method relies on a large number of real data sets, and manual data acquisition is time-consuming and labor-intensive, making it difficult to expand to multiple filter modes. .
Using the image demosaic method based on diffusion model, the demosaic data set is constructed through the inverse ISP imaging process, the network is trained using the diffusion model, and combined with the mosaic encoder and the conditional noise predictor, the unified demosaic processing in different color filter array modes is achieved. .
It improves the integration and imaging quality of image demosaics, and can generate a variety of high-quality RGB images in different color filtering array modes, solving the problem of model training data and improving processing efficiency.
Smart Images

Figure CN118735771B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and relates to an image de-mosaicing method and device and a storage medium based on a diffusion model, and in particular to a method for performing unified de-mosaicing processing on images acquired by different color filter array modes. Background Art
[0002] The Image Single Process (ISP) imaging process includes steps such as demosaicing of the original raw image, white balance, tone correction, gamma correction, and color space conversion. Traditional CMOS sensors use various color filter arrays to perform lossy filtering on real color light, and then perform the ISP imaging process to synthesize full-color RGB images. Among them, different demosaicing methods will change the distribution of data and affect the final image quality. Traditional CMOS sensors use Bayer filter arrays to capture images and use interpolation processing in the demosaicing step. Modern deep learning-based demosaicing methods are committed to building a mapping relationship between mosaic images and RGB images. However, different color filter array patterns require different network models for processing. At the same time, deep learning methods themselves require a large number of high-quality data sets that conform to the real world.
[0003] Different color filter arrays perform different lossy filtering on incident photons, resulting in different brightness responses and color accuracy. A solution to adapt to scenes with complex changes in light response is to use a self-transforming filter array, which transforms between Bayer, Quad, Nona, and Q×Q filter arrays according to the number of incident photons. The self-transforming filter array merges adjacent color pixels to obtain higher sensitivity when the number of incident photons is insufficient, and reduces the pixel group unit size to obtain a high-resolution image when the number of incident photons is sufficient. Deep learning-based methods have achieved good results in processing images collected by different color filter arrays. However, these methods usually only focus on a specific demosaicing method, which requires the use of another demosaicing model when processing images collected by multiple color filter arrays, which increases storage consumption and computing expenses.
[0004] Due to the lossy lighting imaging of the sensor, image demosaicing becomes an ill-posed problem. Pekkucuksen et al. used an interpolation-based method to perform multi-channel demosaicing on mosaic images, and Lefkimmiatis et al. used a deep learning method to find the mapping relationship between mosaic images and RGB images. However, the interpolation-based method cannot cope with the complex distribution of real-world images, and the existing neural network algorithm only focuses on a single filtering mode, making it difficult to uniformly process mosaic images in different modes.
[0005] The deep learning-based method relies on the authenticity of the training data. Manual data collection is a time-consuming and labor-intensive task. Therefore, it is of great practical significance to synthesize mosaic images from RGB images according to different color filter array modes. A widely used synthesis method is to use deep learning methods to construct the relationship between RGB images and mosaic images. This method has achieved good results on the Bayer array, but it cannot be extended to the synthesis of a wide range of original images due to the lack of real data samples in other color filter array modes. Therefore, a deep learning method is urgently needed to uniformly demosaic the original images in different modes and finally obtain diverse and high-quality RGB images. Summary of the invention
[0006] According to the problems that the existing original image demosaicing method cannot uniformly process mosaic images collected by different color filter arrays or the processed RGB images are not rich, the present invention proposes an image demosaicing method and device and storage medium based on a diffusion model. The present invention can uniformly and efficiently process mosaic images collected by different color filter arrays, thereby improving the integration, lightness and imaging quality of original image demosaicing.
[0007] To achieve the above objectives, the present invention adopts the following technical solutions.
[0008] In a first aspect, the present invention provides an image demosaicing method based on a diffusion model, comprising the following steps:
[0009] Step 1: Obtain a real-world RGB image and use the inverse ISP imaging process to construct a demosaic dataset, which includes the RGB image, the corresponding noisy mosaic image, and pattern information;
[0010] Step 2: Build a demosaicing network based on the diffusion model, and train, verify and test it;
[0011] Step 2.1: Construct a mosaic encoder based on the pattern information of the noisy mosaic image;
[0012] Step 2.2: Construct a conditional noise predictor, and use the forward-backward steps to gradually denoise the random noise image according to the input noisy mosaic image, and finally generate a rich high-quality RGB image;
[0013] Step 3: Use the trained, verified and tested diffusion model-based demosaicing network to achieve image demosaicing.
[0014] Preferably, the inverse ISP imaging process in step 1 is to synthesize the three-channel RGB image into a clean mosaic image under four channels according to the mosaic color filter array mode, and add Poisson noise and Gaussian noise to the clean mosaic image to obtain a noisy mosaic image that conforms to the real world;
[0015] The pattern information refers to a mask under a mosaic color filter array mode; the mosaic color filter array mode includes Bayer, non-Bayer Quad (Quad), Nonacell (Nona) and Quad-by-Quad (Q×Q) array modes.
[0016] Preferably, the mosaic encoder adopts a dual-module parallel spatial adaptive network to perform mosaic conditional encoding on noisy mosaic images under different mosaic color filter array modes, which includes a mosaic encoding module, a weight generation module, and a global connection module; the weight generation module is based on the pattern information representing the spatial position of the image. Generate dynamic weights; the mosaic encoding module uses a spatial adaptive convolution layer to receive the dynamic weights generated by the weight generation module and perform feature updates; the global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the final output of the mosaic encoder;
[0017] More preferably, the main framework of the weight generation module adopts a U-Net architecture, including an encoder, a decoder, and an intermediate layer, the encoder includes N first feature extraction layers connected in series, the decoder includes N first upsampling layers connected in series, N>1, and the intermediate layer is located between the encoder and the decoder; a jump connection is used between the first feature extraction layer and the first upsampling layer located in the same layer;
[0018] Each first feature extraction layer includes a first residual block and a first downsampling block connected in series;
[0019] The first upsampling layer includes a first residual block and a first upsampling block connected in series;
[0020] The middle layer includes a first residual block;
[0021] In each first feature extraction layer, the input pattern information passes through the first residual block, expands the number of channels, and then performs channel fusion with the input pattern information to obtain the output of the current first residual block; the output of the current first residual block enters the next first feature extraction layer or the intermediate layer through the first downsampling block in the encoder stage, and at the same time, performs channel fusion with the output of the previous decoding layer or the intermediate layer in the decoder stage, and then passes through the first upsampling block to obtain the output of the current decoding layer, and serves as the input of the next decoding layer of the current weight generation module; in addition, the dynamic weight generated by the first residual block in the weight generation module is also used as one of the inputs of the corresponding layer of the mosaic coding module;
[0022] The main framework of the mosaic coding module adopts the U-Net architecture, including an encoder, a decoder, and an intermediate layer. The encoder includes N second feature extraction layers connected in series, the decoder includes N second upsampling layers connected in series, and the intermediate layer is located between the encoder and the decoder; the second feature extraction layer and the second upsampling layer located in the same layer adopt a jump connection;
[0023] Each second feature extraction layer includes a second residual block and a second downsampling block connected in series, the middle layer is composed of a second residual block, and the second upsampling layer includes a second residual block and a second upsampling block connected in series;
[0024] Each second feature extraction layer receives the dynamic weights generated by the first residual block in the first feature extraction layer of the same layer of the weight generation module, and passes the input noisy mosaic image through the second residual block, expands the number of channels, and then performs channel fusion with the input noisy mosaic image to obtain the output of the current second residual block; the output of the current second residual block enters the second feature extraction layer or the middle layer of the next layer through the second downsampling block in the encoder stage, and at the same time, performs channel fusion with the output of the previous decoding layer or the middle layer and the dynamic weights generated by the first residual block in the decoding layer of the same layer of the weight generation module in the decoder stage, and then passes through the second upsampling block to obtain the output of the current decoding layer, and serves as the input of the next decoding layer of the current mosaic encoding module;
[0025] The global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the output of the mosaic encoder;
[0026] The loss function of the mosaic encoder is as follows:
[0027]
[0028] in, represents the network parameters of the mosaic encoder, represents the total number of training samples, represents the network mapping function of the mosaic encoder, Indicates Noisy mosaic image samples, Indicates The pattern information corresponding to the mosaic noisy image samples, Indicates RGB image samples.
[0029] Preferably, the conditional noise predictor is a network based on a diffusion model, and the diffusion process is: Image gradually converted into a simple Gaussian distribution by adding noise ,in represents the total number of diffusion steps; the reverse process is: Image of the moment The original RGB image is gradually denoised and converted into a reparameterized distribution ;
[0030] The main framework of the conditional noise predictor adopts the U-Net structure, including the pre-convolution block, encoder, decoder, intermediate layer, and post-convolution block;
[0031] The encoder includes N third feature extraction layers connected in series, the decoder includes N third upsampling layers connected in series, the intermediate layer is located between the encoder and the decoder; the third feature extraction layer and the third upsampling layer located in the same layer adopt a jump connection;
[0032] Each third feature extraction layer includes two third residual blocks and a third downsampling block in series, specifically:
[0033] The first and third feature extraction layers will RGB image at the moment Image processed by the pre-convolutional block, output of the mosaic encoder , at time t, the first third residual block undergoes channel transformation, the second third residual block expands the number of channels of the output of the first third residual block and then performs channel fusion with the output at time t to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the same decoding layer through a jump connection, and at the same time, the output of the second third residual block passes through the third downsampling block as the input of the third feature extraction layer of the next layer;
[0034] The third feature extraction layers of the second to N layers transform the output of the third feature extraction layer of the previous layer at time t through the first third residual block, and the second third residual block expands the number of channels of the output of the first third residual block and performs channel fusion with time t to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the decoding layer of the same layer through a jump connection, and the output of the second third residual block is used as the input of the third feature extraction layer of the next layer or the input of the intermediate layer through the third downsampling block;
[0035] The intermediate layer includes two third residual blocks connected in series;
[0036] Each third upsampling layer includes two third residual blocks and a third upsampling block connected in series, specifically:
[0037] Each third upsampling layer is the output of the third upsampling layer or the middle layer of the previous layer, and the output is transformed by the first third residual block at time t. The second third residual block expands the number of channels of the output of the first third residual block and performs channel fusion with the output of the second third residual block in the same coding layer at time t to obtain the output of the second third residual block; the output of the second third residual block passes through the third upsampling block as the input of the third upsampling layer of the next layer or the final output of the decoder;
[0038] The loss function of the conditional noise predictor uses the L1 distance between the noise predicted by the conditional noise predictor and the noise added during the diffusion process, and its formula is as follows:
[0039]
[0040] in, represents the network parameters of the conditional noise predictor, The number of diffusion steps is The noise added when Indicates the number of steps Next, according to the kth image sample at the current time t And corresponding mosaic condition information The noise predicted by the conditional noise predictor.
[0041] In a second aspect, the present invention provides an image demosaicing system based on the above method, comprising:
[0042] A data acquisition module, which acquires a noisy mosaic image and its pattern information in a mosaic color filter array mode;
[0043] The data processing module is responsible for restoring the noisy mosaic image to the original clear RGB image using the trained, verified and tested diffusion model-based demosaicing network to achieve image demosaicing.
[0044] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the image de-mosaicing method.
[0045] In a fourth aspect, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the image de-mosaicing method is implemented.
[0046] Beneficial effects of the present invention:
[0047] (1) The present invention uses the inverse ISP imaging process to artificially synthesize mosaic samples that meet the real-world distribution under different color filter arrays, solving the problem of model training data;
[0048] (2) The present invention introduces the spatial adaptive convolution mechanism into the UNet network model and uses it for the mosaic encoder, which can adaptively learn the mapping functions of different spatial positions according to the given spatial position, and can obtain the real pixel spatial position distribution when the R, G, G, B channel pixels are misaligned, thereby improving the overall quality of the model mapping, and can be extended to the demosaicing processing under different color filter arrays;
[0049] (3) The present invention integrates the output of the mosaic encoder into the conditional noise predictor, and shortens the conversion path from mosaic image to RGB image under the guidance of high-quality mosaic conditional coding; under the conditions of mosaic images with different color filter arrays, the conditional noise predictor based on the diffusion model can gradually denoise the random noise under the simple Gaussian distribution to obtain a high-quality RGB image, thereby improving the integration of demosaicing of the original image. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a general flow chart of the unified demosaicing method of original images based on the diffusion model disclosed in the present invention;
[0051] Figure 2 It is a flow chart of synthesizing a demosaiced data set using an inverse ISP process in the present invention;
[0052] Figure 3 It is a schematic diagram of a mosaic encoding module and a weight generating module in the present invention;
[0053] Figure 4 An overview of the network architecture of the mosaic encoder in the present invention (left) and a schematic diagram of the spatially adaptive convolutional layer (right);
[0054] Figure 5 is an overview of the network architecture of the conditional noise predictor in the present invention (left) and a schematic diagram of the third residual block (right);
[0055] Figure 6 1 is a comparison diagram of the application effects of the unified de-mosaicing method for original images based on the diffusion model disclosed in the present invention in the Bayer and Quad mosaic filter array modes; wherein (a) is a noisy mosaic image in the Bayer mosaic filter array mode, (b) is a PS processing result of (a), (c) is a processing result of the present invention in (a), (d) is a noisy mosaic image in the Quad mosaic filter array mode, (e) is a PS processing result of (d), and (f) is a processing result of the present invention in (d);
[0056] Figure 7 This is a comparison diagram of the application effects of the unified de-mosaicing method for original images based on the diffusion model disclosed in the present invention in Nona and QxQ mosaic filter array modes; wherein (a) is a noisy mosaic image in the Nona mosaic filter array mode, (b) is the PS processing result of Figure (a), (c) is the processing result of the present invention in Figure (a), (d) is a noisy mosaic image in the QxQ mosaic filter array mode, (e) is the PS processing result of Figure (d), and (f) is the processing result of the present invention in Figure (d). DETAILED DESCRIPTION
[0057] In order to better illustrate the invention and advantages of this project, the invention content is further described below with reference to the drawings and examples.
[0058] Example 1.
[0059] This embodiment discloses an image demosaicing method based on a diffusion model, uses an inverse ISP imaging process to synthesize a demosaicing data set; constructs a demosaicing network based on a diffusion model, sets appropriate hyperparameters to train network parameters; uniformly processes mosaic images in different modes and generates a variety of high-quality RGB images. The present invention can uniformly process mosaic images collected by different color filter arrays and generate a variety of high-quality RGB images, expanding the application scope of image demosaicing. The present invention can be applied to multiple fields such as video surveillance and low-light imaging.
[0060] like Figure 1 As shown, the image demosaicing method based on the diffusion model disclosed in this embodiment includes the following steps:
[0061] Step S1: obtaining an RGB image in the real world, and constructing a demosaicing data set using an inverse ISP imaging process, the data set including the RGB image, the corresponding noisy mosaic image and pattern information; the pattern information refers to a mask in a mosaic color filter array mode, and its value is 0 or 1, respectively indicating that there is no information or information on the noisy mosaic image, and the corresponding noisy mosaic image has the same number of channels, channel order and resolution;
[0062] Step S2: constructing a demosaicing network based on a diffusion model, and training, validating and testing it;
[0063] Step S2.1: constructing a mosaic encoder according to the pattern information of the noisy mosaic image;
[0064] Step S2.2: construct a conditional noise predictor, and use the forward-backward steps to gradually denoise the random noise image according to the input noisy mosaic image, and finally generate a rich high-quality RGB image;
[0065] Step S3: using the trained, verified and tested diffusion model-based demosaicing network to achieve image demosaicing;
[0066] Wherein, step S1 specifically comprises:
[0067] To obtain RGB images in the real world, this embodiment uses 8× resolution RGB images from the public DIV2K and Flickr2K datasets, and uses the inverse ISP imaging process to synthesize noisy mosaic images from the RGB images according to four mosaic filter array modes of Bayer, Quad, Nona and Q×Q in the order of R, G, G, B channels;
[0068] Select noisy mosaic images in turn, and construct pattern information according to the mosaic color filter array pattern they have. The pattern information consists of {0, 1} masks, which respectively indicate that there is no information or information on the original noisy mosaic image, and the corresponding noisy mosaic image has the same number of channels, channel order and resolution; the demosaicing dataset in step 1 consists of an RGB image, a noisy mosaic image and pattern information, which is expressed as ,in Indicates Color filter array modes, They represent four mosaic filter array modes: Bayer, Quad, Nona and Q×Q. represents the total number of samples, Indicates samples, Indicates Noisy mosaic image samples, Indicates The pattern information corresponding to the mosaic noisy image samples, Indicates RGB image samples;
[0069] like Figure 2 The inverse ISP imaging process is to synthesize the three-channel RGB image into a clean mosaic image under four channels according to the mosaic color filter array mode, and add Poisson noise and Gaussian noise to the clean mosaic image to obtain a noisy mosaic image that conforms to the real world;
[0070] The mosaic encoder described in step S2.1 adopts a dual-module parallel spatial adaptive network to perform mosaic conditional encoding on noisy mosaic images under different mosaic color filter array modes, and can extract the pattern information of the original image to guide the conditional encoding of the corresponding mosaic image.
[0071] like Figure 3-4 As shown in the figure, the dual-module parallel spatial adaptive network includes a parallel mosaic encoding module and a weight generation module, as well as a global connection module; the mosaic training dataset is of , The mosaic encoding module and the weight generation module are input respectively. The mosaic encoding module performs pattern encoding on the pattern information of the original image. The weight generation module receives the feature map encoding information generated by the encoder and generates mosaic weights at specific spatial positions to ensure that the mosaic image has corresponding mapping functions at different spatial positions. In other words, the weight generation module generates the mosaic weights according to the pattern information representing the spatial position of the image. Generate dynamic weights. The mosaic encoding module uses a spatial adaptive convolution layer to receive the dynamic weights generated by the weight generation module and perform feature updates. The global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the final output of the mosaic encoder. Specifically:
[0072] (1) Weight generation module
[0073] The weight generation module is based on the pattern information representing the spatial position of the image. Generating Dynamic Weights ;
[0074] (1)
[0075] in, Indicates the number of input channels, represents the index of the i-th channel, , , Respectively represent the index of the pixel value of the mode information, Represents a noisy mosaic image The corresponding mode information, represents the convolution kernel in the weight generation module, , Represent the height and width of the convolution kernel respectively;
[0076] The main framework of the weight generation module adopts the U-Net architecture, which includes an encoder, a decoder, and an intermediate layer. The encoder includes four first feature extraction layers connected in series, and the decoder includes four upsampling layers connected in series. The intermediate layer is located between the encoder and the decoder; the first feature extraction layer and the upsampling layer located in the same layer adopt a jump connection;
[0077] Each first feature extraction layer includes a first residual block and a first downsampling block connected in series, the middle layer is composed of a first residual block, and the upsampling layer includes a first residual block and a first upsampling block connected in series;
[0078] In each first feature extraction layer, the input pattern information passes through the first residual block, and the number of channels is expanded by 2× to perform channel fusion with the input pattern information to obtain the output of the current first residual block; the output passes through the first downsampling block in the encoder stage to enter the next first feature extraction layer or the intermediate layer, and at the decoder stage, the output is channel fused with the output of the previous decoding layer or the intermediate layer, and then passes through the first upsampling block to obtain the corresponding output, which serves as the input of the next decoding layer of the current weight generation module; in addition, the dynamic weight generated by the first residual block in the weight generation module is also used as one of the inputs of the corresponding layer components of the mosaic coding module;
[0079] The first residual block includes a first convolutional layer, an activation function, a second convolutional layer, and an activation function; the specific implementation process is as follows: the first convolutional layer transforms the number of channels of the pattern information into a ratio of 2× and 1 / 2× in the encoder and decoder stages respectively, the second convolutional layer uses 1×1 convolution for feature extraction, and uses LeakyReLU as the activation function;
[0080] The first downsampling block uses a 4×4 convolutional layer with a stride of 2 to transform the output size to a 1 / 2× ratio;
[0081] The first upsampling block uses bilinear interpolation to transform the output size to a 2× scale;
[0082] (2) Mosaic coding module
[0083] The mosaic encoding module first performs convolution operations on the noisy mosaic image and its feature map, and then accepts the corresponding weights generated by the weight generation module. Perform feature update, expressed as:
[0084] (2)
[0085] (3)
[0086] in represents the output feature image under the adaptive convolution mechanism of the mosaic coding module, represents a noisy mosaic feature image, represents the convolution kernel in the mosaic encoding module, , Represent the height and width of the convolution kernel respectively, Representing images After the convolution kernel The image after operation, H and W represent the image height and width;
[0087] The main framework of the mosaic coding module adopts the U-Net architecture with the same architecture as the weight generation module, including an encoder, a decoder, and an intermediate layer. The encoder includes 4 second feature extraction layers connected in series, and the decoder includes 4 upsampling layers connected in series. The intermediate layer is located between the encoder and the decoder; the second feature extraction layer and the upsampling layer located in the same layer adopt a jump connection;
[0088] Each second feature extraction layer includes a second residual block and a second downsampling block connected in series, the middle layer is composed of a second residual block, and the upsampling layer includes a second residual block and a second upsampling block connected in series;
[0089] Each second feature extraction layer receives the dynamic weights generated by the first residual block in the first feature extraction layer of the same layer of the weight generation module, and passes the input noisy mosaic image through the second residual block, expands the number of channels by a ratio of 2×, and performs channel fusion with the input noisy mosaic image to obtain the output of the second residual block; the output enters the second feature extraction layer or the middle layer of the next layer through the second downsampling block at the encoder stage, and at the same time, at the decoder stage, the output is channel fused with the output of the previous decoding layer or the middle layer, and the dynamic weights generated by the first residual block in the decoding layer of the same layer of the weight generation module, and then passes through the second upsampling block to obtain the corresponding output, which is used as the input of the next decoding layer of the current mosaic encoding module;
[0090] The second residual block includes a first adaptive convolution layer, an activation function, a second adaptive convolution layer, and an activation function; the specific implementation process is as follows:
[0091] The first adaptive convolution layer transforms the number of channels of the noisy mosaic image into a ratio of 2× and 1 / 2× in the encoder and decoder stages respectively, and receives the dynamic weights generated by the weight generation module corresponding to the first convolution layer to perform feature update according to formula (2). The second convolution layer uses 1×1 convolution for feature extraction, and receives the dynamic weights generated by the weight generation module corresponding to the second convolution layer to perform feature update according to formula (2). The activation function is the same as the activation function in the weight generation module.
[0092] The second down-sampling block and the second up-sampling block have the same structure as the first down-sampling block and the first up-sampling block in the weight generation module respectively;
[0093] (3) Global connection module
[0094] The global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the output of the mosaic encoder;
[0095] The global connection module includes a global first residual block and a global second residual block, and the noisy mosaic image is numerically added to the input noisy mosaic image through the global second residual block to obtain the output of the global connection module;
[0096] The global first residual block includes a global convolution layer, an activation function, and a global convolution layer; the global second residual block includes a global adaptive convolution layer, an activation function, a global adaptive convolution layer, and an activation function; the specific implementation process is as follows:
[0097] The global convolution layer extracts spatial information from the pattern information without changing the number of channels and image size. The global adaptive convolution layer receives the dynamic weights generated by the corresponding global convolution layer and updates the features according to formula (2). The activation function is the same as the activation function in the weight generation module.
[0098] (4) Loss function of mosaic encoder
[0099] The L1 distance between the mosaic image and the output image obtained by the mosaic encoder is used as the loss function to perform gradient descent optimization on the mosaic encoder model parameters until convergence.
[0100] The loss function of the mosaic encoder uses the L1 distance between the output of the mosaic encoder and the corresponding RGB image, and its formula is as follows:
[0101] (4)
[0102] in, represents the network parameters of the mosaic encoder, represents the total number of training samples, represents the network mapping function of the mosaic encoder, Indicates Noisy mosaic image samples, Indicates The pattern information corresponding to the mosaic noisy image samples, Indicates RGB image samples;
[0103] The main framework of the conditional noise predictor described in step S2.2 adopts a U-Net structure, which includes a pre-convolution block, an encoder, a decoder, an intermediate layer, and a post-convolution block. Figure 5 ;
[0104] The pre-convolution block and the post-convolution block are respectively distributed at the starting point and the ending point of the conditional noise predictor, and are composed of a 2D convolution layer and a Mish activation function;
[0105] The encoder includes four third feature extraction layers connected in series, the decoder includes four upsampling layers connected in series, the intermediate layer is located between the encoder and the decoder; the third feature extraction layer and the upsampling layer located in the same layer adopt a jump connection;
[0106] Each third feature extraction layer includes two third residual blocks connected in series and a third downsampling block, the middle layer is composed of two third residual blocks connected in series, and the upsampling layer includes two third residual blocks connected in series and an upsampling block; the specific implementation process is as follows:
[0107] Encoding stage: The first and third feature extraction layers will Image processed by the pre-convolutional block, output of the mosaic encoder , t undergoes channel transformation through the first third residual block, and the second third residual block performs channel fusion with t by expanding the output of the first third residual block by a ratio of 2×, to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the same decoding layer through a jump connection, and the output of the second third residual block is used as the input of the third feature extraction layer of the next layer through the third downsampling block;
[0108] The second to fourth third feature extraction layers use the output of the third feature extraction layer of the previous layer, t, to perform channel transformation through the first third residual block, and the second third residual block uses the output of the first third residual block to expand the number of channels by a ratio of 2× and perform channel fusion with t to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the decoding layer of the same layer through a jump connection, and the output of the second third residual block is used as the input of the third feature extraction layer of the next layer or the input of the intermediate layer through the third downsampling block;
[0109] Decoding stage: Each upsampling layer is the output of the previous upsampling layer or the middle layer, t is transformed through the first third residual block, and the second third residual block expands the number of channels of the output of the first third residual block by a ratio of 1 / 2× and performs channel fusion with the output of the second third residual block in the same coding layer at time t to obtain the output of the second third residual block; the output of the second third residual block passes through the third upsampling block as the input of the next upsampling layer or the final output of the decoder;
[0110] The final output of the decoder is processed by the post-convolution block to obtain the output of the conditional noise predictor t ;
[0111] The third residual block is composed of two convolution blocks, the first convolution block is used to transform the number of channels of the image to be iterated into a ratio of 2× and 1 / 2× in the encoder and decoder stages respectively, and the step number t is sinusoidally encoded and converted into an equal-channel addable tensor, which is numerically added to the output of the first convolution block to complete channel fusion and obtain the input of the second convolution block; the second convolution block performs feature extraction without changing the number of channels;
[0112] The third downsampling block uses a 4×4 convolutional layer with a stride of 2 to transform the output size to a 1 / 2× ratio;
[0113] The third upsampling block uses bilinear interpolation to transform the output size to a 2× scale;
[0114] The conditional noise predictor is a noise prediction network based on a diffusion model. The diffusion process of the conditional noise predictor is: Transformed into a simple Gaussian distribution ,in represents the total number of diffusion steps; The image at a moment is defined as:
[0115] (5)
[0116] in For a given hyperparameter, , Indicates The noisy RGB image at the moment, represents the noise term that conforms to the standard Gaussian distribution;
[0117] The reverse process will Transformed into a reparameterized distribution ,have:
[0118] (6)
[0119] (7)
[0120] in, Indicates The noise predicted by the time conditional noise predictor, Indicates the mosaic condition information obtained by the mosaic encoder. Indicates The image variance at time instant, , represents the auxiliary term that conforms to the Gaussian distribution, When , the auxiliary item is 0, then ;
[0121] When training the conditional noise predictor, we first sample noise from a standard Gaussian distribution, and then Randomly select steps Get ; From the demosaiced dataset Sampling No. Noisy mosaic image , will Noisy mosaic images and corresponding pattern information Input mosaic encoder to get mosaic condition information , and the input conditional noise predictor predicts the current noise at time t;
[0122] The loss function of the conditional noise predictor uses the L1 distance between the noise predicted by the conditional noise predictor and the noise added during the diffusion process, and its formula is as follows:
[0123] (8)
[0124] in, represents the network parameters of the conditional noise predictor, The number of diffusion steps is The noise added when Indicates the number of steps Next, according to the kth image sample at the current moment And corresponding mosaic condition information the noise predicted by the conditional noise predictor;
[0125] Step S3 specifically uses the trained diffusion model-based demosaicing network to reversely reconstruct the noisy mosaic image with a specific mosaic color filter array pattern. and its mode information Go to the mosaic encoder to get the mosaic condition information , and then randomly sample noise from a Gaussian distribution from the number of steps Iterate according to formulas (6) and (7) until the original clear RGB image is restored.
[0126] The image demosaicing method, device and storage medium based on diffusion model of the present invention can uniformly and efficiently process mosaic images collected by different color filter arrays: Figure 6 Figures (a)-(f) and attached Figure 7 Figures (a)-(f) show the de-mosaicing effect of the present method under the conditions of Bayer, Quad, Nona and QxQ color filter arrays. The RGB image reconstructed using Photoshop will amplify the noise of the mosaic image and have lower human eye perception quality. In comparison, the de-mosaicing method of the present invention can effectively remove the noise of the mosaic image and have richer image details and higher color performance quality.
[0127] An embodiment of the present invention provides a computing device. Specifically, the electronic device includes a memory and a processor. The memory stores executable code. When the processor executes the executable code, the method described in any one of the implementation modes is implemented.
[0128] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (Non-volatile Memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0129] The bus may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.
[0130] Among them, the memory is used to store the program, and the processor executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor or implemented by the processor.
[0131] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0132] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.
[0133] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0134] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. Image demosaicing method based on diffusion model, characterized by The following steps are involved: Step 1: Obtain a real-world RGB image and use the inverse ISP imaging process to construct a demosaic dataset, which includes the RGB image, the corresponding noisy mosaic image and pattern information; the pattern information refers to the mask under the mosaic color filter array mode; the mosaic color filter array mode includes Bayer, Quad, Nona and Q×Q array modes; Step 2: Build a demosaicing network based on the diffusion model, and train, verify and test it; Step 2.1: Construct a mosaic encoder based on the noisy mosaic image and the corresponding pattern information; Step 2.2: Construct a conditional noise predictor, and use the forward-backward steps to gradually denoise the random noise image according to the input noisy mosaic image, and finally generate a rich high-quality RGB image; Step 3: Use the trained, verified and tested diffusion model-based demosaicing network to achieve image demosaicing; The mosaic encoder adopts a dual-module parallel spatial adaptive network to perform mosaic conditional encoding on noisy mosaic images under different mosaic color filter array modes, which includes a mosaic encoding module, a weight generation module, and a global connection module; the weight generation module is based on the pattern information representing the spatial position of the image. Generate dynamic weights; the mosaic coding module introduces a spatial adaptive convolution mechanism, receives the dynamic weights generated by the weight generation module and performs feature update; The global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the final output of the mosaic encoder; The main framework of the weight generation module adopts the U-Net architecture, including an encoder, a decoder, and an intermediate layer. The encoder includes N first feature extraction layers connected in series, and the decoder includes N first upsampling layers connected in series, N>1. The intermediate layer is located between the encoder and the decoder; a jump connection is used between the first feature extraction layer and the first upsampling layer located in the same layer; Each first feature extraction layer includes a first residual block and a first downsampling block connected in series; The first upsampling layer includes a first residual block and a first upsampling block connected in series; The middle layer includes a first residual block; In each first feature extraction layer, the input pattern information passes through the first residual block, expands the number of channels, and then performs channel fusion with the input pattern information to obtain the output of the current first residual block; The output of the current first residual block enters the first feature extraction layer or the intermediate layer of the next layer through the first downsampling block at the encoder stage, and is channel-fused with the output of the previous decoding layer or the intermediate layer at the decoder stage, and then passes through the first upsampling block to obtain the output of the current decoding layer, and serves as the input of the next decoding layer of the current weight generation module; in addition, the dynamic weight generated by the first residual block in the weight generation module is also used as one of the inputs of the corresponding layer of the mosaic coding module; The main framework of the mosaic coding module adopts the U-Net architecture, including an encoder, a decoder, and an intermediate layer. The encoder includes N second feature extraction layers connected in series, the decoder includes N second upsampling layers connected in series, and the intermediate layer is located between the encoder and the decoder; the second feature extraction layer and the second upsampling layer located in the same layer adopt a jump connection; Each second feature extraction layer includes a second residual block and a second downsampling block connected in series, the middle layer is composed of a second residual block, and the second upsampling layer includes a second residual block and a second upsampling block connected in series; Each second feature extraction layer receives the dynamic weights generated by the first residual block in the first feature extraction layer of the same layer of the weight generation module, and passes the input noisy mosaic image through the second residual block, expands the number of channels, and then performs channel fusion with the input noisy mosaic image to obtain the output of the current second residual block; The output of the current second residual block enters the second feature extraction layer or the intermediate layer of the next layer through the second downsampling block at the encoder stage, and at the decoder stage, performs channel fusion with the output of the previous decoding layer or the intermediate layer, and the dynamic weight generated by the first residual block in the same decoding layer of the weight generation module, and then passes through the second upsampling block to obtain the output of the current decoding layer, which is used as the input of the next decoding layer of the current mosaic coding module; The global connection module adds the noisy mosaic image and the corresponding pattern information after global adaptive convolution processing to the output result of the mosaic encoding module as the output of the mosaic encoder; The loss function of the mosaic encoder is as follows: in, represents the network parameters of the mosaic encoder, represents the total number of training samples, represents the network mapping function of the mosaic encoder, Indicates Noisy mosaic image samples, Indicates The pattern information corresponding to the mosaic noisy image samples, Indicates RGB image samples; The conditional noise predictor is a network based on a diffusion model, and its diffusion process is: the complex RGB image is Image gradually converted into a simple Gaussian distribution by adding noise ,in represents the total number of diffusion steps; the reverse process is: Image of the moment The original RGB image is gradually denoised and converted into a reparameterized distribution ; The main framework of the conditional noise predictor adopts the U-Net structure, including the pre-convolution block, encoder, decoder, intermediate layer, and post-convolution block; The encoder includes N third feature extraction layers connected in series, the decoder includes N third upsampling layers connected in series, the intermediate layer is located between the encoder and the decoder; the third feature extraction layer and the third upsampling layer located in the same layer adopt a jump connection; Each third feature extraction layer includes two third residual blocks and a third downsampling block in series, specifically: The first and third feature extraction layers will RGB image at the moment Image processed by the pre-convolutional block, output of the mosaic encoder , at time t, the first third residual block undergoes channel transformation, the second third residual block expands the number of channels of the output of the first third residual block and then performs channel fusion with the output at time t to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the same decoding layer through a jump connection, and at the same time, the output of the second third residual block passes through the third downsampling block as the input of the third feature extraction layer of the next layer; The third feature extraction layers of the second to N layers transform the output of the third feature extraction layer of the previous layer at time t through the first third residual block, and the second third residual block expands the number of channels of the output of the first third residual block and performs channel fusion with time t to obtain the output of the second third residual block; the output of the second third residual block is used as the input of the second third residual block in the decoding layer of the same layer through a jump connection, and the output of the second third residual block is used as the input of the third feature extraction layer of the next layer or the input of the intermediate layer through the third downsampling block; The intermediate layer includes two third residual blocks connected in series; Each third upsampling layer includes two third residual blocks and a third upsampling block connected in series, specifically: Each third upsampling layer is the output of the third upsampling layer or the middle layer of the previous layer, and the output is transformed by the first third residual block at time t. The second third residual block expands the number of channels of the output of the first third residual block and performs channel fusion with the output of the second third residual block in the same coding layer at time t to obtain the output of the second third residual block; the output of the second third residual block passes through the third upsampling block as the input of the third upsampling layer of the next layer or the final output of the decoder; The loss function of the conditional noise predictor uses the L1 distance between the noise predicted by the conditional noise predictor and the noise added during the diffusion process, and its formula is as follows: in, represents the network parameters of the conditional noise predictor, The number of diffusion steps is The noise added when Indicates the number of steps Next, according to the kth image sample at the current time t And corresponding mosaic condition information The noise predicted by the conditional noise predictor.
2. The method according to claim 1, characterized in that: The inverse ISP imaging process described in step 1 is to synthesize the three-channel RGB image into a clean mosaic image under four channels according to the mosaic color filter array pattern, and add Poisson noise and Gaussian noise to the clean mosaic image to obtain a noisy mosaic image that conforms to the real world.
3. An image demosaicing system based on the method according to any one of claims 1 to 2, characterized in that include: A data acquisition module, which acquires a noisy mosaic image and its pattern information in a mosaic color filter array mode; The data processing module is responsible for restoring the noisy mosaic image to the original clear RGB image using the trained, verified and tested diffusion model-based demosaicing network to achieve image demosaicing.
4. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 2.
5. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 2 is implemented.
