A system and method for image conversion between Bayer format and quad Bayer format

Abstract

This invention provides an image conversion system and method between Bayer format and quad Bayer format. The system includes an image acquisition module that acquires an original Bayer format image and an original quad Bayer format image; a re-mosaic module that converts the original quad Bayer format image and the generated quad Bayer format image into a Bayer format image; a de-mosaic module that converts the original Bayer format image and the generated Bayer format image into a quad Bayer format image; and an image output module that outputs the generated quad Bayer format image and the Bayer format image. By unifying the re-mosaic and de-mosaic processes within the same system framework, the correlation between Bayer and quad Bayer formats can be better captured, and information loss during format conversion can be effectively reduced. The repeated re-mosaic and de-mosaic processes can also enhance each other, improving the conversion accuracy and quality, thus solving the problem of poor performance in existing Bayer and quad Bayer format image conversion methods.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image conversion system and method between Bayer format and quad Bayer format. Background Technology

[0002] In the field of digital image acquisition and processing, image sensors typically use Bayer filter arrays to capture color information, where each pixel captures only one color (such as red, green, and blue). In recent years, the Quad Bayer format has been introduced, which improves low-light sensitivity or dynamic range by combining four adjacent pixels into a superpixel.

[0003] To facilitate image signal processing (ISP) of Bayer format RAW images or quad Bayer format RAW images to generate RGB images, it is often necessary to convert Bayer format to quad Bayer format or vice versa, depending on the requirements. However, existing format conversion methods treat the conversion between ordinary Bayer format and quad Bayer format as an independent task. The format conversion process usually requires complex demosaic interpolation or remosaic algorithms, resulting in loss of image details, artifact generation, or low computational efficiency during the conversion process. Summary of the Invention

[0004] The purpose of this invention is to provide an image conversion system and method between Bayer format and quad Bayer format, so as to solve the problem of poor image conversion effect in existing Bayer format and quad Bayer format conversion methods.

[0005] To solve the above-mentioned technical problems, the present invention provides an image conversion system between Bayer format and quad Bayer format, including an image acquisition module, a de-mosaic module, a de-mosaic module and an image output module; The image acquisition module is used to acquire raw Bayer format images and / or raw quad Bayer format images; The re-mosaic module is used to convert the original four-bayer format image into a Bayer format image, and also to convert the four-bayer format image generated by the de-mosaic module into a Bayer format image; The demosaic module is used to convert the original Bayer format image into a quad Bayer format image, and also to convert the Bayer format image generated by the remosaic module into a quad Bayer format image; The image output module is used to output the four-bayer format image generated by the de-mosaic module and / or the bayer format image generated by the re-mosaic module.

[0006] Optionally, in the image conversion system between Bayer format and quad Bayer format, the re-mosaic module includes a first input convolutional layer, at least one enhanced attention unit, a first output convolutional layer, and a first jump point downsampling unit arranged sequentially. The first input convolutional layer is used to perform convolution processing on the original four-bayer format image to extract the first feature image; At least one of the enhanced attention units is used to extract attention features based on the first feature image; The first output convolutional layer is used to perform residual operations on the first feature image and the attention feature, then perform convolution processing, and then perform residual operations with the original four-bayer format image to obtain the first RGB image; The first jump-point downsampling unit is used to perform jump-point downsampling conversion on the first RGB image to obtain a Bayer format image.

[0007] Optionally, in the image conversion system between the Bayer format and the quad Bayer format, the demosaic module includes a second input convolutional layer, at least one residual sliding window conversion unit, a second output convolutional layer, and a second jump point downsampling unit arranged sequentially. The second input convolutional layer is used to perform convolution processing on the original Bayer format image to extract the second feature image; At least one of the residual sliding window transformation units is used to extract residual features based on the second feature image; The second output convolutional layer is used to perform convolution processing on the residual features, and then perform residual operation with the original Bayer format image to obtain the second RGB image; The second jump-point downsampling unit is used to perform jump-point downsampling conversion on the second RGB image to obtain a four-bayer format image.

[0008] Optionally, in the image conversion system between Bayer format and quad Bayer format, the image conversion system further includes a result evaluation module; the result evaluation module is used to calculate the approximation between the generated Bayer format image and the corresponding original quad Bayer format image to evaluate the quality of the generated Bayer format image; and / or, to calculate the approximation between the generated quad Bayer format image and the corresponding original Bayer format image to evaluate the quality of the generated quad Bayer format image.

[0009] Optionally, in the image conversion system between the Bayer format and the four-Bayer format, the result evaluation module includes a peak signal-to-noise ratio calculation unit, a structural similarity calculation unit, and a cycle consistency constraint unit. The peak signal-to-noise ratio (PSNR) calculation unit is used to calculate the peak SNR between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the peak SNR between the generated four-Bayer format image and the corresponding original Bayer format image; The structural similarity calculation unit is used to calculate the structural similarity between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the structural similarity between the generated four-Bayer format image and the corresponding original Bayer format image; The cycle consistency constraint unit is used to control the reversibility of converting the original four-bayer format image to a Bayer format image, and / or to control the reversibility of converting the original Bayer format image to a four-bayer format image; The result evaluation module evaluates the quality of the generated Bayer format image and / or quad Bayer format image based on the peak signal-to-noise ratio and the structural similarity.

[0010] Optionally, in the image conversion system between the Bayer format and the four-Bayer format, the cycle consistency constraint unit is provided with a cycle consistency loss function, expressed as:

[0011] in, represents the cycle consistency loss value, used to control the reversibility of format conversion; x represents the original Bayer format image, y represents the original quad Bayer format image; G() represents converting the Bayer format image to a quad Bayer format image, and R() represents converting the quad Bayer format image to a Bayer format image; This represents the L1 norm.

[0012] To address the aforementioned technical problems, this invention also provides an image conversion method for converting four-bayer format to Bayer format, implemented using the image conversion system between Bayer format and four-bayer format as described in any of the preceding claims. The image conversion method includes: Obtain the original four-bayer format image; The original quad Bayer format image is converted to a Bayer format image using the re-mosaic module; The demosaic module is used to convert the Bayer format image generated by the heavy mosaic module into a quad Bayer format image. The four-bayer format image generated by the de-mosaic module is sent to the re-mosaic module, and the above steps are repeated a preset number of times to output a Bayer format image.

[0013] Optionally, in the image conversion method for converting four-bayer format to Bayer format, the image conversion method further includes: Calculate the approximation between the generated Bayer format image and the corresponding original four-Bayer format image; The quality of the generated Bayer format image is evaluated based on the calculated approximation.

[0014] To address the aforementioned technical problems, this invention also provides an image conversion method for converting Bayer format to quad Bayer format, implemented using the image conversion system between Bayer format and quad Bayer format as described in any of the preceding claims. The image conversion method includes: Obtain the original Bayer format image; The original Bayer format image is converted to a quad Bayer format image using the demosaic module; Using the re-mosaic module, the quad Bayer format image generated by the de-mosaic module is converted into a Bayer format image; The Bayer format image generated by the re-mosaic module is sent to the de-mosaic module, and the above steps are repeated a preset number of times to output a quad Bayer format image.

[0015] Optionally, in the image conversion method for converting Bayer format to quad Bayer format, the image conversion method further includes: Calculate the approximation between the generated four-bayer format image and the corresponding original bayer format image; The quality of the generated four-Bayer format image is evaluated based on the calculated approximation.

[0016] The image conversion system and method between Bayer format and quad Bayer format provided by this invention includes an image acquisition module, a re-mosaic module, a de-mosaic module, and an image output module. The image acquisition module is used to acquire the original Bayer format image and / or the original quad Bayer format image. The re-mosaic module is used to convert the original quad Bayer format image into a Bayer format image, and also to convert the quad Bayer format image generated by the de-mosaic module into a Bayer format image. The de-mosaic module is used to convert the original Bayer format image into a quad Bayer format image, and also to convert the Bayer format image generated by the re-mosaic module into a quad Bayer format image. The image output module is used to output the quad Bayer format image generated by the de-mosaic module and / or the Bayer format image generated by the re-mosaic module. By unifying the re-mosaic process and the de-mosaic process within the same system loop framework, it is possible not only to better capture the correlation between the Bayer format and the quad Bayer format, but also to effectively reduce information loss during the format conversion process. The repetitive re-mosaic and de-mosaic processes can also enhance each other, improving the conversion accuracy and quality of the image, thus solving the problem of poor image conversion results between the existing Bayer format and the quad Bayer format. Attached Figure Description

[0017] Figure 1 This is a block diagram of the image conversion system between Bayer format and quad Bayer format provided in this embodiment; Figure 2 A flowchart of the image conversion method for converting four Bayer formats to Bayer formats provided in this embodiment; Figure 3 A flowchart of the image conversion method for converting Bayer format to quad Bayer format provided in this embodiment. Detailed Implementation

[0018] The image conversion system and method between Bayer format and quad Bayer format proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, only used to facilitate and clarify the illustration of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different proportions may be used in different drawings to illustrate different aspects.

[0019] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects in order to describe embodiments of the invention, and are not used to describe a specific order or sequence. It should be understood that such uses of terminology are interchangeable where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Currently, deep learning has made progress in image de-mosaic and re-mosaic tasks, but existing methods usually handle the conversion of Bayer format RAW images to RGB images or quad Bayer format RAW images to RGB images separately, and cannot optimize the conversion between the two formats at the same time. This limits image quality improvement and hardware compatibility.

[0021] To address the aforementioned shortcomings of existing methods, this embodiment provides an image conversion system between Bayer format and quad Bayer format, such as... Figure 1 As shown, it includes an image acquisition module, a re-mosaic module, a de-mosaic module, and an image output module; The image acquisition module is used to acquire raw Bayer format images and / or raw quad Bayer format images; The re-mosaic module is used to convert the original four-bayer format image into a Bayer format image, and also to convert the four-bayer format image generated by the de-mosaic module into a Bayer format image; The demosaic module is used to convert the original Bayer format image into a quad Bayer format image, and also to convert the Bayer format image generated by the remosaic module into a quad Bayer format image; The image output module is used to output the four-bayer format image generated by the de-mosaic module and / or the bayer format image generated by the re-mosaic module.

[0022] The image conversion system between Bayer format and quad Bayer format provided in this embodiment unifies the re-mosaic process and the de-mosaic process within the same system loop framework. This not only better captures the correlation between Bayer format and quad Bayer format, but also effectively reduces information loss during the format conversion process. The repeated re-mosaic and de-mosaic processes can also enhance each other, improving the conversion accuracy and quality of the image, thus solving the problem of poor performance in existing image conversion methods between Bayer format and quad Bayer format.

[0023] Furthermore, in this embodiment, the heavy mosaic module R uses a network abstraction to fit the implicit distribution. Conditional distribution for approximating true four-Bayer format RAW images to RGB images The Bayer format RAW image output by the re-mosaic module can be regarded as an implicit distribution. The acquired Bayer format images are then converted into Bayer format image pairs. Where x represents the original Bayer format image and y represents the original quad Bayer format image. This represents the Bayer format image obtained after re-mosaic processing.

[0024] Specifically, in this embodiment, the re-mosaic module includes a first input convolutional layer, at least one enhanced attention unit, a first output convolutional layer, and a first jump point downsampling unit arranged sequentially.

[0025] The first input convolutional layer is used to perform convolution processing on the original four-Bayer format image to extract the first feature image. This process can be represented as follows:

[0026] Where y represents the original four-bayer format image. This represents the convolution process of the first input convolutional layer. This represents the first feature image.

[0027] Furthermore, at least one of the enhanced attention units is used to extract attention features based on the first feature image, a process that can be represented as:

[0028]

[0029] Where i represents the i-th enhanced attention unit; n represents the total number of enhanced attention units, which is a positive integer greater than or equal to 1; This represents the attention feature extracted by the i-th enhanced attention unit; This represents the process by which the i-th enhanced attention unit extracts attention features.

[0030] Furthermore, the first output convolutional layer is used to perform residual operations on the first feature image and the attention feature, then perform convolution processing, and finally perform residual operations with the original four-Bayer format image to obtain the first RGB image. This process can be represented as:

[0031] in, This indicates the convolution process of the first output convolutional layer; "+" indicates the residual operation. This represents the first RGB image obtained corresponding to the original four-bayer format image.

[0032] Furthermore, the first jump-point downsampling unit is used to perform jump-point downsampling conversion on the first RGB image to obtain a Bayer format image, which can be represented as:

[0033] in, This describes the process of obtaining a Bayer format image through jump-point downsampling conversion. This represents the Bayer format image obtained by converting the original four-bayer format image.

[0034] Furthermore, in this embodiment, the demosaic module G generates an RGB image from the original Bayer format RAW image, and then generates a quad-Bayer format image through point sampling. The implicit distribution from the Bayer format image to the quad-Bayer format image is achieved through the demosaic module. To approximate the true distribution Image pairs obtained by the demosaic module are converted from Bayer format images to quad Bayer format images. Where x represents the original Bayer format image and y represents the original quad Bayer format image. This represents the four-bayer format image obtained after mosaic removal.

[0035] Specifically, in this embodiment, the demosaic module includes a second input convolutional layer, at least one residual sliding window conversion unit, a second output convolutional layer, and a second jump point downsampling unit arranged sequentially.

[0036] The second input convolutional layer is used to perform convolution processing on the original Bayer format image to extract the second feature image. This process can be represented as follows:

[0037] Where x represents the original Bayer format image, This indicates the convolution process of the second input convolutional layer. This represents the second feature image.

[0038] Furthermore, at least one of the residual sliding window transformation units is used to extract residual features based on the second feature image, a process that can be represented as:

[0039]

[0040] Where j represents the j-th residual sliding window transformation unit; m represents the total number of residual sliding window transformation units, which is a positive integer greater than or equal to 1; This represents the residual feature extracted by the j-th residual sliding window transformation unit; This represents the process of extracting residual features by the j-th residual sliding window transformation unit.

[0041] Furthermore, the second output convolutional layer is used to perform convolution processing on the residual features, and then perform residual operations with the original Bayer format image to obtain the second RGB image. This process can be represented as follows:

[0042] in, This indicates the convolution process of the second output convolutional layer; "+" indicates the residual operation. This represents the second RGB image obtained corresponding to the original Bayer format image.

[0043] Furthermore, the second jump-point downsampling unit is used to perform jump-point downsampling conversion on the second RGB image to obtain a four-Bayer format image, which can be represented as:

[0044] in, This describes the process of obtaining a four-Bayer format image through jump-point downsampling conversion. This represents the quad Bayer format image obtained by converting the original Bayer format image.

[0045] Furthermore, to better train the re-mosaic module and the de-mosaic module, and to evaluate the quality of the generated quad Bayer format image and the Bayer format image, in this embodiment, the image conversion system further includes a result evaluation module; the result evaluation module is used to calculate the approximation between the generated Bayer format image and the corresponding original quad Bayer format image to evaluate the quality of the generated Bayer format image; and / or, to calculate the approximation between the generated quad Bayer format image and the corresponding original Bayer format image to evaluate the quality of the generated quad Bayer format image.

[0046] Specifically, in this embodiment, the result evaluation module includes a peak signal-to-noise ratio calculation unit, a structural similarity calculation unit, and a cycle consistency constraint unit.

[0047] The peak signal-to-noise ratio (PSNR) calculation unit is used to calculate the PSNR between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the PSNR between the generated four-Bayer format image and the corresponding original Bayer format image. The calculation method for PSNR is well known to those skilled in the art, and will not be elaborated upon here.

[0048] Furthermore, the structural similarity calculation unit is used to calculate the structural similarity SSIM between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the structural similarity SSIM between the generated four-Bayer format image and the corresponding original Bayer format image. The method for calculating structural similarity is well known to those skilled in the art, and will not be elaborated upon here.

[0049] Furthermore, the cycle consistency constraint unit is used to control the reversibility of converting the original four-Bayer format image to a Bayer format image, and / or to control the reversibility of converting the original Bayer format image to a four-Bayer format image. Specifically, in this embodiment, the cycle consistency constraint unit is provided with a cycle consistency loss function, expressed as:

[0050] in, represents the cycle consistency loss value, used to control the reversibility of format conversion; x represents the original Bayer format image, y represents the original quad Bayer format image; G() represents converting the Bayer format image to a quad Bayer format image, and R() represents converting the quad Bayer format image to a Bayer format image; This represents the L1 norm.

[0051] The result evaluation module assesses the quality of the generated Bayer format image and / or quad Bayer format image based on the peak signal-to-noise ratio (PSNR) and the structural similarity. In one specific embodiment, PGap can be defined to evaluate the quality of the generated Bayer format image and / or quad Bayer format image. PGap is a difference index of peak signal-to-noise ratio (PSNR). The smaller the value, the better the image conversion effect and the higher the quality of the generated image.

[0052] For the heavy mosaic module, which is the process of converting a quad Bayer format image to a Bayer format image, the formula can be expressed as:

[0053]

[0054] in, This represents a test dataset with a sample size of S. This represents a re-mosaic module trained on a real dataset. This represents the remosaic module trained on the generated dataset.

[0055] A real dataset can be represented as:

[0056] The generated dataset can be represented as:

[0057] Similarly, the specific calculation formula for PGap used in the de-mosaic module, i.e., the process of converting a Bayer format image to a four-Bayer format image, can be obtained, which will not be elaborated here.

[0058] The total loss function of the image conversion system between the Bayer format and the four-Bayer format provided in this embodiment can be expressed as:

[0059] in, This represents the total loss value, used to comprehensively measure the quality of the transformed image and cycle consistency. This represents the peak signal-to-noise ratio loss, used to measure the pixel-level similarity between the generated image and the real image. A higher value indicates better image quality after conversion. It represents structural similarity loss and is used to measure the structural, brightness, and contrast similarity between the generated image and the real image. This represents the cycle consistency loss value, ensuring the reversibility of format conversion and avoiding conversion distortion; This represents the weight parameter, which needs to be adjusted based on the training data. Its value range can be 0 to 2, and it is used to balance the importance of cycle consistency loss and image quality loss.

[0060] This embodiment also provides an image conversion method for converting four-Bayer format to Bayer format, which is implemented using the image conversion system between Bayer format and four-Bayer format as described above. The image conversion method includes: A1, obtain the original four-bayer format image.

[0061] Specifically, in this embodiment, the original four-bayer format image can be acquired through the image acquisition module.

[0062] A2 uses the heavy mosaic module to convert the original four-bayer format image into a bayer format image.

[0063] Specifically, in this embodiment, the heavy mosaic module R uses a network abstraction to fit the implicit distribution. Conditional distribution for approximating true four-Bayer format RAW images to RGB images The Bayer format RAW image output by the re-mosaic module can be regarded as an implicit distribution. The acquired Bayer format images are then converted into Bayer format image pairs. .

[0064] A3 uses the de-mosaic module to convert the Bayer format image generated by the heavy mosaic module into a quad Bayer format image.

[0065] Specifically, in this embodiment, the demosaic module G generates an RGB image from a Bayer format RAW image and then generates a quad-Bayer format image through point sampling. The implicit distribution from the Bayer format image to the quad-Bayer format image is achieved through the demosaic module. To approximate the true distribution Image pairs obtained by the demosaic module are converted from Bayer format images to quad Bayer format images. .

[0066] A4. Send the four Bayer format image generated by the de-mosaic module into the re-mosaic module, and repeat the above steps a preset number of times to output a Bayer format image.

[0067] That is, repeat steps A2 to A3 a preset number of times to cyclically convert the image between Bayer format and quad Bayer format, thereby better capturing the correlation between the two formats, effectively reducing information loss during the conversion process, and mutually enhancing each other to improve conversion accuracy and image quality. Finally, output the Bayer format image corresponding to the original quad Bayer format image after conversion.

[0068] Preferably, in order to better train the re-mosaic module and the de-mosaic module, and in order to evaluate the quality of the generated Bayer format image, in this embodiment, the image conversion method further includes: calculating the approximation between the generated Bayer format image and the corresponding original four-Bayer format image; and evaluating the quality of the generated Bayer format image based on the calculated approximation.

[0069] Furthermore, this embodiment also provides an image conversion method for converting Bayer format to quad Bayer format, implemented using the image conversion system between Bayer format and quad Bayer format as described above. The image conversion method includes: B1, retrieve the original Bayer format image.

[0070] Specifically, in this embodiment, the original Bayer format image can be acquired through the image acquisition module.

[0071] B2 uses the de-mosaic module to convert the original Bayer format image into a quad Bayer format image.

[0072] Specifically, in this embodiment, the demosaic module G generates an RGB image from the original Bayer format RAW image, and then generates a quad-Bayer format image through point sampling. The implicit distribution from the Bayer format image to the quad-Bayer format image is obtained through the demosaic module. To approximate the true distribution Image pairs obtained by the demosaic module are converted from Bayer format images to quad Bayer format images. Where x represents the original Bayer format image and y represents the original quad Bayer format image. This represents the four-bayer format image obtained after mosaic removal.

[0073] B3 uses the re-mosaic module to convert the quad Bayer format image generated by the de-mosaic module into a Bayer format image.

[0074] Specifically, in this embodiment, the heavy mosaic module R uses a network abstraction to fit the implicit distribution. Conditional distribution for approximating true four-Bayer format RAW images to RGB images The Bayer format RAW image output by the re-mosaic module can be regarded as an implicit distribution. The acquired Bayer format images are then converted into Bayer format image pairs. .

[0075] B4. Send the Bayer format image generated by the re-mosaic module into the de-mosaic module, and repeat the above steps a preset number of times to output a four-bayer format image.

[0076] That is, repeating steps B2 to B3 a preset number of times to cyclically convert the image between the four-bay format and the Bayer format, thereby better capturing the correlation between the two formats, effectively reducing information loss during the conversion process, and mutually enhancing each other to improve conversion accuracy and image quality. Finally, the four-bay format image corresponding to the original Bayer format image is output.

[0077] Preferably, in order to better train the re-mosaic module and the de-mosaic module, and in order to evaluate the quality of the generated four-Bayer format image, in this embodiment, the image conversion method further includes: calculating the approximation between the generated four-Bayer format image and the corresponding original Bayer format image; and evaluating the quality of the generated four-Bayer format image based on the calculated approximation.

[0078] The image conversion system and method between Bayer and Quad Bayer formats provided in this embodiment can generate Quad Bayer images from input Bayer images and vice versa, further improving the model's conversion capability and image quality. The unified framework not only better captures the correlation between the two formats but also effectively reduces information loss during the conversion process. The method of this invention has broad application prospects in fields such as mobile photography, medical imaging, and autonomous driving, providing a new and innovative approach to image sensor data processing.

[0079] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. In addition, different parts between embodiments can also be combined with each other, and this invention does not limit this.

[0080] The image conversion system and method between Bayer format and quad Bayer format provided in this embodiment include an image acquisition module, a re-mosaic module, a de-mosaic module, and an image output module. The image acquisition module is used to acquire the original Bayer format image and / or the original quad Bayer format image. The re-mosaic module is used to convert the original quad Bayer format image into a Bayer format image, and also to convert the quad Bayer format image generated by the de-mosaic module into a Bayer format image. The de-mosaic module is used to convert the original Bayer format image into a quad Bayer format image, and also to convert the Bayer format image generated by the re-mosaic module into a quad Bayer format image. The image output module is used to output the quad Bayer format image generated by the de-mosaic module and / or the Bayer format image generated by the re-mosaic module. By unifying the re-mosaic process and the de-mosaic process within the same system loop framework, it is possible not only to better capture the correlation between the Bayer format and the quad Bayer format, but also to effectively reduce information loss during the format conversion process. The repetitive re-mosaic and de-mosaic processes can also enhance each other, improving the conversion accuracy and quality of the image, thus solving the problem of poor image conversion results between the existing Bayer format and the quad Bayer format.

[0081] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. An image conversion system between Bayer format and quad Bayer format, characterized in that, It includes an image acquisition module, a re-mosaic module, a de-mosaic module, and an image output module; The image acquisition module is used to acquire raw Bayer format images and / or raw quad Bayer format images; The re-mosaic module is used to convert the original four-bayer format image into a Bayer format image, and also to convert the four-bayer format image generated by the de-mosaic module into a Bayer format image; The demosaic module is used to convert the original Bayer format image into a quad Bayer format image, and also to convert the Bayer format image generated by the remosaic module into a quad Bayer format image; The image output module is used to output the four-bayer format image generated by the de-mosaic module and / or the bayer format image generated by the re-mosaic module.

2. The image conversion system between Bayer format and quad Bayer format according to claim 1, characterized in that, The heavy mosaic module includes a first input convolutional layer, at least one enhanced attention unit, a first output convolutional layer, and a first jump point downsampling unit arranged sequentially. The first input convolutional layer is used to perform convolution processing on the original four-bayer format image to extract the first feature image; At least one of the enhanced attention units is used to extract attention features based on the first feature image; The first output convolutional layer is used to perform residual operations on the first feature image and the attention feature, then perform convolution processing, and then perform residual operations with the original four-bayer format image to obtain the first RGB image; The first jump-point downsampling unit is used to perform jump-point downsampling conversion on the first RGB image to obtain a Bayer format image.

3. The image conversion system between Bayer format and quad Bayer format according to claim 1, characterized in that, The demosaic module includes a second input convolutional layer, at least one residual sliding window transformation unit, a second output convolutional layer, and a second jump point downsampling unit arranged sequentially. The second input convolutional layer is used to perform convolution processing on the original Bayer format image to extract the second feature image; At least one of the residual sliding window transformation units is used to extract residual features based on the second feature image; The second output convolutional layer is used to perform convolution processing on the residual features, and then perform residual operation with the original Bayer format image to obtain the second RGB image; The second jump-point downsampling unit is used to perform jump-point downsampling conversion on the second RGB image to obtain a four-bayer format image.

4. The image conversion system between Bayer format and quad Bayer format according to claim 1, characterized in that, The image conversion system further includes a result evaluation module; the result evaluation module is used to calculate the approximation between the generated Bayer format image and the corresponding original four-Bayer format image to evaluate the quality of the generated Bayer format image; and / or, to calculate the approximation between the generated four-Bayer format image and the corresponding original Bayer format image to evaluate the quality of the generated four-Bayer format image.

5. The image conversion system between Bayer format and quad Bayer format according to claim 4, characterized in that, The result evaluation module includes a peak signal-to-noise ratio calculation unit, a structural similarity calculation unit, and a cycle consistency constraint unit; The peak signal-to-noise ratio (PSNR) calculation unit is used to calculate the peak SNR between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the peak SNR between the generated four-Bayer format image and the corresponding original Bayer format image; The structural similarity calculation unit is used to calculate the structural similarity between the generated Bayer format image and the corresponding original four-Bayer format image, and / or to calculate the structural similarity between the generated four-Bayer format image and the corresponding original Bayer format image; The cycle consistency constraint unit is used to control the reversibility of converting the original four-bayer format image to a Bayer format image, and / or to control the reversibility of converting the original Bayer format image to a four-bayer format image; The result evaluation module evaluates the quality of the generated Bayer format image and / or quad Bayer format image based on the peak signal-to-noise ratio and the structural similarity.

6. The image conversion system between Bayer format and quad Bayer format according to claim 5, characterized in that, The cycle consistency constraint unit is configured with a cycle consistency loss function, expressed as: in, represents the cycle consistency loss value, used to control the reversibility of format conversion; x represents the original Bayer format image, y represents the original quad Bayer format image; G() represents converting the Bayer format image to a quad Bayer format image, and R() represents converting the quad Bayer format image to a Bayer format image; This represents the L1 norm.

7. A method for converting images from quad Bayer format to Bayer format, implemented using the image conversion system between Bayer format and quad Bayer format as described in any one of claims 1 to 6, characterized in that, The image conversion method includes: Obtain the original four-bayer format image; The original quad Bayer format image is converted to a Bayer format image using the re-mosaic module; The demosaic module is used to convert the Bayer format image generated by the heavy mosaic module into a quad Bayer format image. The four-bayer format image generated by the de-mosaic module is sent to the re-mosaic module, and the above steps are repeated a preset number of times to output a Bayer format image.

8. The image conversion method for converting four-bayer format to Bayer format according to claim 7, characterized in that, The image conversion method further includes: Calculate the approximation between the generated Bayer format image and the corresponding original four-Bayer format image; The quality of the generated Bayer format image is evaluated based on the calculated approximation.

9. A method for converting a Bayer format image to a quad Bayer format image, implemented using the image conversion system between Bayer and quad Bayer formats as described in any one of claims 1 to 6, characterized in that, The image conversion method includes: Obtain the original Bayer format image; The original Bayer format image is converted to a quad Bayer format image using the demosaic module; Using the re-mosaic module, the quad Bayer format image generated by the de-mosaic module is converted into a Bayer format image; The Bayer format image generated by the re-mosaic module is sent to the de-mosaic module, and the above steps are repeated a preset number of times to output a quad Bayer format image.

10. The image conversion method for converting Bayer format to quad Bayer format according to claim 9, characterized in that, The image conversion method further includes: Calculate the approximation between the generated four-bayer format image and the corresponding original bayer format image; The quality of the generated four-Bayer format image is evaluated based on the calculated approximation.

Images