Four-bayer image conversion method and four-bayer image conversion apparatus
By optimizing the green channel interpolation using a three-level convolutional neural network and combining it with traditional interpolation methods to process the red and blue channels, the contradiction between computational resources and image quality in four-bayer image conversion is resolved, achieving efficient image conversion results.
Patent Information
- Application Number
- CN202610412543.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-03
Smart Images
Figure CN122340368A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a four-bay image conversion method and a four-bay image conversion device. Background Technology
[0002] Quad Bayer sensors are increasingly becoming the widely used pixel arrangement in CMOS image sensors for modern smartphones, cameras, security systems, and other fields. In non-low-light scenarios or when maintaining the original resolution is required, the quad Bayer image must be converted into a regular Bayer image of the same size before subsequent signal processing can proceed.
[0003] Traditional four-Bayer image conversion methods are based on simple interpolation, which results in certain image quality defects. Some improved four-Bayer image conversion methods suffer from high computational resource consumption. Summary of the Invention
[0004] The purpose of this invention is to provide a four-Bayer image conversion method and a four-Bayer image conversion device to solve the problem of balancing computational resource consumption and image quality.
[0005] To solve the above-mentioned technical problems, the present invention provides a four-Bayer image conversion method, which includes:
[0006] Extract the first buffer window centered on a 2×2 pixel point in the red or blue channel of the four-bay image;
[0007] Based on the first cache window, calculate the green channel interpolation estimation results in the horizontal and vertical directions, and calculate the gradient estimation results in the horizontal and vertical directions.
[0008] The green channel interpolation estimation results are input into the first convolutional neural network for processing to obtain the green channel interpolation results in the horizontal and vertical directions.
[0009] The gradient estimation results are input into the second convolutional neural network for processing to obtain the gradient results in the horizontal and vertical directions;
[0010] Based on the gradient results, the green channel interpolation results are weighted and fused to obtain the initial green interpolation fusion result;
[0011] The initial green interpolation fusion result is combined with the green channel value in the four-bay image and input into the third convolutional neural network for processing to obtain the final green interpolation result.
[0012] Based on the final green interpolation result and the four-bay image, the interpolation results for the red and blue channels are calculated using an interpolation method.
[0013] Based on the target Bayer image pattern, select the corresponding interpolated pixels or the original pixels in the four-bayer image to obtain the converted ordinary Bayer image.
[0014] Optionally, the first convolutional neural network and the second convolutional neural network include:
[0015] The horizontal preprocessing convolutional layer increases the dimensionality of the one-dimensional horizontal estimation result.
[0016] Vertical preprocessing convolutional layers increase the dimensionality of the one-dimensional vertical estimation results;
[0017] The horizontal residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the horizontal estimation result after dimensionality increase, and the output is the horizontal feature.
[0018] The vertical residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the vertical estimation result after dimensionality increase, and the output is the vertical feature.
[0019] The horizontal and vertical feature fusion module performs channel splicing on the horizontal and vertical features to obtain the horizontal and vertical fused features.
[0020] A feature downsampling layer downsamples the horizontal and vertical fused features;
[0021] The residual blocks of the downsampled features are densely connected, consisting of multiple connected residual blocks;
[0022] The feature upsampling layer upsamples the downsampled residual output.
[0023] Feature residual connection: The horizontal and vertical fused features are residually connected with the upsampling results.
[0024] Optionally, the third convolutional neural network includes:
[0025] The green channel fusion layer merges the initial green interpolation fusion result with the green channel value in the four-bay image to obtain the initial green channel fusion result, and then uses a convolutional layer to increase the dimensionality of the initial green channel fusion result;
[0026] The green channel downsampling layer is used to spatially downsample the fusion result of the up-dimensional green channel;
[0027] The dense connection of the green channel residual blocks, consisting of multiple residual blocks, is used to perform deep feature enhancement on the downsampled green channel fusion result to obtain the green channel fusion feature.
[0028] A green channel upsampling layer is used to spatially upsample the fused features of the green channel.
[0029] The green channel residual connection is performed by connecting the upsampled green channel fusion features with the green channel fusion result after dimensionality upgrade of the green channel fusion layer to obtain the final green channel fusion result.
[0030] Optionally, the steps of calculating the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window include:
[0031] Based on the 2×2 pixels of the first cache window, green pixels are found in the horizontal and vertical directions respectively, and the green low-frequency interpolation result is obtained by weighting some green pixels in the horizontal and vertical directions.
[0032] Based on the channel color of the first cache window, and based on the 2×2 pixels of the first cache window, interpolation points corresponding to the channel color are found in the horizontal and vertical directions respectively, and weighted to obtain the low-frequency interpolation result corresponding to the channel color.
[0033] Subtract the low-frequency interpolation result corresponding to the channel color from the 2×2 pixels of the first cache window to obtain the high-frequency interpolation result corresponding to the channel color.
[0034] The low-frequency interpolation results of the green channel and the high-frequency interpolation results corresponding to the channel colors are weighted and fused to obtain the green channel interpolation estimation results in the horizontal and vertical directions.
[0035] Optionally, the step of calculating the gradient estimation results in the horizontal and vertical directions based on the first cache window includes:
[0036] Based on the 2×2 pixels of the first cache window, a subtraction operation is performed on some green pixels in the horizontal and vertical directions to calculate the green difference in the horizontal and vertical directions.
[0037] Based on the channel color of the first cache window, and based on the 2×2 pixels of the first cache window, the pixels corresponding to the channel color in the horizontal and vertical directions are subtracted to calculate the difference between the horizontal and vertical channel colors.
[0038] The green difference and the difference of the corresponding channel color are weighted and fused, and the absolute value is taken to obtain the gradient estimation results in the horizontal and vertical directions.
[0039] Optionally, the step of weighted fusion of the green channel interpolation results based on the gradient results to obtain an initial green interpolation fusion result includes:
[0040] The horizontal green channel interpolation result is weighted by taking the reciprocal of the gradient result plus 1 in the horizontal direction.
[0041] The vertical green channel interpolation results are weighted by taking the inverse of the gradient result plus 1 in the vertical direction as the vertical weight.
[0042] The initial green interpolation fusion result is the weighted green channel interpolation result divided by the weight.
[0043] Optionally, based on the final green interpolation result and the four-bay image, the step of calculating the interpolation results for the red and blue channels using interpolation includes:
[0044] Extract a second buffer window centered on a 2×2 pixel point of the same color channel in the four-bay image. If the second buffer window is the green channel, calculate the average color difference in the horizontal and vertical directions based on the final green interpolation result.
[0045] The red channel interpolation result and blue channel interpolation result of the 2×2 pixel point of the second cache window are obtained by subtracting the original channel value of the second cache window from the average color difference in the horizontal and vertical directions, respectively.
[0046] Optionally, based on the final green interpolation result and the four-bay image, the step of calculating the interpolation results for the red and blue channels using interpolation includes:
[0047] A second buffer window is extracted centered on a 2×2 pixel point of the same color channel in the four-bay image, with red and blue as a group of dissimilar colors; if the second buffer window is a red channel or a blue channel, the average color difference of the dissimilar colors around the second buffer window is calculated based on the final green interpolation result;
[0048] The interpolation result of the opposite color channel of the second cache window is obtained by subtracting the final green interpolation result of the 2×2 pixel point of the second cache window from the mean color difference of the opposite color.
[0049] Optionally, the training steps for the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network include:
[0050] Build the training dataset;
[0051] In the individual training phase, the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network are trained independently, respectively; while training any one convolutional neural network, the other two convolutional neural networks are set to pass-through mode.
[0052] In the joint fine-tuning phase, the parameters of the three convolutional neural networks obtained in the individual training phase are loaded into the corresponding networks, and the first, second, and third convolutional neural networks are simultaneously enabled for training, and the parameters of the three convolutional neural networks are jointly updated.
[0053] To solve the above-mentioned technical problems, the present invention also provides a four-bayer image conversion device, which includes: a buffer window extraction unit, a green channel interpolation estimation unit, a gradient estimation unit, a first convolutional neural network, a second convolutional neural network, a green channel interpolation weighted fusion unit, a third convolutional neural network, a red and blue channel interpolation unit, and a conversion output unit.
[0054] The cache window extraction unit extracts the first cache window centered on a 2×2 pixel point in the red or blue channel of the four Bayer image;
[0055] The green channel interpolation estimation unit calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window;
[0056] The gradient estimation unit calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window, and calculates the gradient estimation results in the horizontal and vertical directions.
[0057] The first convolutional neural network takes the green channel interpolation estimation result as input, processes it, and obtains the green channel interpolation results in the horizontal and vertical directions.
[0058] The second convolutional neural network takes the gradient estimation result as input, processes it, and obtains the gradient results in the horizontal and vertical directions.
[0059] The green channel interpolation weighted fusion unit performs weighted fusion on the green channel interpolation results based on the gradient results to obtain the initial green interpolation fusion results;
[0060] The initial green interpolation fusion result after being merged and input into the third convolutional neural network are processed together with the green channel value in the four-bay image to obtain the final green interpolation result.
[0061] The red and blue channel interpolation unit calculates the interpolation results of the red and blue channels based on the final green interpolation result and the four-bay image using an interpolation method.
[0062] The conversion output unit selects the corresponding interpolated pixels or the original pixels in the four-bay image according to the target Bayer image pattern to obtain the converted ordinary Bayer image.
[0063] In summary, in the four-Bayer image conversion method and apparatus provided by this invention, the four-Bayer image conversion method includes: extracting a first buffer window centered on a 2×2 pixel point of the red or blue channel in the four-Bayer image; calculating the green channel interpolation estimation results in the horizontal and vertical directions based on the first buffer window, and calculating the gradient estimation results in the horizontal and vertical directions; inputting the green channel interpolation estimation results into a first convolutional neural network for processing to obtain the green channel interpolation results in the horizontal and vertical directions; and inputting the gradient estimation results into a second convolutional neural network for processing. The process involves: obtaining gradient results in the horizontal and vertical directions; weighting and fusing the green channel interpolation results based on the gradient results to obtain an initial green interpolation fusion result; merging the initial green interpolation fusion result with the green channel values in the four-bay image and inputting it into a third convolutional neural network for processing to obtain the final green interpolation result; calculating the interpolation results of the red and blue channels based on the final green interpolation result and the four-bay image using interpolation methods; and selecting the corresponding interpolation pixels or original pixels from the four-bay image according to the target Bayer image pattern to obtain the converted ordinary Bayer image.
[0064] This configuration, by employing a three-level neural network to optimize the key interpolation stage of the green channel, and then using the optimized green channel in conjunction with the red and blue channels for efficient traditional interpolation, significantly improves image quality while concentrating neural network computing resources on the high-yield key interpolation stage of the green channel, effectively resolving the contradiction between quality and efficiency. Attached Figure Description
[0065] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention.
[0066] Figure 1 This is a schematic diagram of a four-bayer image containing 10×10 pixels, according to an embodiment of the present invention.
[0067] Figure 2 This is a schematic flowchart of the four-Bayer image conversion method according to an embodiment of the present invention.
[0068] Figure 3 This is a schematic diagram of the first convolutional neural network and the second convolutional neural network according to an embodiment of the present invention.
[0069] Figure 4 This is a schematic diagram of the third convolutional neural network according to an embodiment of the present invention.
[0070] Figure 5 This is a schematic diagram of an embodiment of the present invention where the second buffer window is a green channel and the horizontally adjacent pixels are red.
[0071] Figure 6 This is a schematic diagram of an embodiment of the present invention where the second buffer window is a green channel and the horizontally adjacent pixels are blue.
[0072] Figure 7 This is a schematic diagram of the second cache window being the red channel in an embodiment of the present invention.
[0073] Figure 8 This is a schematic diagram of the second cache window being the blue channel in an embodiment of the present invention.
[0074] Figure 9 This is a schematic diagram of a four-bayer image conversion device according to an embodiment of the present invention. Detailed Implementation
[0075] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clarify the explanation of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.
[0076] As used in this invention, the singular forms “a,” “an,” “one,” and “the” include plural objects; the term “or” is generally used to mean “and / or”; the term “a number” is generally used to mean “at least one”; and the term “at least two” is generally used to mean “two or more”. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature; “one end” and “the other end,” and “proximal end” and “distal end” generally refer to two corresponding parts, which include not only endpoints. Furthermore, the terms "installed," "connected," and "attached," as used in this invention, and the term "set" on one element from another, should be interpreted broadly. They generally only indicate a connection, coupling, cooperation, or transmission relationship between the two elements, which can be direct or indirect through an intermediate element. They should not be construed as indicating or implying a spatial relationship between the two elements, meaning one element can be located inside, outside, above, below, or to one side of another element, unless otherwise explicitly stated. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Additionally, directional terms such as above, below, up, down, upward, downward, left, and right are used relative to exemplary embodiments as shown in the figures, with upward or upper directions pointing towards the top of the corresponding figure, and downward or lower directions pointing towards the bottom of the corresponding figure.
[0077] The purpose of this invention is to provide a four-Bayer image conversion method and a four-Bayer image conversion apparatus to solve the problem of balancing computational resource consumption and image quality. The following description refers to the accompanying drawings.
[0078] Please refer to Figure 1 The example shown is a 10×10 pixel quad Bayer image containing several pixels of three colors: R (red), G (green), and B (blue). Each color is grouped into 2×2 pixels, and then the groups are repeated in an RGGB pattern. Quad Bayer images can be obtained from raw raw image data acquired from various CMOS image sensors using the quad Bayer format.
[0079] In low-light scenes, each group of 2×2 pixels can be combined into one large pixel, effectively creating a larger light-sensitive area by sacrificing resolution, thereby increasing light intake and reducing noise. However, in non-low-light scenes, or when it is necessary to maintain the original resolution, it is necessary to convert the arrangement of all pixels to the arrangement of a normal Bayer image, that is, to repeat the RGGB arrangement in the form of each pixel.
[0080] The inventors discovered that, based on the physiological characteristics of the human eye, it has a relatively high sensitivity to green, and the green channel dominates in Bayer images. In a standard Bayer array, the number of green pixels is twice that of red and blue pixels (in an RGGB arrangement: 50% G, 25% R, 25% B). Therefore, the green channel contributes the most to the brightness and spatial resolution of the image, and the accuracy of green interpolation directly affects the final image's resolution, edge sharpness, and detail retention. Simultaneously, the results of green interpolation also influence the accuracy of red and blue interpolation, forming a correlation.
[0081] Based on the above research, to convert a four-bay image into a regular Bayer image, please refer to... Figure 2 This invention provides a four-Bayer image conversion method, which includes:
[0082] Step S1: Extract the first buffer window centered on a 2×2 pixel point of the red or blue channel in the four-bay image;
[0083] Step S2: Based on the first cache window, calculate the green channel interpolation estimation results in the horizontal and vertical directions, and calculate the gradient estimation results in the horizontal and vertical directions;
[0084] Step S3: Input the green channel interpolation estimation result into the first convolutional neural network for processing to obtain the green channel interpolation results in the horizontal and vertical directions;
[0085] Step S4: Input the gradient estimation result into the second convolutional neural network for processing to obtain the gradient results in the horizontal and vertical directions;
[0086] Step S5: Based on the gradient results, perform weighted fusion on the green channel interpolation results to obtain the initial green interpolation fusion result;
[0087] Step S6: Combine the initial green interpolation fusion result with the green channel value in the four-bayer image, and input it into the third convolutional neural network for processing to obtain the final green interpolation result;
[0088] Step S7: Based on the final green interpolation result and the four-bay image, calculate the interpolation results for the red and blue channels using the interpolation method;
[0089] Step S8: Based on the target Bayer image pattern, select the corresponding interpolated pixel or the original pixel in the four-bayer image to obtain the converted ordinary Bayer image.
[0090] The following is combined Figure 1 The four Bayer images shown illustrate the steps described above.
[0091] In step S1, a local window is first extracted as the first buffer window, centered on a 2×2 pixel group of red or blue in the four-bay image, focusing on the key area where green needs to be interpolated. The following section... Figure 1 Centered on the 2×2 pixel group of the red channel R44, R45, R54, and R55, we take it as an example to extract the first buffer window.
[0092] Step S2, which calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window, may optionally include three sub-steps:
[0093] Step S21: Calculate the green low-frequency interpolation result.
[0094] Step S22: Calculate the high-frequency interpolation result for red or blue.
[0095] Step S23: Weighted fusion of green low-frequency interpolation results and red or blue high-frequency interpolation results to obtain green interpolation estimation results.
[0096] Step S21 can be based on the 2×2 pixels of the first cache window to find green pixels in the horizontal and vertical directions respectively, and weight the green pixels in the horizontal and vertical directions to obtain the green low-frequency interpolation result.
[0097] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the horizontal green low-frequency interpolation results G44_hor_lp, G45_hor_lp, G54_hor_lp, and G55_hor_lp at positions R44, R45, R54, and R55 are calculated as follows:
[0098] G44_hor_lp = (G42+G46) / 2.
[0099] G45_hor_lp = (G43+G47) / 2.
[0100] G54_hor_lp = (G52+G56) / 2.
[0101] G55_hor_lp = (G53+G57) / 2.
[0102] The green low-frequency interpolation results G44_ver_lp, G45_ver_lp, G54_ver_lp, and G55_ver_lp in the vertical direction at positions R44, R45, R54, and R55 are calculated as follows:
[0103] G44_ver_lp = (G24+G64) / 2.
[0104] G45_ver_lp = (G25+G65) / 2.
[0105] G54_ver_lp = (G34+G74) / 2.
[0106] G55_ver_lp = (G35+G75) / 2.
[0107] Step S22 involves finding partial interpolation points corresponding to the channel color in the horizontal and vertical directions based on the 2×2 pixels of the first cache window, and weighting these points to obtain a low-frequency interpolation result corresponding to the channel color. Then, the low-frequency interpolation result corresponding to the channel color is subtracted from the 2×2 pixels of the first cache window to obtain a high-frequency interpolation result corresponding to the channel color.
[0108] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the channel color of the first buffer window is red. Step S22 finds the red interpolation points around the first buffer window, and the weighted result is the red low-frequency interpolation result, which in turn yields the red high-frequency interpolation result.
[0109] The horizontal red low-frequency interpolation results R44_hor_lp, R45_hor_lp, R54_hor_lp, and R55_hor_lp at positions R44, R45, R54, and R55 are calculated as follows:
[0110] R44_hor_lp = (R40+2R44+R48) / 4.
[0111] R45_hor_lp = (R41+2R45+R49) / 4.
[0112] R54_hor_lp = (R50+2R54+R58) / 4.
[0113] R55_hor_lp = (R51+2R55+R59) / 4.
[0114] Then, by subtracting the corresponding low-frequency red interpolation results from R44, R45, R54, and R55 respectively, we can obtain the horizontal high-frequency red interpolation results R44_hor_hp, R45_hor_hp, R54_hor_hp, and R55_hor_hp at positions R44, R45, R54, and R55 respectively.
[0115] R44_hor_hp = (2*R44-R40-R48) / 4.
[0116] R45_hor_hp = (2*R45-R41-R49) / 4.
[0117] R54_hor_hp = (2*R54-R50-R58) / 4.
[0118] R55_hor_hp = (2*R55-R51-R59) / 4.
[0119] Similarly, the high-frequency red interpolation results in the vertical direction at positions R44, R45, R54, and R55 are obtained as follows: R44_ver_hp, R45_ver_hp, R54_ver_hp, R55_ver_hp.
[0120] R44_ver_hp = (2*R44-R4-R84) / 4.
[0121] R45_ver_hp = (2*R45-R5-R85) / 4.
[0122] R54_ver_hp = (2*R54-R14-R94) / 4.
[0123] R55_ver_hp = (2*R55-R15-R95) / 4.
[0124] Step S23 involves weighted fusion of the low-frequency green interpolation result and the high-frequency interpolation result corresponding to the channel color to obtain the green channel interpolation estimation results in the horizontal and vertical directions. During weighted fusion, a fusion coefficient `merge_w` can be set for the high-frequency interpolation result of the corresponding channel color. In an example, this fusion coefficient `merge_w` can be configured to 1.
[0125] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the channel color of the first buffer window is red at this time. The weighted fusion in step S23 is the green low-frequency interpolation result and the red high-frequency interpolation result.
[0126] The horizontal green channel interpolation estimation results G44_hor, G45_hor, G54_hor, and G55_hor at positions R44, R45, R54, and R55 are calculated as follows:
[0127] G44_hor = G44_hor_lp + merge_w×R44_hor_hp.
[0128] G45_hor = G45_hor_lp + merge_w×R45_hor_hp.
[0129] G54_hor = G54_hor_lp + merge_w×R54_hor_hp.
[0130] G55_hor = G55_hor_lp + merge_w×R55_hor_hp.
[0131] The vertical green channel interpolation estimation results G44_ver, G45_ver, G54_ver, and G55_ver at positions R44, R45, R54, and R55 are calculated as follows:
[0132] G44_ver = G44_ver_lp + merge_w×R44_ver_hp.
[0133] G45_ver = G45_ver_lp + merge_w×R45_ver_hp.
[0134] G54_ver = G54_ver_lp + merge_w×R54_ver_hp.
[0135] G55_ver = G55_ver_lp + merge_w×R55_ver_hp.
[0136] As is easily understood, if the channel color of the first buffer window extracted in step S1 is blue, step S22 searches for blue interpolation points around the first buffer window, and the weighted result is the low-frequency blue interpolation result, which in turn yields the high-frequency blue interpolation result. Then, step S23 weightedly fuses the low-frequency green interpolation result and the high-frequency blue interpolation result, which also yields the horizontal and vertical green channel interpolation estimation results. The principle is similar to the demonstration shown above, and will not be repeated here.
[0137] Step S2, which calculates the gradient estimation results in the horizontal and vertical directions based on the first cache window, may optionally include three sub-steps:
[0138] Step S24: Calculate the difference between the green pixels in the horizontal and vertical directions.
[0139] Step S25: Calculate the difference between the red or blue pixels in the horizontal and vertical directions.
[0140] Step S26: Weighted fusion of the two difference results and taking the absolute value to obtain the gradient value.
[0141] Step S24 can be based on the 2×2 pixels of the first cache window, and perform a subtraction operation on some green pixels in the horizontal and vertical directions to calculate the green difference in the horizontal and vertical directions.
[0142] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the differences in the horizontal direction of the green pixels at positions R44, R45, R54, and R55, G44_hor_diff, G45_hor_diff, G54_hor_diff, and G55_hor_diff, are calculated as follows:
[0143] G44_hor_diff = (G42-G46) / 2.
[0144] G45_hor_diff = (G43-G47) / 2.
[0145] G54_hor_diff = (G52-G56) / 2.
[0146] G55_hor_diff = (G53-G57) / 2.
[0147] The vertical green pixel differences G44_ver_diff, G45_ver_diff, G54_ver_diff, and G55_ver_diff at positions R44, R45, R54, and R55 are calculated as follows:
[0148] G44_ver_diff = (G24-G64) / 2.
[0149] G45_ver_diff = (G25-G65) / 2.
[0150] G54_ver_diff = (G34-G74) / 2.
[0151] G55_ver_diff = (G35-G75) / 2.
[0152] Step S25 can calculate the difference between the horizontal and vertical corresponding channel colors based on the channel color of the first cache window and the 2×2 pixels of the first cache window.
[0153] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the channel color of the first buffer window is red. Step S25 is to perform a subtraction operation on the red pixels in the horizontal and vertical directions around the first buffer window to calculate the difference between the red pixels in the horizontal and vertical directions.
[0154] The horizontal red differences R44_hor_diff, R45_hor_diff, R54_hor_diff, and R55_hor_diff at positions R44, R45, R54, and R55 are calculated as follows:
[0155] R44_hor_diff = (R40-R48) / 4.
[0156] R45_hor_diff = (R41-R49) / 4.
[0157] R54_hor_diff = (R50-R58) / 4.
[0158] R55_hor_diff = (R51-R59) / 4.
[0159] The vertical red differences R44_ver_diff, R45_ver_diff, R54_ver_diff, and R55_ver_diff at positions R44, R45, R54, and R55 are calculated as follows:
[0160] R44_ver_diff = (R4-R84) / 4.
[0161] R45_ver_diff = (R5-R85) / 4.
[0162] R54_ver_diff = (R14-R94) / 4.
[0163] R55_ver_diff = (R15-R95) / 4.
[0164] Step S26 involves weighted fusion of the green difference and the difference of the corresponding channel color, taking the absolute value to obtain the gradient estimation results in the horizontal and vertical directions. During weighted fusion, a gradient weighting coefficient grad_w can be set for the difference of the corresponding channel color. In an example, this gradient weighting coefficient grad_w can be configured to 0.5.
[0165] Taking the red channels R44, R45, R54, and R55 extracted in step S1 as an example of the first buffer window, the channel color of the first buffer window is red. Step S26 performs weighted fusion of the green difference and the red difference. If the channel color of the first buffer window is blue, the principle is similar, and the same example can be used.
[0166] The gradient estimation results in the horizontal direction at positions R44, R45, R54, and R55, namely Grad44_hor, Grad45_hor, Grad54_hor, and G55rad_hor, are calculated as follows:
[0167] Grad44_hor = | G44_hor_diff + grad_w ×R44_hor_diff |.
[0168] Grad45_hor = | G45_hor_diff + grad_w ×R45_hor_diff |.
[0169] Grad54_hor = | G54_hor_diff + grad_w ×R54_hor_diff |.
[0170] Grad55_hor = | G55_hor_diff + grad_w ×R55_hor_diff |.
[0171] The gradient estimation results in the vertical direction at positions R44, R45, R54, and R55, namely Grad44_ver, Grad45_ver, Grad54_ver, and Grad55_ver, are calculated as follows:
[0172] Grad44_ver = | G44_ver_diff + grad_w ×R44_ver_diff |.
[0173] Grad45_ver = | G45_ver_diff + grad_w ×R45_ver_diff |.
[0174] Grad54_ver = | G54_ver_diff + grad_w ×R54_ver_diff |.
[0175] Grad55_ver = | G55_ver_diff + grad_w ×R55_ver_diff |.
[0176] Steps S3 and S4 involve inputting the green channel interpolation estimation results and gradient estimation results obtained in step S2 into the first convolutional neural network and the second convolutional neural network for optimization processing, respectively.
[0177] Optionally, the first convolutional neural network and the second convolutional neural network are constructed similarly. In one embodiment, the first convolutional neural network and the second convolutional neural network include:
[0178] The horizontal preprocessing convolutional layer increases the dimensionality of the one-dimensional horizontal estimation result.
[0179] Vertical preprocessing convolutional layers increase the dimensionality of the one-dimensional vertical estimation results;
[0180] The horizontal residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the horizontal estimation result after dimensionality increase, and the output is the horizontal feature.
[0181] The vertical residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the vertical estimation result after dimensionality increase, and the output is the vertical feature.
[0182] The horizontal and vertical feature fusion module performs channel splicing on the horizontal and vertical features to obtain the horizontal and vertical fused features.
[0183] A feature downsampling layer downsamples the horizontal and vertical fused features;
[0184] The residual blocks of the downsampled features are densely connected, consisting of multiple connected residual blocks;
[0185] The feature upsampling layer upsamples the downsampled residual output.
[0186] Feature residual connection: The horizontal and vertical fused features are residually connected with the upsampling results.
[0187] Please refer to Figure 3 This demonstrates a first convolutional neural network that optimizes the interpolation estimation results of the green channel.
[0188] The horizontal green channel interpolation estimation result first passes through a 3×3 horizontal preprocessing convolutional layer L1, which increases the dimensionality of the 1-dimensional horizontal interpolation estimation result to n1 dimensions, for example, n1=16; the vertical green channel interpolation estimation result first passes through a 3×3 vertical preprocessing convolutional layer L2, which increases the dimensionality of the 1-dimensional vertical interpolation estimation result to n1 dimensions, for example, n1=16.
[0189] The horizontal residual block dense connection L3 consists of N1 residual blocks connected together, for example, N1=4. In an example, each of the N1 residual blocks consists of a first 3×3 convolutional layer, a nonlinear activation layer ReLU, a second 3×3 convolutional layer, and the input plus the output of the second convolutional layer. The vertical residual block dense connection L4 can also consist of N1 residual blocks connected together, and its structure can be referenced to the horizontal residual block dense connection L3.
[0190] The horizontal and vertical feature fusion module L5 performs horizontal and vertical interpolation feature fusion on the horizontal and vertical residual results, and performs a concat operation, which changes the number of channels to 32.
[0191] The feature downsampling layer L6 downsamples the fused interpolated features. In one example, the feature downsampling layer L6 uses a 3×3 convolutional layer with a stride of 2 to downsample the horizontal and vertical fused interpolated features by a factor of 2.
[0192] The residual block densely connected L7 of the downsampling feature consists of N2=4 residual blocks in one example.
[0193] The feature upsampling layer L8 can use the nearest neighbor upsampling operator.
[0194] The feature residual connection L9 performs a residual connection between the 32-channel horizontal and vertical fused interpolation features output by the horizontal and vertical feature fusion module L5 and the upsampling results of the feature upsampling layer L8. This is then optionally followed by a 3×3 convolution to convert the 32-channel to 2-channel outputs the optimized results for the two channels, corresponding to the horizontal green channel interpolation result NN_G_hor and the vertical green channel interpolation result NN_G_ver, respectively.
[0195] The construction of a second convolutional neural network to optimize the gradient estimation results and Figure 3 The construction of the first convolutional neural network shown is roughly similar. The only differences are the number of dimensions of the preprocessing convolutional layers and the number of residual blocks with dense connections.
[0196] In one example of the second convolutional neural network, the horizontal preprocessing convolutional layer L1 and the vertical preprocessing convolutional layer L2 are used to upscale the gradient estimation results in the horizontal and vertical directions to n²=8 dimensions, respectively. Then, the horizontal residual block dense connection L3 and the vertical residual block dense connection L4 are composed of N³=2 residual blocks. After the horizontal and vertical feature fusion module L5 performs a concat operation, the resulting horizontal and vertical fused gradient features have 16 channels. The feature residual connection L8 performs a residual connection between the 16-channel horizontal and vertical fused gradient features output by the horizontal and vertical feature fusion module L5 and the upsampling result of the feature upsampling layer L7. Then, optionally, a 3×3 convolution is performed to convert the 16-channel to 2-channel gradient, outputting the optimized results for two channels, corresponding to the horizontal gradient result NN_Grad_hor and the vertical gradient result NN_Grad_ver, respectively.
[0197] Optionally, step S5 includes:
[0198] Step S51: Use the reciprocal of the gradient result plus 1 in the horizontal direction as the horizontal weight, and weight the green channel interpolation result in the horizontal direction. That is, the horizontal weight hor_w = 1 / (NN_Grad_hor + 1).
[0199] Step S52: Use the reciprocal of the gradient result in the vertical direction plus 1 as the vertical direction weight, and weight the green channel interpolation result in the vertical direction. That is, the vertical direction weight ver_w = 1 / (NN_Grad_ver+1).
[0200] Step S53: The initial green interpolation fusion result is the weighted green channel interpolation result divided by the weight. In an example, the initial green interpolation fusion result G_merge is calculated as follows:
[0201] G_merge=(hor_w×NN_G_hor+ver_w×NN_G_ver) / (hor_w+ver_w)
[0202] Step S6 is to merge the initial green interpolation fusion result with the green channel value in the four-bay image, and input it into the third convolutional neural network for optimization processing to obtain the final green interpolation result.
[0203] Optionally, the construction of the third convolutional neural network differs from that of the first convolutional neural network. Please refer to [reference needed]. Figure 4 In one embodiment, the third convolutional neural network includes:
[0204] The green channel fusion layer J1 merges the initial green interpolation fusion result with the green channel value in the four-bay image to obtain the initial green channel fusion result, and then uses a convolutional layer to increase the dimensionality of the initial green channel fusion result;
[0205] Green channel downsampling layer J2 is used to spatially downsample the fusion result of the up-dimensional green channel;
[0206] The dense connection J3 of the green channel residual block is composed of multiple residual blocks and is used to perform deep feature enhancement on the downsampled green channel fusion result to obtain the green channel fusion feature.
[0207] Green channel upsampling layer J4 is used to spatially upsample the green channel fusion features;
[0208] The green channel residual connection J5 performs a residual connection between the upsampled green channel fusion features and the green channel fusion result after dimensionality upgrade of the green channel fusion layer to obtain the final green channel fusion result.
[0209] In one example, the green channel fusion layer J1 merges the initial green interpolation fusion result G_merge obtained in step S5 with the green channel values in the original four-bayer image to obtain the initial green channel fusion result. This initial green channel fusion result is then augmented to n3=16 dimensions using a 3×3 convolutional layer.
[0210] The green channel downsampling layer J2 performs 3×3 downsampling on the green channel fusion result that has been upgraded to 16 dimensions. In one example, a 3×3 convolutional layer with a stride of 2 is used to perform a 2x downsampling on the green channel fusion result.
[0211] In one example, the dense connection J3 of the green channel residual blocks consists of N5=4 residual blocks connected together.
[0212] Green channel upsampling layer J4, if the nearest neighbor upsampling operator can be used.
[0213] The green channel residual connection J5 connects the green channel fusion features upsampled from the green channel upsampling layer J4 with the 16-channel green channel fusion result upscaled from the green channel fusion layer J1. Then, optionally, a 3×3 convolution is performed to convert the 16-channel to 1-channel outputs the final green interpolation result G_interp.
[0214] Optionally, before performing steps S3 to S6, the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network can be trained. In one embodiment, the training steps for the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network include:
[0215] Step S91: Construct the training dataset;
[0216] Step S92: Individual training phase, the first convolutional neural network, the second convolutional neural network and the third convolutional neural network are trained independently respectively; when training any one convolutional neural network, the other two convolutional neural networks are set to pass-through mode;
[0217] Step S93: Joint fine-tuning stage, load the parameters of the three convolutional neural networks obtained in the individual training stages into the corresponding networks, and simultaneously enable the first convolutional neural network, the second convolutional neural network and the third convolutional neural network for training, and jointly update the parameters of the three convolutional neural networks.
[0218] In one example, step S91, which constructs the training dataset, includes: randomly selecting 1024 images from the super-resolution dataset; the input for each group is to convert the images in the dataset in RGB three-channel format into the corresponding raw data according to the four-bayer format; the output for each group is to extract the G channel of the image and save it as 1-channel data.
[0219] In the separate training phase of step S92, for example, when training the first convolutional neural network, the second and third convolutional neural networks are set to pass-through mode, meaning their inputs equal their outputs. In this case, the entire model degenerates into network optimization only for interpolating the green channels in different directions. Then, the loss function is set to MSE, and the device calculates the mean squared error (MSE) loss between the output G interpolation result and the G channel result of the training data labels. After backpropagation, the model parameters in the first convolutional neural network are updated. Similarly, the second and third convolutional neural networks are trained separately.
[0220] Step S93 is the joint training step of the three convolutional neural networks. The parameters of the three convolutional neural networks trained separately in step S92 are merged and loaded into the corresponding networks, and all three convolutional neural networks are turned on for mixed training, and the parameters of the three convolutional neural networks are jointly updated.
[0221] Step S7, based on the final green interpolation result obtained in step S6, interpolates the red and blue pixels of each pixel in the original four-bayer image. The inventors found that the red and blue channels primarily affect color and have a relatively small impact on resolution. Optimizing the interpolation of the red and blue channels using a neural network would consume significant computational resources with minimal gains. Therefore, in this embodiment, step S7 directly uses interpolation to calculate the interpolation results for the red and blue channels, resulting in a simple calculation method with low resource consumption.
[0222] Furthermore, the interpolation in step S7 efficiently utilizes the final green interpolation result optimized by the neural network. Under the premise that the final green interpolation result has high quality, the interpolation accuracy of the red and blue channels can also be well guaranteed.
[0223] The following exemplifies the process of calculating the interpolation results for the red and blue channels in step S7 using interpolation. It is easy to understand that the interpolation of the red and blue channels, based on 2×2 pixel groups of different colors in a four-bay image, has two types. One is interpolating the green 2×2 pixel group, in which case the red and blue channel interpolation results for that pixel group need to be calculated separately. The other is interpolating the red or blue 2×2 pixel group, in which case only the other contrasting color (red and blue are a pair of contrasting colors; the contrasting color of red is blue, and the contrasting color of blue is red) needs to be interpolated, and the interpolation results for the contrasting color channels can be calculated. Therefore, step S7 can include steps S7A and S7B for the above two types respectively. Step S7A interpolates the green 2×2 pixel group, and step S7B interpolates the red or blue 2×2 pixel group.
[0224] In one embodiment, step S7A includes:
[0225] Step S7A1: Extract the second buffer window centered on the 2×2 pixel point of the same color channel in the four Bayer image. If the second buffer window is the green channel, calculate the average color difference in the horizontal and vertical directions based on the final green interpolation result.
[0226] Step S7A2: Subtract the original channel values of the 2×2 pixels of the second buffer window from the average color difference values in the horizontal and vertical directions respectively to obtain the red channel interpolation results and blue channel interpolation results of the 2×2 pixels of the second buffer window.
[0227] Please refer to Figure 5 and Figure 6 In step S7A1, a 2×2 pixel point (G5, G6, G9, G10) of the green channel is extracted from the original four-bayer image as a second buffer window. At this point, two situations exist: the first situation is that the pixels adjacent horizontally to the second buffer window are red (R), and the pixels adjacent vertically to the second buffer window are blue (B), such as... Figure 5 As shown. The second case is where the pixels adjacent horizontally in the second buffer window are blue (B), and the pixels adjacent vertically are red (R), as shown... Figure 6 As shown.
[0228] Please refer to Figure 5 For the first case, calculating the average color difference in the horizontal direction refers to the average color difference of red:
[0229] R_diff_avg1 = (G_interp(R4)-R4 + G_interp(R7)-R7) / 2.
[0230] R_diff_avg2 = (G_interp(R8)-R8 + G_interp(R11)-R11) / 2.
[0231] The calculation of the average color difference in the vertical direction refers to the average color difference of blue:
[0232] B_diff_avg1 = (G_interp(B1)-B1 + G_interp(B13)-B13) / 2.
[0233] B_diff_avg2 = (G_interp(B2)-B2 + G_interp(B14)-B14) / 2.
[0234] In the above formula, G_inperp(x) represents the final green interpolation result at position x after neural network optimization. It can be understood that in the calculation of the color difference mean, the final green interpolation result G_interp optimized by the neural network is used to calculate the difference between the red and blue pixels, thus significantly improving the accuracy of the color difference mean.
[0235] Then, in step S7A2, the original channel value (green channel value) of the 2×2 pixel point of the center of the second buffer window is subtracted from the average color difference in the horizontal and vertical directions, respectively, to obtain the red channel interpolation result and the blue channel interpolation result at the position of the second buffer window:
[0236] R5 = G5 - R_diff_avg1; B5 = G5 - B_diff_avg1.
[0237] R6 = G6 - R_diff_avg1; B6 = G6 - B_diff_avg1.
[0238] R9 = G9 - R_diff_avg2; B9 = G9 - B_diff_avg2.
[0239] R10 = G10- R_diff_avg2; B10 = G10- B_diff_avg2.
[0240] Please refer to Figure 6 For the second case, calculating the average color difference in the horizontal direction refers to the average color difference of blue:
[0241] B_diff_avg1 = (G_interp(B4)-B4 + G_interp(B7)-B7) / 2.
[0242] B_diff_avg2 = (G_interp(B8)-B8 + G_interp(B11)-B11) / 2.
[0243] The calculation of the average color difference in the vertical direction refers to the average color difference of red:
[0244] R_diff_avg1 = (G_interp(R1)-R1 + G_interp(R13)-R13) / 2.
[0245] R_diff_avg2 = (G_interp(R2)-R2 + G_interp(R14)-R14) / 2.
[0246] In step S7A2, the original channel value (green channel value) of the 2×2 pixel point of the second buffer window at the center is subtracted from the average color difference in the horizontal and vertical directions to obtain the red channel interpolation result and the blue channel interpolation result at the position of the second buffer window. Please refer to the explanation of the first case, which will not be repeated here.
[0247] In one embodiment, step S7B includes:
[0248] Step S7B1: Extract a second buffer window centered on a 2×2 pixel point of the same color channel in the four-bay image, with red and blue as a group of dissimilar colors; if the second buffer window is a red channel or a blue channel, calculate the average color difference of the dissimilar colors around the second buffer window based on the final green interpolation result;
[0249] Step S7B2: Subtract the final green interpolation result of the 2×2 pixel point of the second cache window from the average color difference of the different colors to obtain the interpolation result of the different color channel of the 2×2 pixel point of the second cache window.
[0250] Please refer to Figure 7 Taking the extraction of a 2×2 pixel point (R5, R6, R9, R10) of the red channel from the original four-bay image as the second buffer window as an example, the dissimilar color referred to in steps S7B1 and S7B2 is blue (B), and the average color difference of the dissimilar color around the second buffer window to be calculated in step S7B1 refers to the average color difference of blue:
[0251] B_diff_avg = (G_interp(B0)-B0+G_interp(B3)-B3+G_interp(B12)-B12+G_interp(B15)-B15) / 4
[0252] In step S7B2, the final green interpolation result G_interp of the 2×2 pixel point of the center of the second buffer window is subtracted from the average color difference of the opposite color (blue) to obtain the interpolation result of the opposite color channel (blue channel) at the position of the second buffer window:
[0253] B5 = G_interp(R5) - B_diff_avg.
[0254] B6 = G_interp(R6) - B_diff_avg.
[0255] B9 = G_interp(R9) - B_diff_avg.
[0256] B10 = G_interp(R10) - B_diff_avg.
[0257] Please refer to Figure 8 When extracting a 2×2 pixel point (B5, B6, B9, B10) of the blue channel from the original four-bay image as the second buffer window, the dissimilar color referred to in steps S7B1 and S7B2 is red (R), and the average color difference of the dissimilar colors around the second buffer window to be calculated in step S7B1 refers to the average color difference of red:
[0258] R_diff_avg=(G_interp(R0)-R0+G_interp(R3)-R3+G_interp(R12)-R12+G_interp(R15)-R15) / 4
[0259] In step S7B2, the final green interpolation result G_interp of the 2×2 pixel point of the center of the second buffer window is subtracted from the mean value of the red color difference to obtain the interpolation result of the red channel at the position of the second buffer window:
[0260] R5 = G_interp(B5)-R_diff_avg.
[0261] R6 = G_interp(B6)-R_diff_avg.
[0262] R9 = G_interp(B9)-R_diff_avg.
[0263] R10 = G_interp(B10)-R_diff_avg.
[0264] After completing steps S6 and S7 to obtain the final green interpolation result and the red-blue channel interpolation results, step S8 is the conversion and output step of the ordinary Bayer image. The conversion of the ordinary Bayer image needs to be performed according to the target Bayer image mode. Taking the output target Bayer image mode as BGGR mode as an example:
[0265] When the output position and the position and color of the input four-bay image are the same, the original pixels of the four-bay image are used directly;
[0266] When the output position is green and the input four-bay image is not green, the final green interpolation result G_interp output in step S6 is used; when the output position is red or blue and the input four-bay image is not the same color, the interpolation result of the red and blue channels output in step S7 is used, that is, the corresponding interpolation pixel is selected.
[0267] Optionally, the four-bayer image can be preprocessed before extracting the first cache window; after obtaining the converted ordinary Bayer image, the ordinary Bayer image can be postprocessed.
[0268] In one example, preprocessing includes black background correction for pixels at each location in the four-Bayer image, subtraction of blc, and gain multiplication (r_gain, b_gain) correction for red and blue pixels. Preprocessing is an optional step; if the method of this embodiment is applied after the black background subtraction and gain multiplication correlation modules in the ISP, then preprocessing of the four-Bayer image is not required.
[0269] In one example, post-processing includes dividing each pixel location by the gains r_gain and b_gain of the red and blue pixel locations, and then adding the global black background blc value. Post-processing is an optional step; if the method of this embodiment is applied after the black background reduction and gain multiplication correlation modules in the ISP, then post-processing may not be performed on the converted output image.
[0270] Based on the four-Bayer image conversion method described above, please refer to... Figure 9 The present invention also provides a four-bayer image conversion device, which includes: a buffer window extraction unit 11, a green channel interpolation estimation unit 12, a gradient estimation unit 13, a first convolutional neural network 14, a second convolutional neural network 15, a green channel interpolation weighted fusion unit 16, a third convolutional neural network 17, a red and blue channel interpolation unit 18, and a conversion output unit 19.
[0271] The cache window extraction unit 11 extracts the first cache window centered on a 2×2 pixel point in the red or blue channel of the four Bayer image;
[0272] The green channel interpolation estimation unit 12 calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window;
[0273] The gradient estimation unit 13 calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window, and calculates the gradient estimation results in the horizontal and vertical directions.
[0274] The first convolutional neural network 14 takes the green channel interpolation estimation result as input, processes it, and obtains the green channel interpolation results in the horizontal and vertical directions.
[0275] The second convolutional neural network 15 takes the gradient estimation result as input, processes it, and obtains the gradient results in the horizontal and vertical directions.
[0276] The green channel interpolation weighted fusion unit 16 performs weighted fusion on the green channel interpolation results based on the gradient results to obtain an initial green interpolation fusion result;
[0277] The third convolutional neural network 17 inputs the merged initial green interpolation fusion result and the green channel value in the four-bay image, processes them, and obtains the final green interpolation result.
[0278] The red and blue channel interpolation unit 18 calculates the interpolation results of the red and blue channels based on the final green interpolation result and the four-bay image using an interpolation method.
[0279] The conversion output unit 19 selects the corresponding interpolated pixels or the original pixels in the four-bay image according to the target Bayer image pattern to obtain the converted ordinary Bayer image.
[0280] The four-Bayer image conversion apparatus may also optionally be capable of performing various steps or sub-steps in the above-described four-Bayer image conversion method. This embodiment does not limit the specific configuration of the four-Bayer image conversion apparatus; those skilled in the art can configure it according to actual needs.
[0281] In summary, the four-Bayer image conversion method and apparatus provided by this invention include: extracting a first buffer window centered on a 2×2 pixel point of the red or blue channel in the four-Bayer image; calculating the horizontal and vertical green channel interpolation estimation results based on the first buffer window, and calculating the horizontal and vertical gradient estimation results; inputting the green channel interpolation estimation results into a first convolutional neural network for processing to obtain the horizontal and vertical green channel interpolation results; and inputting the gradient estimation results into a second convolutional neural network for processing. The process involves obtaining gradient results in the horizontal and vertical directions; weighted fusion of the green channel interpolation results based on these gradient results to obtain an initial green interpolation fusion result; merging this initial green interpolation fusion result with the green channel values in the four-bay image and inputting it into a third convolutional neural network for processing to obtain the final green interpolation result; calculating the interpolation results for the red and blue channels using interpolation based on the final green interpolation result and the four-bay image; and selecting the corresponding interpolation pixels or original pixels from the four-bay image according to the target Bayer image pattern to obtain the converted ordinary Bayer image. This configuration, by employing a three-level neural network to optimize the key interpolation stage of the green channel, and then utilizing the optimized green channel in conjunction with the red and blue channels using efficient traditional interpolation, significantly improves image quality while concentrating neural network computing resources on the high-yield key interpolation stage of the green channel, effectively resolving the contradiction between quality and efficiency.
[0282] It should be noted that the above embodiments can be combined with each other. The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.
Claims
1. A four-Bayer image conversion method, characterized in that, include: Extract the first buffer window centered on a 2×2 pixel point in the red or blue channel of the four-bay image; Based on the first cache window, calculate the green channel interpolation estimation results in the horizontal and vertical directions, and calculate the gradient estimation results in the horizontal and vertical directions. The green channel interpolation estimation results are input into the first convolutional neural network for processing to obtain the green channel interpolation results in the horizontal and vertical directions. The gradient estimation results are input into the second convolutional neural network for processing to obtain the gradient results in the horizontal and vertical directions; Based on the gradient results, the green channel interpolation results are weighted and fused to obtain the initial green interpolation fusion result; The initial green interpolation fusion result is combined with the green channel value in the four-bay image and input into the third convolutional neural network for processing to obtain the final green interpolation result. Based on the final green interpolation result and the four-bay image, the interpolation results for the red and blue channels are calculated using an interpolation method. Based on the target Bayer image pattern, select the corresponding interpolated pixels or the original pixels in the four-bayer image to obtain the converted ordinary Bayer image.
2. The quaternary image conversion method of claim 1, wherein, The first convolutional neural network and the second convolutional neural network include: The horizontal preprocessing convolutional layer increases the dimensionality of the one-dimensional horizontal estimation result. Vertical preprocessing convolutional layers increase the dimensionality of the one-dimensional vertical estimation results; The horizontal residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the horizontal estimation result after dimensionality increase, and the output is the horizontal feature. The vertical residual blocks are densely connected, consisting of multiple connected residual blocks. The input is the vertical estimation result after dimensionality increase, and the output is the vertical feature. The horizontal and vertical feature fusion module performs channel splicing on the horizontal and vertical features to obtain the horizontal and vertical fused features. A feature downsampling layer downsamples the horizontal and vertical fused features; The residual blocks of the downsampled features are densely connected, consisting of multiple connected residual blocks; The feature upsampling layer upsamples the downsampled residual output. Feature residual connection: The horizontal and vertical fused features are residually connected with the upsampling results.
3. The quaternary image conversion method of claim 1, wherein, The third convolutional neural network includes: The green channel fusion layer merges the initial green interpolation fusion result with the green channel value in the four-bay image to obtain the initial green channel fusion result, and then uses a convolutional layer to increase the dimensionality of the initial green channel fusion result; The green channel downsampling layer is used to spatially downsample the fusion result of the up-dimensional green channel; The dense connection of the green channel residual blocks, consisting of multiple residual blocks, is used to perform deep feature enhancement on the downsampled green channel fusion result to obtain the green channel fusion feature. A green channel upsampling layer is used to spatially upsample the fused features of the green channel. The green channel residual connection is performed by performing a residual connection between the upsampled green channel fusion features and the green channel fusion result after dimensionality upgrade of the green channel fusion layer to obtain the final green channel fusion result.
4. The quaternary image conversion method of claim 1, wherein, Based on the first cache window, the steps for calculating the green channel interpolation estimation results in the horizontal and vertical directions include: Based on the 2×2 pixels of the first cache window, green pixels are found in the horizontal and vertical directions respectively, and the green low-frequency interpolation result is obtained by weighting some green pixels in the horizontal and vertical directions. Based on the channel color of the first cache window, and based on the 2×2 pixels of the first cache window, interpolation points corresponding to the channel color are found in the horizontal and vertical directions respectively, and weighted to obtain the low-frequency interpolation result corresponding to the channel color. Subtract the low-frequency interpolation result corresponding to the channel color from the 2×2 pixels of the first cache window to obtain the high-frequency interpolation result corresponding to the channel color. The low-frequency interpolation results of the green channel and the high-frequency interpolation results corresponding to the channel colors are weighted and fused to obtain the green channel interpolation estimation results in the horizontal and vertical directions.
5. The quaternary image conversion method of claim 1, wherein, Based on the first cache window, the steps for calculating the gradient estimation results in the horizontal and vertical directions include: Based on the 2×2 pixels of the first cache window, a subtraction operation is performed on some green pixels in the horizontal and vertical directions to calculate the green difference in the horizontal and vertical directions. Based on the channel color of the first cache window, and based on the 2×2 pixels of the first cache window, the pixels corresponding to the channel color in the horizontal and vertical directions are subtracted to calculate the difference between the horizontal and vertical channel colors. The green difference and the difference of the corresponding channel color are weighted and fused, and the absolute value is taken to obtain the gradient estimation results in the horizontal and vertical directions.
6. The quaternary image conversion method of claim 1, wherein, The steps for weighted fusion of the green channel interpolation results based on the gradient results to obtain the initial green interpolation fusion result include: The horizontal green channel interpolation result is weighted by taking the reciprocal of the gradient result plus 1 in the horizontal direction. The vertical green channel interpolation results are weighted by taking the inverse of the gradient result plus 1 in the vertical direction as the vertical weight. The initial green interpolation fusion result is the weighted green channel interpolation result divided by the weight.
7. The quaternary image conversion method of claim 1, wherein, Based on the final green interpolation result and the four-bay image, the steps for calculating the interpolation results of the red and blue channels using interpolation include: Extract a second buffer window centered on a 2×2 pixel point of the same color channel in the four-bay image. If the second buffer window is the green channel, calculate the average color difference in the horizontal and vertical directions based on the final green interpolation result. The red channel interpolation result and blue channel interpolation result of the 2×2 pixel point of the second cache window are obtained by subtracting the original channel value of the second cache window from the average color difference in the horizontal and vertical directions, respectively.
8. The four-Bayer image conversion method according to claim 1, characterized in that, Based on the final green interpolation result and the four-bay image, the steps for calculating the interpolation results of the red and blue channels using interpolation include: A second buffer window is extracted centered on a 2×2 pixel point of the same color channel in the four-bay image, with red and blue as a group of dissimilar colors; if the second buffer window is a red channel or a blue channel, the average color difference of the dissimilar colors around the second buffer window is calculated based on the final green interpolation result; The interpolation result of the opposite color channel of the second cache window is obtained by subtracting the final green interpolation result of the 2×2 pixel point of the second cache window from the mean color difference of the opposite color.
9. The quaternary image conversion method of claim 1, wherein, The training steps for the first, second, and third convolutional neural networks include: Build the training dataset; In the individual training phase, the first convolutional neural network, the second convolutional neural network, and the third convolutional neural network are trained independently, respectively; while training any one convolutional neural network, the other two convolutional neural networks are set to pass-through mode. In the joint fine-tuning phase, the parameters of the three convolutional neural networks obtained in the individual training phases are loaded into the corresponding networks, and the first, second, and third convolutional neural networks are simultaneously enabled for training, and the parameters of the three convolutional neural networks are jointly updated.
10. A four-bayer image conversion device, characterized in that, include: The system includes a buffer window extraction unit, a green channel interpolation estimation unit, a gradient estimation unit, a first convolutional neural network, a second convolutional neural network, a green channel interpolation weighted fusion unit, a third convolutional neural network, a red and blue channel interpolation unit, and a conversion output unit. The cache window extraction unit extracts the first cache window centered on a 2×2 pixel point in the red or blue channel of the four Bayer image; The green channel interpolation estimation unit calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window; The gradient estimation unit calculates the green channel interpolation estimation results in the horizontal and vertical directions based on the first cache window, and calculates the gradient estimation results in the horizontal and vertical directions. The first convolutional neural network takes the green channel interpolation estimation result as input, processes it, and obtains the green channel interpolation results in the horizontal and vertical directions. The second convolutional neural network takes the gradient estimation result as input, processes it, and obtains the gradient results in the horizontal and vertical directions. The green channel interpolation weighted fusion unit performs weighted fusion on the green channel interpolation results based on the gradient results to obtain the initial green interpolation fusion results; The initial green interpolation fusion result after being merged and input into the third convolutional neural network are processed together with the green channel value in the four-bay image to obtain the final green interpolation result. The red and blue channel interpolation unit calculates the interpolation results of the red and blue channels based on the final green interpolation result and the four-bay image using an interpolation method. The conversion output unit selects the corresponding interpolated pixels or the original pixels in the four-bay image according to the target Bayer image pattern to obtain the converted ordinary Bayer image.