Image reconstruction method, device, module, equipment, storage medium and program product

Abstract

The present application provides an image reconstruction method, apparatus, module, device, storage medium and program product, wherein the image reconstruction method includes: obtaining white channel information, green channel information, red channel information and blue channel information of a native domain image, wherein the native domain image is in Bayer format; extracting gradient information of the white channel information; interpolating the green channel information, red channel information and blue channel information of the native domain image according to the white channel information and the extracted gradient information; merging the interpolated green channel information, red channel information and blue channel information to obtain a reconstructed native domain image. The technical solution guides the interpolation of the green, red and blue channel information in combination with the white channel information, thereby improving the accuracy of the interpolation, reducing the noise of the reconstructed high-resolution native domain image, and improving the quality of image reconstruction.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image reconstruction method, device, module, equipment, storage medium and program product. Background Art

[0002] With the continuous development of high-resolution imaging technology, people's demand for image resolution is growing. Super-resolution (SR) technology has received widespread attention.

[0003] Super-resolution technology improves image resolution by restoring high-frequency detail information in low-resolution images. Currently, the commonly used super-resolution technologies are mainly divided into RGB (Red Green Blue) domain super-resolution technology and Raw domain super-resolution technology.

[0004] RGB domain super-resolution technology needs to first restore the low-resolution image of the three channels of the Raw domain image based on the demosaicing technology, and then improve the image resolution in the RGB domain based on interpolation methods such as bilinear interpolation algorithm, bicubic interpolation algorithm, deep learning technology, etc. Due to the introduction of demosaicing technology, irreversible interpolation traces including pseudo color and aliasing are introduced before the enlargement process, thereby increasing the noise on the enlarged image.

[0005] Raw domain super-resolution technology overcomes the above drawbacks and directly reconstructs low-resolution Raw domain images into high-resolution Raw domain images through interpolation algorithms. However, since the Raw domain data information is relatively sparse in each channel, there is significant pseudo-color in the reconstructed high-resolution image, and the reconstruction result is not ideal. Summary of the invention

[0006] The present application provides an image reconstruction method, apparatus, module, device, storage medium and program product, which improve the accuracy and quality of image super-resolution reconstruction by introducing white channel information.

[0007] In a first aspect, the present application provides an image reconstruction method, comprising:

[0008] Acquire white channel information, green channel information, red channel information and blue channel information of a native domain image, wherein the native domain image is in a Bayer format; extract gradient information of the white channel information; interpolate the green channel information, red channel information and blue channel information of the native domain image according to the white channel information and the extracted gradient information; merge the interpolated green channel information, red channel information and blue channel information to obtain a reconstructed native domain image.

[0009] In a possible implementation, interpolating green channel information, red channel information, and blue channel information of the native domain image respectively according to the white channel information and the extracted gradient information includes:

[0010] The green channel information of the native domain image is interpolated according to the white channel information and the extracted gradient information; and the red channel information and the blue channel information are interpolated respectively based on the green channel information after the interpolation processing.

[0011] In a possible implementation manner, extracting the gradient information of the white channel information includes:

[0012] The white channel information is up-sampled to obtain high-resolution white channel information; and the gradient information of the white channel information and the high-resolution white channel information are respectively extracted.

[0013] In a possible implementation, interpolating green channel information of the native domain image according to the white channel information and the extracted gradient information includes:

[0014] According to the white channel information and its gradient information, the missing pixels in the green channel information are interpolated to obtain first interpolated green channel information; the first interpolated green channel information is mapped to a high-resolution grid to obtain high-resolution green channel information; based on the high-resolution white channel information and its gradient information, the high-resolution green channel information is interpolated to obtain interpolated green channel information.

[0015] In a possible implementation, interpolating missing pixels in the green channel information according to the white channel information and its gradient information to obtain first interpolated green channel information includes:

[0016] The first color difference information is obtained by subtracting the difference between the green channel information and the white channel information; based on the gradient information of the white channel information, the missing pixels in the first color difference information are interpolated; and the interpolated first color difference information is added to the white channel information to obtain the first interpolated green channel information.

[0017] In a possible implementation, interpolating the high-resolution green channel information based on the high-resolution white channel information and its gradient information to obtain interpolated green channel information includes:

[0018] Based on the high-resolution white channel information and its gradient information, a first pixel in the high-resolution green channel information is interpolated to obtain second interpolated green channel information; based on the high-resolution white channel information and its gradient information, the remaining missing pixels in the second interpolated green channel information are interpolated to obtain interpolated green channel information; wherein the first pixel is a pixel at a position in the high-resolution green channel information where a pixel is missing and the sum of the horizontal and vertical coordinates is an even number.

[0019] In a possible implementation, interpolating the red channel information and the blue channel information respectively based on the green channel information after interpolation processing includes:

[0020] For any channel information of the red channel information and the blue channel information, the channel information is mapped to a high-resolution grid to obtain high-resolution channel information; a difference value of the high-resolution channel information is subtracted from the green channel information after interpolation processing to obtain second color difference information; the second color difference information is interpolated; the green channel information after interpolation processing is subtracted from the second color difference information after interpolation processing to obtain red channel information after interpolation processing or blue channel information after interpolation processing.

[0021] In a possible implementation, the method further includes:

[0022] Convert the reconstructed native domain image to RGB format for display.

[0023] In a possible implementation, the method further includes:

[0024] The native domain image is acquired based on an RGBW (Red Green Blue White) sensor.

[0025] In a second aspect, the present application provides an image reconstruction device, comprising:

[0026] A channel information acquisition module is used to acquire white channel information, green channel information, red channel information and blue channel information of a native domain image, wherein the native domain image is in a Bayer format; a white channel gradient extraction module is used to extract gradient information of the white channel information; an interpolation module is used to interpolate the green channel information, red channel information and blue channel information of the native domain image according to the white channel information and the extracted gradient information; and a reconstruction module is used to merge the green channel information, red channel information and blue channel information after interpolation processing to obtain a reconstructed native domain image.

[0027] In a possible implementation, the interpolation module includes:

[0028] A green channel interpolation unit is used to interpolate the green channel information of the native domain image according to the white channel information and the extracted gradient information; and interpolate the red channel information and the blue channel information respectively based on the interpolated green channel information.

[0029] In a possible implementation, the white channel gradient extraction module is specifically used to:

[0030] The white channel information is up-sampled to obtain high-resolution white channel information; and the gradient information of the white channel information and the high-resolution white channel information are respectively extracted.

[0031] In a possible implementation, the green channel interpolation unit includes:

[0032] According to the white channel information and its gradient information, the missing pixels in the green channel information are interpolated to obtain first interpolated green channel information; the first interpolated green channel information is mapped to a high-resolution grid to obtain high-resolution green channel information; based on the high-resolution white channel information and its gradient information, the high-resolution green channel information is interpolated to obtain interpolated green channel information.

[0033] In a possible implementation, the green channel first interpolation subunit is specifically configured to:

[0034] The first color difference information is obtained by subtracting the difference between the green channel information and the white channel information; based on the gradient information of the white channel information, the missing pixels in the first color difference information are interpolated; and the interpolated first color difference information is added to the white channel information to obtain the first interpolated green channel information.

[0035] In a possible implementation, the green channel second interpolation subunit is specifically configured to:

[0036] Based on the high-resolution white channel information and its gradient information, a first pixel in the high-resolution green channel information is interpolated to obtain second interpolated green channel information; based on the high-resolution white channel information and its gradient information, the remaining missing pixels in the second interpolated green channel information are interpolated to obtain interpolated green channel information; wherein the first pixel is a pixel at a position in the high-resolution green channel information where a pixel is missing and the sum of the horizontal and vertical coordinates is an even number.

[0037] In a possible implementation, the red and blue channel interpolation unit is specifically used to:

[0038] For any channel information of the red channel information and the blue channel information, the channel information is mapped to a high-resolution grid to obtain high-resolution channel information; a difference value of the high-resolution channel information is subtracted from the green channel information after interpolation processing to obtain second color difference information; the second color difference information is interpolated; the green channel information after interpolation processing is subtracted from the second color difference information after interpolation processing to obtain red channel information after interpolation processing or blue channel information after interpolation processing.

[0039] In a possible implementation manner, the device further includes:

[0040] The format conversion module is used to convert the reconstructed native domain image into RGB format for display.

[0041] In a possible implementation manner, the device further includes:

[0042] The image acquisition module is used to acquire the native domain image based on the RGBW sensor.

[0043] In a third aspect, the present application provides an image reconstruction modeling group, comprising: a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image reconstruction method provided in the first aspect of the present application.

[0044] In a fourth aspect, the present application provides a terminal device, comprising an image sensor and the image reconstruction model group provided in the third aspect of the present application; wherein the image sensor is used to capture native domain images.

[0045] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image reconstruction method provided in the first aspect of the present application is implemented.

[0046] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image reconstruction method provided in the first aspect of the present application.

[0047] The image reconstruction method, apparatus, module, device, storage medium and program product provided in the embodiments of the present application are for super-resolution reconstruction of native domain images in Bayer format. After obtaining the native domain image, the channel information of the four channels of white, green, red and blue of the native domain image is extracted, and the gradient information of the white channel information is extracted. Then, the white channel information and the extracted gradient information are combined to guide the interpolation of the RGB three-channel information, thereby achieving high-resolution amplification of the RGB three-channel. The interpolated RGB three-channel information is merged to obtain a reconstructed high-resolution native domain image, so as to perform subsequent processing based on the reconstructed native domain image. By combining the interpolation of the RGB channel information guided by the white channel information, a more accurate gradient-guided interpolation is provided, the accuracy of the interpolation is improved, the false color and artificial interpolation traces on the reconstructed image are greatly reduced, the quality of the image super-resolution reconstruction is improved, and the visual effect of the image is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 Schematic diagram of the RGB domain super-resolution technology processing process;

[0050] Figure 2 A schematic diagram of a flow chart of a first embodiment of an image reconstruction method provided in an embodiment of the present application;

[0051] Figure 3 For this application Figure 2 A schematic diagram of a native domain image in the illustrated embodiment;

[0052] Figure 4 A schematic diagram of a flow chart of a second embodiment of an image reconstruction method provided in an embodiment of the present application;

[0053] Figure 5 For this application Figure 4 A schematic diagram of the interpolation process of step S401 in the illustrated embodiment;

[0054] Figure 6 For this application Figure 4 A schematic diagram of a process of obtaining red channel information after interpolation processing in the illustrated embodiment;

[0055] Figure 7 A schematic diagram of a flow chart of a third embodiment of the image reconstruction method provided in the embodiment of the present application;

[0056] Figure 8 For this application Figure 7 A schematic diagram of a first interpolation green channel information generation process in the illustrated embodiment;

[0057] Fig.9A For this application Figure 4 A schematic diagram of the first interpolation process in step S403 in the illustrated embodiment;

[0058] Fig. 9B For this application Figure 4 A schematic diagram of the second interpolation process in step S403 in the illustrated embodiment;

[0059] Fig.10 A schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of the structure of the image reconstruction model group provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] First, the terms involved in this application are explained:

[0063] CFA (Color Filter Array): Color filter array, used before the CCD (Charge Coupled Device) sensor of a digital camera, is used to remove some components in the spectrum so that each pixel only retains the color components corresponding to a specific wavelength of light, such as red, blue, and green.

[0064] Byer CFA: Bayer color filter array arrangement mode is a commonly used CFA arrangement mode. In each 2×2 array, two green light filters are located at the diagonal position, and a red light filter and a blue light filter are placed in the remaining two positions respectively, that is, it is composed of 1 / 2 green light filter, 1 / 4 red light filter and 1 / 4 blue light filter.

[0065] Demosaicking: De-mosaicing technology is used to convert the native domain image in Bayer format into an RGB domain image through an interpolation algorithm.

[0066] As users' requirements for imaging quality of imaging systems, such as mobile terminals and TV sets, continue to increase, super-resolution technology has been rapidly developed and widely used. Through super-resolution technology, high-frequency detail information in low-resolution images can be restored, thereby achieving high-resolution reconstruction of images.

[0067] Super-resolution technology is mainly divided into RGB domain super-resolution technology and Raw domain super-resolution technology.

[0068] Figure 1 It is a schematic diagram of the RGB domain super-resolution technology processing process. Figure 1 As shown, for the RGB domain super-resolution technology, since the format of the native domain image 101 collected by the imaging system camera is a Bayer format, that is, the arrangement of the pixels in the odd rows is: GRGRGR..., and the arrangement of the pixels in the even rows is: BGBGBG..., it is necessary to convert it into an RGB domain image based on the de-mosaic technology. In this process, it is necessary to supplement the information of the missing channels of each pixel through an interpolation algorithm, that is, first extract the information of each channel (including the R channel, the G channel, and the B channel) in the native domain image 101, and then obtain the complete R channel, G channel, and B channel information through interpolation, and merge the information of each channel to obtain the RGB domain image 102, and then perform super-resolution reconstruction on the RGB domain image 102, complete the image resolution improvement and image size enlargement in the RGB domain, and obtain a high-resolution reconstructed image 103 in the Bayer format. In this application, "G" indicates that the color of the pixel at the corresponding position is green, "R" indicates that the color of the pixel at the corresponding position is red, and "B" indicates that the color of the pixel at the corresponding position is blue.

[0069] The introduction of de-mosaicing technology results in irreversible artificial interpolation traces, which are further amplified in the subsequent super-resolution reconstruction process, resulting in significant pseudo-color in the reconstructed image.

[0070] In order to overcome the drawback of RGB domain super-resolution technology, namely the drawback of pseudo-color caused by demosaicing technology, a Raw domain super-resolution technology is proposed. The low-resolution Raw image is reconstructed into a high-resolution Raw domain image by directly using an interpolation algorithm.

[0071] However, since the data of low-resolution Raw images in each channel are relatively sparse, they cannot provide accurate gradients for interpolation, resulting in low interpolation accuracy. As a result, the reconstructed high-resolution images produce more noise, such as false color, blurred high-frequency details, etc., and the reconstruction results are not ideal.

[0072] In order to improve the quality of image reconstruction, the present application provides a Raw domain super-resolution image reconstruction method combined with white channel information. By combining the white channel information and the gradient of the extracted white channel information, the reconstruction of each channel information of the Raw domain image is guided, and more dimensional information is provided for gradient calculation for the sparse Raw image, thereby improving the accuracy of interpolation of each channel, thereby improving the accuracy of image super-resolution reconstruction, so that the reconstructed image exhibits significant pseudo-color suppression effect and strong detail recovery ability, thereby improving the quality of image super-resolution reconstruction.

[0073] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0074] Figure 2 The flowchart of the first embodiment of the image reconstruction method provided in the embodiment of the present application is a schematic diagram. The image reconstruction method can be executed by an image reconstruction modeling group, such as an image reconstruction chip, a processor, etc., and can also be executed by a terminal device, such as the above-mentioned imaging system, such as Figure 2 As shown, the image reconstruction method specifically includes the following steps:

[0075] S201, obtaining white channel information, green channel information, red channel information and blue channel information of a native domain image.

[0076] The native domain image is in the Bayer format, and includes information of four channels: red (R), green (G), blue (B), and white (W), namely, white channel information, green channel information, red channel information, and blue channel information.

[0077] Specifically, a native domain image may be collected based on an RGBW sensor, and the collected native domain image may be sent to an image reconstruction group to perform high-resolution reconstruction of a low-resolution native domain image.

[0078] For example, Figure 3 For this application Figure 2 The schematic diagram of the native domain image in the embodiment shown in the figure includes information of four channels, and the arrangement of the information of each channel is as follows: Figure 3 As shown in the figure, "W" indicates that the color of the pixel at the corresponding position is gray (the grayscale value can be between 0 and 255). Compared with the traditional native domain image collected by the RGB sensor, Figure 1 The native domain image 101 in the RGBW sensor has more white channel information, that is, Figure 3 The information corresponding to W in . Figure 3, the arrangement of 4N+1 rows of the native domain image captured by the RGBW sensor is: WRWGWRWG..., the arrangement of 4N+2 rows of pixels is: RWGWRWGW..., the arrangement of 4N+3 rows is: WGWBWGWB..., the arrangement of 4N+4 rows of pixels is: GWBWGWBW..., N is an integer greater than or equal to 0.

[0079] Specifically, the corresponding channel information may be extracted based on the position of each channel information, thereby obtaining the white channel information, green channel information, red channel information and blue channel information of the native domain image.

[0080] S202: Extract gradient information of the white channel information.

[0081] The gradient information is the maximum rate of change of the white channel information, that is, the W channel information, in the corresponding direction. The gradient information is used to indicate the interpolation weight and interpolation direction in the interpolation process.

[0082] In a possible implementation, the gradient of the white channel information may be extracted based on a gradient operator, such as one or more of a Sobel operator, a Canny operator, an HP (Heterogeneity Projection) operator, and the like.

[0083] Optionally, extracting gradient information of the white channel information includes:

[0084] The white channel information is up-sampled to obtain high-resolution white channel information; and gradient information of the white channel information and the high-resolution white channel information are extracted respectively.

[0085] Specifically, by upsampling the white channel information, the size of the white channel information is expanded to twice the original size, that is, the n×n white channel information is upsampled to 2n×2n high-resolution white channel information.

[0086] In a possible implementation, the dialogue channel information W can be obtained based on a bilinear interpolation algorithm, a Bicubic interpolation algorithm (also known as a bicubic interpolation algorithm), a transposed convolution algorithm, etc. LR Upsample to obtain high-resolution white channel information W SR .

[0087] Based on the Sobel operator and HP operator, the white channel information W is extracted respectively. LR And high-resolution white channel information W SR The gradient of the white channel information W is obtained LR The gradient information and high-resolution white channel information W SR The gradient information.

[0088] Among them, the Sobel operator can be defined as:

[0089]

[0090]

[0091] Among them, S0, S 90 , S 45 and S 135 They represent the Sobel operators in the horizontal (0°), vertical (90°), diagonal (45°) and anti-diagonal (135°) directions respectively.

[0092] The HP operator is defined as:

[0093]

[0094] Among them, H0 and H 90 denote the HP operators in the horizontal (0°) and vertical (90°) directions, respectively, and T denotes transpose.

[0095] Based on the above Sobel operator and HP operator respectively with the white channel information W LR and high-resolution white channel information W SR The gradient of is convolved to obtain the white channel information W LR Sobel gradient in four directions: SL LR0 , SL LR90 , SL LR45 and SL LR135 , HP gradients in two directions: HP LR0 and HP LR90 , that is, the white channel information W is obtained LR Gradient information; and high-resolution white channel information W SR Sobel gradient in four directions: SL SR0 , SL SR90 , SL SR45 and SL SR135 , HP gradients in two directions: HP SR0 and HP SR90 , that is, high-resolution white channel information W is obtained SR The gradient information.

[0096] In order to improve the diversity of gradient information, we can first LR Up-sample to obtain the white channel information mapped to high resolution, which is recorded as high resolution white channel information W SR ; Then, the original white channel information and the gradient information of the high-resolution white channel information are extracted respectively to obtain the gradient information of the white channel information and the gradient information of the high-resolution white channel information, so as to guide subsequent interpolation operations based on the gradient information of multiple dimensions to improve the accuracy of interpolation.

[0097] S203 , interpolating green channel information, red channel information, and blue channel information of the native domain image respectively according to the white channel information and the extracted gradient information.

[0098] Specifically, it can be based on the white channel information and the extracted gradient information, such as the white channel information W LR The gradient information and high-resolution white channel information W SR The gradient information of the green channel information G of the Raw domain image is LR , Red channel information R LR and blue channel information B LR Interpolation is performed to obtain the reconstructed green channel information, the reconstructed red channel information and the reconstructed blue channel information, that is, the interpolated green channel information G SR , red channel information R after interpolation SR And the blue channel information B after interpolation SR .

[0099] In a possible implementation, interpolating green channel information, red channel information, and blue channel information of the native domain image respectively according to the white channel information and the extracted gradient information includes:

[0100] The green channel information of the native domain image is interpolated according to the white channel information and the extracted gradient information; and the red channel information and the blue channel information are interpolated respectively based on the green channel information after the interpolation processing.

[0101] Specifically, based on the white channel information W LR Gradient information, dialogue channel information W LR Green channel information G LR The corresponding color difference model is adaptively interpolated to obtain complete color difference information, and then the complete color difference information is combined with the green channel information G LR Adding them together, we get the complete green channel information; then mapping the complete green channel information to the high-resolution space to get the initial high-resolution green channel information. SR The missing pixels in the initial high-resolution green channel information are interpolated to obtain the interpolated green channel information G SR .

[0102] Among them, the white channel information W LR Green channel information G LR The corresponding pixel in the color difference model is the green channel information G at the corresponding position LR The pixels and white channel information W in LR The difference in pixels.

[0103] Furthermore, based on the complete green channel information and the green channel information G after difference processing, SR , respectively for the red channel information R LR and blue channel information B LR Interpolate to obtain the red channel information R after interpolation SR And the blue channel information B after interpolation SR .

[0104] Since the G channel information accounts for a relatively high proportion in the native domain image, during image reconstruction, the W channel information is first combined to guide the high-resolution reconstruction of the G channel information. Compared with the channel reconstruction based only on the G channel itself, the accuracy of the G channel information reconstruction is improved. Then, the reconstruction of the R / B channel information is guided based on the more accurate reconstructed G channel information, thereby improving the accuracy of the reconstruction of the R / B channel information.

[0105] S204, merging the green channel information, the red channel information and the blue channel information after the interpolation processing to obtain a reconstructed native domain image.

[0106] The image reconstruction method provided in the embodiment of the present application is aimed at super-resolution reconstruction of native domain images in Bayer format. After obtaining the native domain image, the channel information of the four channels of white (W), green (G), red (R) and blue (B) of the native domain image is extracted, and the gradient information of the white channel information is extracted, and then the white channel information and the extracted gradient information are combined to guide the interpolation of the RGB three-channel information, so as to achieve high-resolution amplification of the RGB three-channel, and the interpolated RGB three-channel information is merged to obtain a reconstructed high-resolution native domain image, so as to perform subsequent processing based on the reconstructed native domain image. By combining the interpolation of the RGB channel information guided by the white channel information, a more accurate gradient-guided interpolation is provided, the accuracy of the interpolation is improved, the false color and artificial interpolation traces on the reconstructed image are greatly reduced, the quality of the image super-resolution reconstruction is improved, and the visual effect of the image is optimized.

[0107] In order to enable readers to have a deeper understanding of the implementation principle of the embodiments of the present application, the following Figures 4 to 9B right Figure 2 The illustrated embodiments are further refined or explained.

[0108] Figure 4 The flowchart of the second embodiment of the image reconstruction method provided in the embodiment of the present application is as follows: Figure 4 As shown, the above S203 can be implemented by the following steps:

[0109] S401, interpolating missing pixels in the green channel information according to the white channel information and its gradient information to obtain first interpolated green channel information.

[0110] Among them, the green channel information G LR The missing pixel is the green channel information G LR The pixels without corresponding pixel values ​​in the green channel information G LR The missing pixels are the red channel information R in the native domain image. LR and blue channel information B LR The pixel at the corresponding position.

[0111] Specifically, it can be based on an adaptive interpolation algorithm, such as an edge-aware interpolation algorithm, an edge-preserving interpolation algorithm, etc., an interpolation algorithm based on deep learning, etc., based on the white channel information W LR and its gradient information, for the green channel information G LR The missing pixels are interpolated to obtain the first interpolated green channel information, that is, the complete green channel information.

[0112] For example, Figure 5 For this application Figure 4 The schematic diagram of the interpolation process in step S401 in the embodiment shown in FIG. Figure 5 As shown, the green channel information G of the extracted native domain image LR In the image, the missing pixels account for 1 / 2, which is the red channel information R in the native domain image. LR and blue channel information B LR The corresponding pixel, combined with the white channel information W LR and its gradient information, for the green channel information G LR The missing pixels in the image are interpolated, that is, the first interpolation process is performed to obtain the complete green channel information, that is, the first difference green channel information, which is Full G LR , the first difference green channel information Full G LR The “g” in represents the pixel at the corresponding position of the green channel information estimated by interpolation.

[0113] S402: Map the first interpolated green channel information to a high-resolution grid to obtain high-resolution green channel information.

[0114] Specifically, the low-resolution first difference green channel information may be mapped to the high-resolution grid by scaling up the size to obtain the high-resolution green channel information.

[0115] In a possible implementation, the mapping rule corresponding to the high-resolution grid may be:

[0116] H(2i,2j)=L(i,j)

[0117] Wherein, L(i, j) represents the element of the i-th row and j-th column of the low-resolution channel information, which can be the green channel information GLR , Red channel information R LR Or blue channel information B LR H(2i, 2j) is the element in the ith row and jth column of the corresponding high-resolution channel information, which can be the element in the 2ith row and 2jth column of the high-resolution green channel information, the high-resolution red channel information or the high-resolution blue channel information.

[0118] S403, interpolating the high-resolution green channel information based on the high-resolution white channel information and its gradient information to obtain interpolated green channel information.

[0119] Among them, the green channel information G after interpolation processing SR Each pixel at each position in the image corresponds to a known green light pixel value, which is the complete, high-resolution green channel information. The green channel information G after interpolation processing SR Compared with the green channel information G extracted from the native domain image LR , the size is enlarged by 2 times, that is, the height and width are both twice the original, and each position corresponds to a pixel value. The green channel information G extracted from the native domain image LR Only half of the positions have corresponding pixel values.

[0120] Based on the low-resolution white channel information W LR and its gradient information, and the green channel information G LR After the first interpolation process, in order to improve the accuracy of interpolation, a second interpolation process is performed on the high-resolution green channel information based on the high-resolution white channel information and its gradient information, so that the processed green channel information is obtained through these two interpolation processes.

[0121] Specifically, it can be based on an adaptive interpolation algorithm, such as an edge-aware interpolation algorithm, an edge-preserving interpolation algorithm, etc., an interpolation algorithm based on deep learning, etc., based on high-resolution white channel information W SR and its gradient information, interpolate the high-resolution green channel information to obtain the complete high-resolution green channel information, that is, the interpolated green channel information G SR .

[0122] In a possible implementation, based on the high-resolution white channel information W SR and its gradient information, and adaptively interpolate the color difference model corresponding to the high-resolution green channel information and the high-resolution white channel information. SR Added to the interpolated color difference model, the interpolated green channel information G is obtained. SR .

[0123] Through the above two interpolation processes, the green channel information G at low resolution guided by the white channel information is realized. LR Green channel information at high resolution (green channel information after interpolation G SR ), which improves the accuracy of green channel information reconstruction or interpolation and provides an accurate, sufficient and reliable data basis for the subsequent reconstruction of the red and blue channels.

[0124] S404: For any channel information of the red channel information and the blue channel information, map the channel information to a high-resolution grid to obtain high-resolution channel information.

[0125] The red channel information and the blue channel information may be mapped to the high-resolution grids respectively with reference to the method of step S402 to obtain the high-resolution red channel information and the blue channel information. The specific process will not be described in detail here.

[0126] Since the red channel information and the blue channel information have the same proportion in the native domain image and are set in similar positions, the same interpolation process can be used to interpolate the red channel information R LR Or blue channel information B LR Interpolation is performed. In this embodiment, the red channel information R LR Taking the example for subsequent explanation, the blue channel information B can be obtained by replacing the "red channel" with the "blue channel". LR The interpolation or reconstruction process.

[0127] In one embodiment, S404 may be executed after or during any step before S405, such as before S401 or in parallel with S402. The present application does not limit the steps executed by S404. Figure 4 In the example, S404 is executed after S403.

[0128] S405 , obtaining second color difference information by subtracting the difference between the high-resolution channel information and the green channel information after the interpolation process.

[0129] The second color difference information is the same as the part of the high-resolution channel information (high-resolution red channel information or high-resolution blue channel information) where the pixel value exists, and the pixel value at a certain position in the second color difference information is the green channel information G after interpolation processing. SR The difference between the pixel value at the corresponding position and the pixel value at the corresponding position of the high-resolution channel information.

[0130] The second color difference information and the high-resolution channel information include the same number of pixel values, and the positions where the pixel values ​​exist are the same.

[0131] Taking the high-resolution red channel information as an example, the green channel information G after interpolation can be SR The pixel at the corresponding position is subtracted from each pixel with a pixel value in the high-resolution red channel information to obtain the second color difference information D r , so that the second color difference information D r Only the color difference corresponding to the pixels where the high-resolution red channel information exists or the pixels where the values ​​are retained.

[0132] S406: interpolate the second pixel in the second color difference information.

[0133] The second pixel is a pixel point in the second color difference information corresponding to the position of the red channel information or the blue channel information in the reconstructed native domain image.

[0134] Specifically, each second pixel in the second color difference information may be interpolated based on a Bilinear interpolation algorithm, so that the second color difference information includes information required for reconstruction of red channel information or blue channel information in a Bayer morphology.

[0135] S407, subtracting the interpolated green channel information from the interpolated second color difference information to obtain interpolated red channel information or interpolated blue channel information.

[0136] Taking the red channel as an example, the pixel value at the corresponding position of the green channel information after the difference processing can be subtracted from the pixel value at the position where the second color difference information after the interpolation processing has a value, so as to obtain the red channel information R after the interpolation processing. SR , that is, high-resolution red channel information.

[0137] Among them, the red channel information R after interpolation processing SR It is only necessary to ensure compliance with the Bayer morphology, and there is no need to interpolate all its positions, so as to reduce the interpolation processing time and improve the processing efficiency.

[0138] For example, Figure 6 For this application Figure 4 The schematic diagram of the process of obtaining the red channel information after interpolation processing in the embodiment shown is as follows: Figure 6 As shown, first, the low-resolution red channel information R of the extracted native domain image is LR Mapped to the high-resolution grid, the initial high-resolution red channel information R is obtained SR1 ; Then the green channel information G after interpolation SR The pixel value at the corresponding position is subtracted from the pixel value at the position where the high-resolution red channel information has a value, thereby obtaining the second color difference information D r , the second color difference information D rOnly the color difference corresponding to the position where the high-resolution red channel information has a value is retained (the green channel information G after interpolation processing SR The difference between the pixel value at the position corresponding to the high-resolution red channel information); Based on the Bilinear interpolation algorithm, the second color difference information D r The position to be interpolated and the position where the second pixel is located are interpolated to obtain the second color difference information I_D after interpolation processing r ;exist Figure 6 The position to be interpolated in is determined by the second color difference information D r The black square in the middle; then the green channel information G after interpolation processing SR The subtraction operation between the second color difference information after the difference processing is performed to obtain the reconstructed red channel information conforming to the Bayer morphology, that is, the red channel information R after the above interpolation processing SR . Figure 6 In the example, “d1” represents the second color difference information D r The original pixel in “d2” represents the estimated second color difference information D after interpolation. r The pixels in the image are obtained to obtain the second color difference information I_D r In the figure, “r” represents the reconstructed red channel information R after interpolation processing. SR The pixels in . Figure 6 It can be seen that the red channel information R after interpolation processing SR Compared with the original red channel information R LR The size is doubled. Using the same processing method, we can get the blue channel information B after interpolation processing. SR .

[0139] Then, by merging the green channel information G after interpolation SR , red channel information R after interpolation SR And the blue channel information B after interpolation SR , you can get an enlarged, high-resolution native domain image.

[0140] The high-resolution reconstruction of the red and blue channel information is guided by the completely processed green channel information. Since the green channel includes complete channel information, it provides sufficient data for constructing color difference information, which improves the accuracy of red and blue channel interpolation, thereby improving the accuracy of high-resolution reconstruction of the red and blue channel information.

[0141] Figure 7 The flowchart of the third embodiment of the image reconstruction method provided in the embodiment of the present application is as follows: Figure 7 As shown, the above S401 can be implemented by the following steps:

[0142] S701, obtaining first color difference information by subtracting the difference between the green channel information and the white channel information.

[0143] Among them, the first color difference information only has pixel values ​​at the position corresponding to the position where the green channel information has values, and the pixel value of a pixel point where the first color difference information has pixel values ​​is the difference between the pixel value at the corresponding position on the green channel information and the pixel value at the corresponding position on the white channel information.

[0144] The first color difference information and the green channel information include the same number of pixel values, and the positions where the pixel values ​​exist are the same.

[0145] S702: interpolate missing pixels in the first color difference information based on the gradient information of the white channel information.

[0146] Specifically, an adaptive interpolation algorithm may be used to interpolate missing pixels in the first color difference information based on gradient information of the channel information.

[0147] In a possible implementation, for each missing pixel in the first color difference information, the missing pixel may be interpolated according to four known pixels adjacent to the pixel based on an adaptive interpolation algorithm.

[0148] First color difference information D g1 In the figure, the missing pixel D in row i and column j is g1 The weights of the four adjacent known pixels of (i, j) are:

[0149]

[0150]

[0151]

[0152]

[0153]

[0154] Among them, w 11 、w 12 、w 13 and w 14 Corresponding to D in the first color difference information g1 (i-1, j), D g1 (i+1, j), D g1 (i, j-1) and D g1 The weights of the four pixels (i, j+1); SL LR90 (i, j) represents the Sobel operator S 90 Calculated low-resolution white channel information WLR The gradient value of the i-th row and j-th column; SL LR0 (i, j) represents the low-resolution white channel information W calculated based on the Sobel operator S0 LR The gradient value in the i-th row and j-th column.

[0155] Based on this weight, the missing pixel D g1 The estimated or interpolated value of (i, j) is:

[0156]

[0157] Where a1 = w 11 ×D g1 (i-1, j), b1 = w 12 ×D g1 (i+1, j), c1=w 13 ×D g1 (i, j-1), d1 = w 14 ×D g1 (i, j+1); HP LR90 (i, j) is based on the HP operator H 90 Calculated low-resolution white channel information W LR The gradient value of the i-th row and j-th column; HP LR0 (i, j) is the low-resolution white channel information W calculated based on the HP operator H0 LR The gradient value in the i-th row and j-th column.

[0158] S703: Add the first color difference information after the interpolation processing to the white channel information to obtain first interpolated green channel information.

[0159] For example, Figure 8 For this application Figure 7 The schematic diagram of the first interpolation green channel information generation process in the embodiment shown is as follows: Figure 8 As shown, through the green channel information G LR Subtract the white channel information W LR The pixels at the corresponding positions obtain the first color difference information D g1 , based on the extracted white channel information W LR Gradient information of the first color difference information D g1 Perform adaptive interpolation. Figure 8 In the example, the adaptive interpolation algorithm is an edge-aware interpolation algorithm, and the first color difference information after interpolation processing is obtained, that is, the complete first color difference information Full D g1 , so that the first color difference information after interpolation processing Full D g1 and white channel information W LR Add them together to get the first interpolated green channel information, that is, the complete green channel information Full WLR . Figure 8 "g1" represents the first color difference information D g1 The pixels originally present in the image, “g2” represents the first color difference information D estimated by edge-aware interpolation. g1 The pixels in the image are obtained to obtain the first color difference information Full D after interpolation. g1 "G" indicates green channel information G LR The pixels originally present in the image, “g” represents the estimated green channel information G through interpolation. LR The missing pixels in the image are then used to obtain the complete green channel information Full W LR .

[0160] In this embodiment, based on the gradient information of the white channel information, the color difference model between the white channel information and the green channel information (the first color difference information D g1 ) is interpolated, and then added to the white channel to perform low-resolution interpolation of the green channel information, which completes the missing pixels in the green channel information and obtains complete green channel information, providing a data basis for subsequent high-resolution reconstruction. And through the information subtraction, interpolation and addition method, the algorithm steps are simple and the calculation cost is low.

[0161] In a possible implementation manner, the above S403 may specifically include the following steps:

[0162] Based on the high-resolution white channel information and its gradient information, a first pixel in the high-resolution green channel information is interpolated to obtain second interpolated green channel information; wherein the first pixel is a pixel at a pixel point in the high-resolution green channel information where a pixel is missing and the sum of the horizontal and vertical coordinates is an even number; based on the high-resolution white channel information and its gradient information, the remaining missing pixels in the second interpolated green channel information are interpolated to obtain green channel information after interpolation processing.

[0163] After mapping the first interpolation green channel information to the high-resolution grid to obtain the high-resolution green channel information, in order to complete the missing pixels in the high-resolution green channel information, it is necessary to combine the high-resolution white channel information W SR The high-resolution green channel information is interpolated based on its gradient information. It is mainly divided into two interpolations. The first interpolation is used to estimate the first pixel in the high-resolution green channel information, and the second interpolation is used to estimate the remaining missing pixels in the high-resolution green channel information.

[0164] For details, please refer to Figure 7Correspondingly, the first pixel in the high-resolution green channel information is interpolated based on the high-resolution white channel information and its gradient information. That is, the third color difference information D can be obtained by subtracting the difference between the high-resolution white channel information and the high-resolution green channel information. g3 ; Based on the gradient information of the high-resolution white channel information, interpolate the first pixel in the third color difference information; add the interpolated third color difference information to the high-resolution white channel information to obtain second interpolated green channel information.

[0165] In a possible implementation, for the third color difference information D g3 For each first pixel in the image, the missing first pixel can be interpolated according to four known pixels adjacent to the first pixel based on an adaptive interpolation algorithm.

[0166] The third color difference information D g3 In the example, the missing first pixel D in row i and column j is g3 The weights of the four adjacent known pixels of (i, j) are:

[0167]

[0168]

[0169]

[0170]

[0171] Among them, w 31 、w 32 、w 33 and w 34 Corresponding to D in the third color difference information g3 (i-1, j), D g3 (i+1, j), D g3 (i, j-1) and D g3 The weights of the four pixels (i, j+1); SL SR135 (i, j) represents the Sobel operator S 135 Calculated high-resolution white channel information W SR The gradient value of the i-th row and j-th column; SL SR45 (i, j) represents the Sobel operator S 45 Calculated high-resolution white channel information W SR The gradient value in the i-th row and j-th column.

[0172] Based on this weight, pixel D g3 The estimated or interpolated value of (i, j) is:

[0173]

[0174] Where a3 = w 31 ×D g3 (i-1, j), b3 = w 32 ×D g3 (i+1, j), c3=w 33 ×D g3 (i, j-1), d3 = w 34 ×D g3 (i, j+1).

[0175] After interpolating the first pixel in the high-resolution green channel information, the second interpolation process of the high-resolution green channel information is entered, that is, based on the high-resolution white channel information and its gradient information, the remaining missing pixels in the second interpolation green channel information are interpolated to obtain the green channel information after interpolation processing.

[0176] The second interpolation process of the high-resolution green channel information can be specifically as follows:

[0177] The high-resolution white channel information W can be subtracted from the second interpolation green channel information SR The difference between g4 Based on the gradient information of the high-resolution white channel information, the missing pixels in the fourth color difference information are interpolated; the fourth color difference information after interpolation is added to the high-resolution white channel information to obtain the green channel information after interpolation, that is, the complete high-resolution green channel information G SR .

[0178] In a possible implementation, for the fourth color difference information D g4 For each missing pixel in the image, an adaptive interpolation algorithm can be used to interpolate the missing pixel according to four known pixels adjacent to the pixel.

[0179] Fourth color difference information D g4 In the figure, the missing pixel D in row i and column j is g4 The weights of the four adjacent known pixels of (i, j) are:

[0180]

[0181]

[0182]

[0183]

[0184] Among them, w 31 、w 32 、w 33 and w34 Corresponding to D in the fourth color difference information g4 (i-1, j), D g4 (i+1, j), D g4 (i, j-1) and D g4 The weights of the four pixels (i, j+1); SL SR90 (i, j) represents the Sobel operator S 90 Calculated high-resolution white channel information W SR The gradient value of the i-th row and j-th column; SL SR0 (i, j) represents the high-resolution white channel information W calculated based on the Sobel operator S0 SR The gradient value in the i-th row and j-th column.

[0185] Based on this weight, pixel D g4 The estimated or interpolated value of (i, j) is:

[0186]

[0187] Where a4 = w 41 ×D g4 (i-1, j), b4=w 42 ×D g4 (i+1, j), c4=w 43 ×D g4 (i, j-1), d4 = w 44 ×D g4 (i, j+1).

[0188] The fourth color difference information after interpolation processing is obtained through the above interpolation, that is, the complete fourth color difference information Full D g4 , and then the fourth color difference information after interpolation is Full D g4 With the high-resolution white channel information W SR Add together to get the green channel information G after interpolation SR .

[0189] Fig.9A For this application Figure 4 The schematic diagram of the first interpolation process in step S403 in the embodiment shown in FIG. Fig. 9B For this application Figure 4 The schematic diagram of the second interpolation process in step S403 in the embodiment shown in FIG. Fig.9A and 9B As shown in Figure 2, the first interpolation process is as follows:

[0190] After obtaining the high-resolution green channel information G SR1 Afterwards, first, the high-resolution green channel information G SR1 Subtract high-resolution white channel information WSR The pixel at the corresponding position in the image is obtained to obtain the third color difference information D g3 , and then through the high-resolution white channel information W SR The gradient information of the third color difference information D g3 The first pixel in the high-resolution green channel information is interpolated, that is, the pixel at the position marked by the black triangle in the high-resolution green channel information is interpolated, which can be specifically based on the third color difference information D g3 The four pixels diagonally adjacent to the pixel to be interpolated are used to estimate the pixel to be interpolated. Fig.9A In the example, the interpolation algorithm is an edge-aware interpolation algorithm, so as to obtain the third color difference information I_D after interpolation processing. g3 , the third color difference information I_D after interpolation g3 With the high-resolution white channel information W SR Add together to get the second interpolation green channel information G SR2 Among them, "r1" represents the third color difference information D g3 The pixels originally present in the image, “r2” represents the third color difference information D estimated by edge-aware interpolation. g3 The pixels in the image are obtained to obtain the third color difference information I_D after interpolation. g3 "G" represents high-resolution green channel information G SR1 The pixels originally present in the image, and “g” represents the high-resolution green channel information G estimated through the first interpolation process. SR1 The missing pixels in the image are obtained to obtain the second interpolated green channel information G SR2 .

[0191] The second interpolation process is as follows:

[0192] Continue to use the second interpolation green channel information G SR2 Subtract high-resolution white channel information W SR The pixel at the corresponding position in the image is obtained by g4 , and then through the high-resolution white channel information W SR The gradient information of the fourth color difference information D g4 The missing pixels in the image are interpolated, which can be specifically based on the fourth color difference information D g4 The missing pixel is estimated by using the four pixels above, below, left and right of the missing pixel. Fig. 9B In the example, the interpolation algorithm is an edge-aware interpolation algorithm, so as to obtain the fourth color difference information I_D after interpolation processing. g4 , the fourth color difference information I_D after interpolation g4 With the high-resolution white channel information W SR Add together to get the processed green channel information G SR, that is, the complete green channel information. Among them, "r3" represents the fourth color difference information D g4 The pixels originally present in the image, “r4” represents the fourth color difference information D estimated by edge-aware interpolation. g4 The pixels in the image are obtained to obtain the fourth color difference information I_D after interpolation. g4 "G" represents the second interpolation green channel information G SR2 The pixels originally existing in the image, "g" represents the estimated second interpolation green channel information G through the second interpolation process. SR2 The missing pixels in the image are processed to obtain the green channel information G SR .

[0193] In the present application, the symbols in the diagrams corresponding to various information channels represent pixels in the channel information, and do not represent specific pixel values. Different symbols are only used to distinguish pixels obtained through different processes.

[0194] Through the interpolation method provided in the above embodiment, combined with the high-resolution white channel information W SR The high-resolution green channel information is interpolated using its gradient, with high interpolation accuracy. In addition, the accuracy of missing pixel estimation is improved by performing interpolation twice.

[0195] Fig.10 A schematic diagram of the structure of an image reconstruction device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the image reconstruction device includes: a channel information acquisition module 1010, a white channel gradient extraction module 1020, an interpolation module 1030 and a reconstruction module 1040.

[0196] Among them, the channel information acquisition module 1010 is used to obtain white channel information, green channel information, red channel information and blue channel information of the native domain image, wherein the native domain image is in Bayer format; the white channel gradient extraction module 1020 is used to extract the gradient information of the white channel information; the interpolation module 1030 is used to interpolate the green channel information, red channel information and blue channel information of the native domain image according to the white channel information and the extracted gradient information; the reconstruction module 1040 is used to merge the green channel information, red channel information and blue channel information after interpolation processing to obtain a reconstructed native domain image.

[0197] In a possible implementation, the interpolation module 1030 includes:

[0198] A green channel interpolation unit is used to interpolate the green channel information of the native domain image according to the white channel information and the extracted gradient information; and interpolate the red channel information and the blue channel information respectively based on the interpolated green channel information.

[0199] In a possible implementation, the white channel gradient extraction module 1020 is specifically used to:

[0200] The white channel information is up-sampled to obtain high-resolution white channel information; and the gradient information of the white channel information and the high-resolution white channel information are respectively extracted.

[0201] In a possible implementation, the green channel interpolation unit includes:

[0202] According to the white channel information and its gradient information, the missing pixels in the green channel information are interpolated to obtain first interpolated green channel information; the first interpolated green channel information is mapped to a high-resolution grid to obtain high-resolution green channel information; based on the high-resolution white channel information and its gradient information, the high-resolution green channel information is interpolated to obtain interpolated green channel information.

[0203] In a possible implementation, the green channel first interpolation subunit is specifically configured to:

[0204] The first color difference information is obtained by subtracting the difference between the green channel information and the white channel information; based on the gradient information of the white channel information, the missing pixels in the first color difference information are interpolated; and the interpolated first color difference information is added to the white channel information to obtain the first interpolated green channel information.

[0205] In a possible implementation, the green channel second interpolation subunit is specifically configured to:

[0206] Based on the high-resolution white channel information and its gradient information, a first pixel in the high-resolution green channel information is interpolated to obtain second interpolated green channel information; based on the high-resolution white channel information and its gradient information, the remaining missing pixels in the second interpolated green channel information are interpolated to obtain interpolated green channel information; wherein the first pixel is a pixel at a position in the high-resolution green channel information where a pixel is missing and the sum of the horizontal and vertical coordinates is an even number.

[0207] In a possible implementation, the red and blue channel interpolation unit is specifically used to:

[0208] For any channel information of the red channel information and the blue channel information, the channel information is mapped to a high-resolution grid to obtain high-resolution channel information; a difference value of the high-resolution channel information is subtracted from the green channel information after interpolation processing to obtain second color difference information; the second color difference information is interpolated; the green channel information after interpolation processing is subtracted from the second color difference information after interpolation processing to obtain red channel information after interpolation processing or blue channel information after interpolation processing.

[0209] In a possible implementation manner, the device further includes:

[0210] The format conversion module is used to convert the reconstructed native domain image into RGB format for display.

[0211] In a possible implementation manner, the device further includes:

[0212] The image acquisition module is used to acquire the native domain image based on the RGBW sensor.

[0213] The image reconstruction device provided in the embodiment of the present application can be used to execute the image reconstruction method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be elaborated here.

[0214] Fig.11 A schematic diagram of the structure of the image reconstruction group provided in the embodiment of the present application is shown in FIG. Fig.11 The image reconstruction model group shown includes: a memory 1110 and at least one processor 1120.

[0215] The memory 1110 stores computer-executable instructions; and at least one processor 1120 executes the computer-executable instructions stored in the memory 1110 , so that the at least one processor 1120 executes the image reconstruction method provided in any embodiment of the present application.

[0216] The memory 110 and the processor 1120 are connected via a bus 1130 .

[0217] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps provided in the embodiments corresponding to the image reconstruction method of the present application, and no further elaboration will be given here.

[0218] An embodiment of the present application also provides a terminal device, including an image sensor and the image reconstruction model group provided in the above embodiment of the present application; wherein the image sensor is used to capture native domain images.

[0219] In one embodiment, the image sensor is an RGBW sensor.

[0220] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, an image reconstruction method provided in any embodiment of the present application is implemented.

[0221] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the image reconstruction method provided in any embodiment of the present application.

[0222] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image reconstruction method, characterized in that: include: Acquire white channel information, green channel information, red channel information, and blue channel information of a native domain image, wherein the native domain image is in a Bayer format; Extracting gradient information of the white channel information; Interpolate the green channel information of the native domain image according to the white channel information and the extracted gradient information; interpolate the red channel information and the blue channel information respectively based on the interpolated green channel information; The green channel information, the red channel information and the blue channel information after interpolation processing are combined to obtain a reconstructed native domain image; Interpolating green channel information of the native domain image according to the white channel information and the extracted gradient information, including: Interpolating missing pixels in the green channel information according to the white channel information and its gradient information to obtain first interpolated green channel information; The first interpolated green channel information is mapped to a high-resolution grid to obtain high-resolution green channel information; based on the high-resolution white channel information and its gradient information, the high-resolution green channel information is interpolated to obtain interpolated green channel information; wherein the high-resolution white channel information is obtained by upsampling the white channel information.

2. The method according to claim 1, characterized in that Extracting gradient information of the white channel information includes: The white channel information and the gradient information of the high-resolution white channel information are extracted respectively.

3. The method according to claim 2, characterized in that Interpolating missing pixels in the green channel information according to the white channel information and its gradient information to obtain first interpolated green channel information, including: Obtaining first color difference information according to a difference value between the green channel information and the white channel information; interpolating missing pixels in the first color difference information based on the gradient information of the white channel information; The first color difference information after interpolation processing is added to the white channel information to obtain first interpolated green channel information.

4. The method according to claim 2, characterized in that: Based on the high-resolution white channel information and its gradient information, the high-resolution green channel information is interpolated to obtain interpolated green channel information, including: Based on the high-resolution white channel information and its gradient information, interpolate the first pixel in the high-resolution green channel information to obtain second interpolated green channel information; Based on the high-resolution white channel information and its gradient information, interpolate the remaining missing pixels in the second interpolated green channel information to obtain interpolated green channel information; The first pixel is a pixel at a pixel point in the high-resolution green channel information where a pixel is missing and the sum of the horizontal and vertical coordinates is an even number.

5. The method according to any one of claims 1 to 4, characterized in that: Based on the green channel information after interpolation processing, interpolating the red channel information and the blue channel information respectively includes: For any channel information of the red channel information and the blue channel information, mapping the channel information to a high-resolution grid to obtain high-resolution channel information; Subtracting the difference between the high-resolution channel information and the green channel information after interpolation processing to obtain second color difference information; interpolating a second pixel in the second color difference information; Subtracting the interpolated green channel information from the interpolated second color difference information to obtain the interpolated red channel information or the interpolated blue channel information; The second pixel is a pixel point in the second color difference information corresponding to the position of the red channel information or the blue channel information in the reconstructed native domain image.

6. An image reconstruction device, characterized in that: include: A channel information acquisition module, used to acquire white channel information, green channel information, red channel information and blue channel information of a native domain image, wherein the native domain image is in a Bayer format; A white channel gradient extraction module, used to extract gradient information of the white channel information; The interpolation module includes: a green channel interpolation unit; The green channel interpolation unit is used to interpolate the green channel information of the native domain image according to the white channel information and the extracted gradient information; and interpolate the red channel information and the blue channel information respectively based on the interpolated green channel information; A reconstruction module is used to merge the green channel information, the red channel information and the blue channel information after interpolation processing to obtain a reconstructed native domain image; The green channel interpolation unit includes: interpolating missing pixels in the green channel information according to the white channel information and its gradient information to obtain first interpolated green channel information; mapping the first interpolated green channel information to a high-resolution grid to obtain high-resolution green channel information; interpolating the high-resolution green channel information based on the high-resolution white channel information and its gradient information to obtain interpolated green channel information; wherein the high-resolution white channel information is obtained by upsampling the white channel information.

7. An image reconstruction system, characterized in that: include: memory and at least one processor; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image reconstruction method according to any one of claims 1 to 5.

8. A terminal device, characterized in that: comprising an image sensor and the image reconstruction group as claimed in claim 7; Wherein, the image sensor is used to collect native domain images.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the image reconstruction method according to any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image reconstruction method according to any one of claims 1 to 5 is implemented.

Images