Image reconstruction method and device, electronic equipment, storage medium and chip
By performing channel separation and interpolation processing on Bayer Pattern images and combining with deep learning networks, the problems of low RGB image quality and huge computing power in the prior art are solved, and high-quality image reconstruction in intelligent mobile terminals are realized.
Patent Information
- Application Number
- CN202310592936.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the RGB image quality is not high during the image signal processing process of RAW images acquired through Bayer CFA, and due to the huge computing power, it cannot be effectively applied in smart mobile terminals.
A method of image reconstruction is proposed, by channel separation of Bayer Pattern images, R, G, and B channel images are acquired, interpolation processing and deep learning network reconstruction, and combined with Demosaic algorithm and convolutional neural network, the data volume of deep learning network is reduced and power consumption to improve RGB image quality.
The reconstructed RGB image quality is improved, the processor performance requirements and power consumption is reduced, and the image reconstruction is possible in smart mobile terminals, reducing the data processing volume and computing power requirements of deep learning networks.
Smart Images

Figure CN119027522A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image reconstruction method, device, electronic device, storage medium and chip. Background Art
[0002] In the related art, the color information of the real object is obtained by adding a color filter array (CFA) to a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. The most commonly used arrangement of CFA is Bayer CFA. In the RAW image of the Bayer pattern obtained by Bayer CFA, only one color component of R, G, and B can be obtained at each pixel, and the G component is twice the R component and the B component.
[0003] In the Image Signal Process (ISP) pipeline, the demosaicing algorithm can be used to complete the color components of each pixel, and the RGB image corresponding to the object can be obtained by channel merging. However, the quality of the obtained RGB image is not high, and due to its huge computing power, it cannot be applied to smart mobile terminals through deep learning networks. Summary of the invention
[0004] The present disclosure provides an image reconstruction method, device, electronic device, storage medium and chip to solve the problems in the related art and improve the quality of reconstructed RGB images.
[0005] A first aspect of the present disclosure provides an image reconstruction method, the method comprising:
[0006] Perform channel separation on the target image, and respectively obtain an R channel image, a G channel image, and a B channel image corresponding to the target image; wherein the target image is a Bayer Pattern image;
[0007] Performing interpolation processing on the G channel image to obtain a first reconstructed G channel image;
[0008] Based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image, obtaining a second reconstructed G channel image using the first channel image reconstruction network;
[0009] Based on the second reconstructed G channel image, R channel image and B channel image, using a second channel image reconstruction network to obtain a second reconstructed B channel image and a second reconstructed B channel image;
[0010] The second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image are channel-merged to obtain an RGB image corresponding to the target image.
[0011] In some embodiments, the channel separation of the target image to obtain an R channel image, a G channel image, and a B channel image corresponding to the target image respectively includes:
[0012] Traverse each pixel point of the target image, obtain the R or G or B pixel value corresponding to each pixel point in the target image, and the coordinate position corresponding to each pixel point;
[0013] Based on the R, G or B pixel value corresponding to each pixel point in the target image and the coordinates corresponding to each pixel point, the R channel image, G channel image and B channel image corresponding to the target image are respectively obtained.
[0014] In some embodiments, performing interpolation processing on the G channel image to obtain a first reconstructed G channel image includes:
[0015] Determine the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator;
[0016] Based on the edge directions of the pixels that need to be interpolated in the G-channel image, the G-channel image is interpolated to obtain a first reconstructed G-channel image.
[0017] In some embodiments, the interpolation processing of the G channel image based on the edge direction of the pixel points that need to be interpolated in the G channel image includes:
[0018] If the edge direction of the pixel point to be interpolated in the G channel image is a horizontal edge, interpolation is performed using the pixel values of the pixel points adjacent to the left and right of the pixel point to be interpolated;
[0019] If the edge direction of the pixel point to be interpolated in the G channel image is a vertical edge, the interpolation is performed using the pixel values of the pixel points above and below the pixel point to be interpolated.
[0020] In some embodiments, the method of obtaining a second reconstructed G channel image using a first channel image reconstruction network based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image includes:
[0021] The R channel image, the G channel image, the B channel image and the first reconstructed G channel image are taken as input, and a second reconstructed G channel image is obtained by using a first channel image reconstruction network based on a convolutional neural network.
[0022] In some embodiments, the method of obtaining second reconstructed R and B channel images based on the second reconstructed G channel image, R channel image and B channel image using a second channel image reconstruction network includes:
[0023] Based on the second reconstructed G channel image, interpolate the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image;
[0024] Based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image, a second reconstructed R channel image and a second reconstructed B channel image are acquired by using a second channel image reconstruction network.
[0025] In some embodiments, the interpolation processing is performed on the R channel image and the B channel image based on the second reconstructed G channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image, including:
[0026] Based on the second reconstructed G-channel image, determining a difference in pixel values of adjacent pixels in the second reconstructed G-channel image;
[0027] Based on the difference between pixel values of adjacent pixels in the second reconstructed G channel image, the R channel image and the B channel image are interpolated to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0028] In some embodiments, the interpolation processing is performed on the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image, including:
[0029] Traversing each pixel point in the target image, determining the first pixel point coordinates corresponding to the R channel pixel value in the target image, and the second pixel point coordinates corresponding to the B channel pixel value in the target image;
[0030] Interpolation processing is performed on the pixel points corresponding to the second pixel point coordinates in the R channel image and the pixel points corresponding to the first pixel point coordinates in the R channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0031] In some embodiments, the obtaining of the second reconstructed R channel image and the second reconstructed B channel image by using the second channel image reconstruction network based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image includes:
[0032] The second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image are taken as input, and a second reconstructed R channel image and a second reconstructed B channel image are obtained by using a second channel image reconstruction network based on a convolutional neural network.
[0033] A second aspect of the present disclosure provides an image reconstruction device, the device comprising:
[0034] A channel separation unit is used to perform channel separation on a target image, and obtain an R channel image, a G channel image and a B channel image corresponding to the target image respectively; wherein the target image is a Bayer Pattern image;
[0035] An interpolation unit, configured to perform interpolation processing on the G channel image to obtain a first reconstructed G channel image;
[0036] A first reconstruction unit is used to obtain a second reconstructed G channel image by using a first channel image reconstruction network based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image;
[0037] A second reconstruction unit is used to obtain a second reconstructed R channel image and a second reconstructed B channel image by using a second channel image reconstruction network based on the second reconstructed G channel image, R channel image and B channel image;
[0038] The merging unit is used to perform channel merging on the second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image to obtain an RGB image corresponding to the target image.
[0039] A third aspect of the present disclosure provides an electronic device, including:
[0040] at least one processor;
[0041] and a memory communicatively coupled to the at least one processor;
[0042] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method described in the embodiment of the first aspect of the present disclosure.
[0043] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0044] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.
[0045] In summary, the image reconstruction method, device, electronic device, storage medium and chip proposed in the present disclosure. The method includes: performing channel separation on the target image, and obtaining the R channel image, G channel image and B channel image corresponding to the target image respectively; wherein the target image is a Bayer Pattern image; performing interpolation processing on the G channel image to obtain a first reconstructed G channel image; based on the R channel image, G channel image, B channel image and the first reconstructed G channel image, using the first channel image reconstruction network to obtain the second reconstructed G channel image; based on the second reconstructed G channel image, R channel image and B channel image, using the second channel image reconstruction network to obtain the second reconstructed R channel image and the second reconstructed B channel image; merging the second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image through channels to obtain the RGB image corresponding to the target image. The solution provided by the present disclosure performs channel separation on the target image of the Bayer Pattern, and obtains a single-channel R channel image, a G channel image, and a B channel image, which only contains the pixel value corresponding to one channel among the R, G, and B channels, thereby reducing the amount of data of the first channel image reconstruction network and the second channel image reconstruction network, thereby reducing the learning difficulty, processor performance requirements, and power consumption of the first channel image reconstruction network and the second channel image reconstruction network, making it possible to apply image reconstruction to smart mobile terminals with limited processor performance. Furthermore, by reconstructing the G channel image to obtain the first reconstructed G channel image, not only can the quality of the second reconstructed G channel image be improved, and thus the quality of the reconstructed RGB image can be improved; by using an algorithm to reconstruct the G channel image in advance, the amount of data processing of the first channel image reconstruction network is indirectly reduced, and the power consumption of the first channel image reconstruction network is reduced.
[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0048] Figure 1A schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0049] Figure 2 A flowchart of an image reconstruction method provided by an embodiment of the present disclosure;
[0050] Figure 3 A flowchart of an image reconstruction method provided for an application example of the present disclosure;
[0051] Figure 4 A schematic diagram of an R channel image, a G channel image, and a B channel image provided by an embodiment of the present disclosure;
[0052] Figure 5 A schematic diagram of a second reconstructed G channel image provided by an embodiment of the present disclosure;
[0053] Figure 6 A schematic diagram of a second reconstructed R / B channel image provided by an embodiment of the present disclosure;
[0054] Figure 7 A schematic diagram of reconstructing an RGB image provided by an embodiment of the present disclosure;
[0055] Figure 8 A schematic diagram of the structure of an image reconstruction device provided by an embodiment of the present disclosure;
[0056] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;
[0057] Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0058] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0059] In the related art, a CCD or CMOS image sensor can be used to convert the captured light source signal of a real object into raw data of a digital signal, which is called a RAW image. Since CCD or CMOS captures the brightness information of the light source signal, the generated RAW image usually only contains the brightness information of each pixel that constitutes the image, that is, a grayscale image. However, each pixel in the full-color image corresponding to the real object is composed of three primary colors: R, G, and B. Therefore, in order to further obtain the three primary color information of each pixel, a filter must be placed in front of the CCD array or CMOS array, which only allows light of a specific color to pass through.
[0060] CFA was created to meet this demand. In the related art, the most widely used CFA is Bayer CFA. Bayer CFA consists of alternating red-green and blue-green filters. Since the human eye is most sensitive to green, the number of green filters in Bayer CFA is twice that of red and blue filters. That is, in the RAW image of the Bayer Pattern obtained by Bayer CFA, only one color component of R, G, and B can be obtained at each pixel, and the G component is twice the R component and the B component.
[0061] The process of converting RAW images to RGB images is called ISP pipeline. The core algorithm of ISP pipeline is Demosaic algorithm. In related technologies, there are two main solutions for reconstructing RGB images using Demosaic algorithm:
[0062] Solution A: First, use the interpolation method to fill in the missing R, G, or B values of each pixel in the target image to obtain the complete R, G, and B information of each pixel; then reconstruct the RGB image by channel merging.
[0063] Solution B: Use a deep learning network based on unet++ to obtain a reconstructed RGB image.
[0064] However, the above solution still has the following defects:
[0065] Solution A: In the process of interpolating and reconstructing the R channel image, the G channel image, and the B channel image, it is necessary to determine the edge direction corresponding to each pixel point and determine the interpolation direction based on the edge direction. However, since the edge detection method has poor accuracy in edge detection or detection of areas with fine textures, the zipper effect problem exists in the reconstructed image of high-frequency textures.
[0066] Solution B: When using a deep learning network to obtain a reconstructed RGB image, the associations within and between the R, G, and B channels can be accurately obtained, thereby obtaining a high-quality RGB image. However, a large number of training sets are usually required to learn the associations within and between the R, G, and B channels, and then determine appropriate parameters for the deep learning network. Therefore, the computing power required to reconstruct RGB images using a deep learning network is huge, the processor performance requirements are high, and the power consumption is high, which is usually not suitable for deployment on smart mobile terminals.
[0067] The image reconstruction method provided by the present invention combines the Demosaic algorithm with the deep learning network in artificial intelligence (AI), and reconstructs the G image channel at the front end of the deep learning network through the Demosaic algorithm, thereby reducing the amount of data of the deep learning network, reducing its power consumption and the requirements for processor performance, and at the same time utilizing the powerful learning ability of the deep learning network, thereby completing the ISP pipeline by combining the Demosaic algorithm with AI, and finally obtaining a higher quality reconstructed RGB image.
[0068] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image reconstruction method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 101, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires an image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the image 102 is sent to the target application. Among them, the electronic device 101 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.
[0069] In some optional embodiments, the electronic device 101 may also be a vehicle-mounted device or an Internet of Vehicles device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.
[0070] A camera may be installed on the electronic device 101, and an image 102 may be acquired through the installed camera. The camera may be divided into laser cameras, visible light cameras, and other types according to the different acquired images 102. The laser camera may acquire the image 102 formed by irradiating the laser onto the object, and the visible light camera may acquire the image formed by irradiating the visible light onto the object. Several cameras may be installed on the electronic device 101, and the installation positions are not limited. For example, a camera may be installed on the front panel of the electronic device 101, and two cameras may be installed on the back panel. The camera may also be installed inside the electronic device 101 in an embedded manner, and then the camera may be opened by rotating or sliding. Specifically, a front camera and a rear camera may be installed on the electronic device 101. The front camera and the rear camera may acquire the image 102 from different perspectives. Generally, the front camera may acquire the image 102 from the front perspective of the electronic device, and the rear camera may acquire the image 102 from the back perspective of the electronic device.
[0071] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.
[0072] Several applications can be installed in the electronic device 101. An application refers to software written in an electronic device for a certain application purpose. The electronic device 101 can provide services to users through the application. When the application needs to capture the image 102, an image capture instruction will be initiated. The electronic device 101 will call the camera module according to the image capture instruction to capture the image 102. The image capture instruction refers to an instruction used to trigger the image capture operation.
[0073] The electronic device 101 is also provided with a processor, and the image processing module in the processor can reconstruct the image 102 captured by the camera module, for example, execute the image reconstruction method provided in the present disclosure.
[0074] The embodiments of the present disclosure can be applied to scenes that require image processing, for example, image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, image processing in night scene shooting or video mode, etc. For another example, it can also be applied to the ISP module as a key module that affects the processor. These include but are not limited to the following scenes:
[0075] Image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, night scene shooting, image processing in video mode, etc.
[0076] Figure 2 The following is a flow chart of an image reconstruction method provided by an embodiment of the present disclosure. Figure 2 As shown, the method includes:
[0077] Step 201, performing channel separation on a target image, and respectively obtaining an R channel image, a G channel image, and a B channel image corresponding to the target image; wherein the target image is a Bayer Pattern image;
[0078] In one embodiment, in the Bayer Pattern image, due to the effect of Bayer CFA, each pixel has only the pixel value of one of the R, G, and B channels, and the pixel values of the other two channels are missing. The number of pixels with the pixel value of the G channel is twice the number of pixels with the pixel value of the R channel or the B channel.
[0079] In one embodiment, the image of the Bayer Pattern includes but is not limited to any one of the following formats:
[0080] RGGB, BGGR, GRBG, GBRG.
[0081] In one embodiment, performing channel separation on the target image to respectively obtain an R channel image, a G channel image, and a B channel image corresponding to the target image includes:
[0082] Traverse each pixel point of the target image, obtain the R or G or B pixel value corresponding to each pixel point in the target image, and the coordinate position corresponding to each pixel point;
[0083] It should be noted that among the pixels constituting the target image, half of the pixels have pixel values of the G channel, a quarter of the pixels have pixel values of the R channel, and a quarter of the pixels have pixel values of the B channel. Therefore, in the single-channel R channel image, G channel image, and B channel image, the pixel values of some pixels are missing.
[0084] Based on the R, G or B pixel value corresponding to each pixel point in the target image and the coordinates corresponding to each pixel point, the R channel image, G channel image and B channel image corresponding to the target image are respectively obtained; wherein the missing pixel values in the R channel image, G channel image and B channel image are filled with 0.
[0085] In one embodiment, OpenCV or MATLAB can also be used to achieve channel separation of the target image, which is not limited in the present disclosure. In one embodiment, other values can also be used to fill the missing pixel values in the R channel image, the G channel image, and the B channel image, such as 255, 128, etc., which are not limited in the present disclosure.
[0086] In one embodiment, based on the coordinates corresponding to each pixel point, restoring the pixel value of the corresponding channel can achieve phase alignment between the pixel point and the pixel value of the corresponding channel, thereby reducing the difficulty of reconstructing the R channel image, the G channel image, and the B channel image. And filling the missing pixel values in the R channel image, the G channel image, and the B channel image with 0 can reduce the amount of calculation for reconstructing the R channel image, the G channel image, and the B channel image.
[0087] Step 202, performing interpolation processing on the G channel image to obtain a first reconstructed G channel image;
[0088] In one embodiment, the G channel image may be interpolated using an interpolation algorithm such as a nearest neighbor interpolation method, a bilinear interpolation method, and an edge adaptive interpolation algorithm to obtain a first reconstructed G channel image. The interpolation algorithm used is not limited in the present disclosure.
[0089] Step 203, based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image, using the first channel image reconstruction network to obtain a second reconstructed G channel image;
[0090] In one embodiment, the first channel image reconstruction network may adopt one of a convolutional neural network (CNN), a recurrent neural network (RSN), or a long / short term memory (LSTM). Taking the first channel image reconstruction network adopting CNN as an example, the process of obtaining the second reconstructed G channel image using the first channel image reconstruction network is described:
[0091] First, a training set of the first channel image reconstruction network is obtained; the input of the training set is the same as that of the present disclosure, which is the R channel image, the G channel image, the B channel image and the first reconstructed G channel image, and the output of the training set is the second reconstructed G channel image; secondly, the first channel image reconstruction network with optimal parameters is trained and obtained using the training set; finally, the second reconstructed G channel image is obtained using the first channel image reconstruction network.
[0092] In one embodiment, by utilizing the powerful learning ability of the first channel image reconstruction network, the correlation information within and between the R, G, and B channels can be better learned, thereby obtaining a second reconstructed G channel image with higher quality.
[0093] Based on this, in one embodiment, based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image, obtaining the second reconstructed G channel image using the first channel image reconstruction network includes:
[0094] The R channel image, the G channel image, the B channel image and the first reconstructed G channel image are taken as input, and a second reconstructed G channel image is obtained by using a first channel image reconstruction network based on a convolutional neural network.
[0095] Step 204, based on the second reconstructed G channel image, R channel image and B channel image, using a second channel image reconstruction network to obtain a second reconstructed R channel image and a second reconstructed B channel image;
[0096] In one embodiment, the second channel image reconstruction network may adopt one of a convolutional neural network (CNN), a recurrent neural network (RSN), or a long / short term memory network (LSTM). Taking the second channel image reconstruction network adopting CNN as an example, the process of obtaining the second reconstructed R channel image and the second reconstructed B channel image using the second channel image reconstruction network is described:
[0097] First, a training set of the second channel image reconstruction network is obtained; the input of the training set is the same as that of the present disclosure, which is the second reconstructed G channel image, R channel image and B channel image, and the output of the training set is the second reconstructed R channel image and the second reconstructed B channel image; secondly, the training set is used to train and obtain the second channel image reconstruction network with optimal parameters; finally, the second channel image reconstruction network is used to obtain the second reconstructed R channel image and the second reconstructed B channel image.
[0098] In one embodiment, the powerful learning ability of the second channel image reconstruction network can be used to better learn the correlation information within and between the R, G, and B channels, thereby obtaining a higher quality second reconstructed R channel image and a second reconstructed B channel image.
[0099] Based on this, in one embodiment, based on the second reconstructed G channel image, R channel image and B channel image, obtaining a second reconstructed R channel image and a second reconstructed B channel image using a second channel image reconstruction network includes:
[0100] Based on the second reconstructed G channel image, interpolate the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image;
[0101] Based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image, a second reconstructed R channel image and a second reconstructed B channel image are acquired by using a second channel image reconstruction network.
[0102] Step 205 , merging the second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image to obtain an RGB image corresponding to the target image.
[0103] In one embodiment, the second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image may be merged through OpenCV, MATLAB, or Python to obtain an RGB image corresponding to the target image. Taking the second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image as an example, the following description is given.
[0104] The second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image are read, and the second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image are merged in the order of B, G, and R channels.
[0105] In summary, the present disclosure provides an image reconstruction method, including: performing channel separation on a target image, and respectively obtaining an R channel image, a G channel image, and a B channel image corresponding to the target image; wherein the target image is an image of a Bayer Pattern; performing interpolation processing on the G channel image, and obtaining a first reconstructed G channel image; based on the R channel image, the G channel image, the B channel image, and the first reconstructed G channel image, using a first channel image reconstruction network to obtain a second reconstructed G channel image; based on the second reconstructed G channel image, the R channel image, and the B channel image, using a second channel image reconstruction network to obtain a second reconstructed R channel image and a second reconstructed B channel image; performing channel merging on the second reconstructed R channel image, the second reconstructed G channel image, and the second reconstructed B channel image, and obtaining an RGB image corresponding to the target image.
[0106] The solution provided by this disclosure,
[0107] Firstly, by performing channel separation on the target image of the Bayer Pattern, the obtained single-channel R channel image, G channel image and B channel image only contain the pixel value corresponding to one of the R, G and B channels, thereby reducing the data volume of the first channel image reconstruction network and the second channel image reconstruction network, thereby reducing the learning difficulty of the first channel image reconstruction network and the second channel image reconstruction network, the requirements on processor performance and the power consumption, making it possible to apply image reconstruction in smart mobile terminals with limited processor performance.
[0108] Secondly, by reconstructing the G channel image to obtain the first reconstructed G channel image, not only can the quality of the second reconstructed G channel image be improved, thereby improving the quality of the reconstructed RGB image; but also by using the algorithm to reconstruct the G channel image in advance, the data processing amount of the first channel image reconstruction network is indirectly reduced, and the power consumption of the first channel image reconstruction network is reduced.
[0109] In one embodiment, in the above step 202, an interpolation algorithm such as a nearest neighbor interpolation method, a bilinear interpolation method, and an edge adaptive interpolation algorithm may be used to perform interpolation processing on the G channel image to obtain a first reconstructed G channel image.
[0110] However, in the nearest neighbor interpolation algorithm, the pixel points to be interpolated in the G channel image are mapped to the target image of the Bayer Pattern, and the pixel values of the pixels corresponding to the integer coordinates closest to the pixel points to be interpolated in the target image are obtained as the pixel values of the pixel points to be interpolated. This results in an obvious block effect in the G channel image after interpolation processing using the nearest neighbor interpolation algorithm, and the image quality is not high.
[0111] In the bilinear interpolation method, the average value of the G pixel values of the four pixels adjacent to the pixel to be interpolated in the G channel image is directly used to interpolate the pixel to be interpolated, without considering the direction of interpolation and ignoring the correlation between channels. This results in a large number of zipper effects and false color effects at the edge of the image.
[0112] The adaptive interpolation algorithm determines the edge direction of the pixel that needs to be interpolated through the pixel values of the R channel or B channel corresponding to the pixels around the pixel that needs to be interpolated in the G channel image. Therefore, the correlation between the color channels is taken into account, reducing the zipper effect and pseudo-color effect in the first reconstructed G channel image.
[0113] Based on this, in one embodiment, the interpolation processing is performed on the G channel image to obtain a first reconstructed G channel image, including:
[0114] Determine the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator;
[0115] In one embodiment, the edge detection operator may adopt the edge detection operator proposed by Hamilton and Adams, that is, an edge detection operator composed of the gradient of the luminance signal and the second-order differential of the chrominance signal.
[0116] Based on this, in one embodiment, determining the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator includes:
[0117] The edge direction of the pixel points that need to be interpolated in the G channel image is determined by a Hamilton edge detection operator.
[0118] Based on the edge directions of the pixels that need to be interpolated in the G-channel image, the G-channel image is interpolated to obtain a first reconstructed G-channel image.
[0119] In one embodiment, after the edge direction of the pixel points that need to be interpolated in the G channel image is determined, the pixel points that need to be interpolated can be interpolated along the edge direction.
[0120] Based on this, in one embodiment, the interpolation processing of the G channel image based on the edge direction of the pixel points that need to be interpolated in the G channel image includes:
[0121] If the edge direction of the pixel point to be interpolated in the G channel image is a horizontal edge, interpolation is performed using the pixel values of the pixel points adjacent to the left and right of the pixel point to be interpolated;
[0122] If the edge direction of the pixel point to be interpolated in the G channel image is a vertical edge, the interpolation is performed using the pixel values of the pixel points above and below the pixel point to be interpolated.
[0123] In one embodiment, after the interpolation processing of the G channel image is completed and the first reconstructed G channel image is obtained, based on the color difference constant theory, the R channel image and the B channel image can be interpolated on the basis of the first reconstructed G channel image to obtain the first reconstructed R channel image and the first reconstructed B channel image. The color difference constant theory is that for adjacent pixel points, the difference of the pixel values of the single channel is the same.
[0124] Based on this, in one embodiment, interpolating the R channel image and the B channel image based on the second reconstructed G channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image includes:
[0125] Based on the second reconstructed G-channel image, determining a difference in pixel values of adjacent pixels in the second reconstructed G-channel image;
[0126] Based on the difference in pixel values of adjacent pixels in the second reconstructed G channel image and according to the color difference constancy theory, the R channel image and the B channel image are interpolated to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0127] In one embodiment, in order to reduce the amount of data that needs to be processed by the second channel image reconstruction network, when interpolating the R channel image and the B channel image, only one quarter of the number of pixels is interpolated. Thus, not only can the amount of data that needs to be processed by the second channel image reconstruction network be reduced, but the quality of the second reconstructed R channel image and the second reconstructed B channel image can also be improved, that is, by interpolating only one quarter of the number of pixels, the power consumption of the second channel image reconstruction network and the quality of the acquired image are effectively balanced.
[0128] In one embodiment, considering the correlation between different color channels, the one-quarter number of pixels may be selected from pixels having R channel pixel values or pixels having B channel pixel values in the target image.
[0129] Based on this, in one embodiment, the interpolation processing is performed on the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image, including:
[0130] Traversing each pixel point in the target image, determining the first pixel point coordinates corresponding to the R channel pixel value in the target image, and the second pixel point coordinates corresponding to the B channel pixel value in the target image;
[0131] Interpolation processing is performed on the pixel points corresponding to the second pixel point coordinates in the R channel image and the pixel points corresponding to the first pixel point coordinates in the R channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0132] The image reconstruction method provided by the present disclosure,
[0133] First, CNN was introduced when obtaining the second reconstructed G channel image, the second reconstructed B channel image and the second reconstructed R channel image. Based on the powerful learning ability of CNN, it can better learn the correlation within and between R, G, and B channels, thereby making the reconstructed RGB image better than the deployable Demosaic algorithm.
[0134] Secondly, through the phase alignment operation in the channel separation process, the difficulty of reconstructing the R, G, and B channel images is reduced, and the amount of data that needs to be processed when reconstructing the R, G, and B channel images is reduced, which provides convenience for using CNN to implement the ISPpipeline in smart mobile terminals with limited processor performance.
[0135] Again, before using the first channel image reconstruction network to obtain the second reconstructed G channel image, the edge detection algorithm is used to determine the edge direction of the pixel points that need to be interpolated, and the first reconstructed G channel image is obtained through an adaptive interpolation algorithm, which lays a solid foundation for using the first channel image reconstruction network to obtain high-quality second reconstructed G channel image, and indirectly solves the problem of insufficient computing power of the first channel image reconstruction network.
[0136] Again, through the color difference constancy theory in the Demosaic algorithm, on the basis of the high-quality second reconstructed G channel image, the interpolation processing of one-quarter pixel points of the R channel image and the B channel image was completed in advance through the color difference method, and the first reconstructed R channel image and the first reconstructed B channel image were obtained, effectively removing the edge pseudo-color effect in the reconstructed RGB image.
[0137] In summary, the image reconstruction method provided by the present invention combines the edge direction judgment, color difference constant prior and powerful learning ability of CNN in the Demosaic algorithm, and can obtain higher quality reconstructed RGB images; and the first channel image reconstruction network and the second channel image reconstruction network need to process a small amount of data, low power consumption, and low performance requirements for the processor, and can be deployed in smart mobile terminals.
[0138] The image reconstruction method provided by the present disclosure is further illustrated below with an application example.
[0139] like Figure 3 As shown, the image reconstruction method provided by the application example of the present disclosure includes the following steps:
[0140] Step 301, performing channel separation on the target image, and obtaining R, G, and B channel images corresponding to the target image respectively;
[0141] like Figure 4 As shown, the image format of the Bayer Pattern in the application example of the present disclosure is the BGGR format, and after separation, 0 is used to fill the missing pixel values in the single-channel R, G, and B channel images.
[0142] Step 302, determining the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator;
[0143] In the application example of the present disclosure, the pixel point G that needs to be interpolated in the G channel image i,j Horizontal edge detection operator DH i,j for:
[0144] DH i,j =ABS(2*C i,j -G i,j-2 -G i,j+2 )+ABS(G i,j+1 -G i,j-1 ).
[0145] Specifically, the pixel point G that needs to be interpolated in the G channel image i,j Vertical edge detection operator DV i,j for:
[0146] DV i,j =ABS(2*C i,j -G i-2,j -G i+2,j )+ABS(G i+1,j -G i-1,j ).
[0147] Step 303, interpolating the G channel image based on the edge directions of the pixels to be interpolated in the G channel image to obtain a first reconstructed G channel image;
[0148] In the application example of this disclosure,
[0149] Step 304, taking the R, G, B channel images and the first reconstructed G channel image as input, and obtaining a second reconstructed G channel image by using a first channel image reconstruction network based on a convolutional neural network;
[0150] In the application example of the present disclosure, the R, G, B channel images and the first reconstructed G channel image are merged into four channels as input of the first channel image reconstruction network; wherein, in the first channel image reconstruction network based on the convolutional neural network, two convolutional layers as shown in the following Table 1 are used:
[0151] layer input output kernel stride padding Conv_1 1*4*H*W 1*8*H*W 8*4*3*3 1 1 Conv_2 1*8*H*W 1*1*H*W 1*4*3*3 1 1
[0152] Table 1 Convolutional layers of the first channel image reconstruction network
[0153] In Table 1, i.e., in the first convolution layer Conv_1 and the second convolution layer Conv_2, the sliding stride of the convolution kernel with a scale of 3*3 is 1, and the edge length is 1. H*W represents the size of the input image.
[0154] like Figure 5 As shown, Figure 5 The image on the left is a reference image (real image) of the G channel image, the middle image is a second reconstructed G channel image obtained using step 304 of the application example of the present disclosure, and the image on the right is a reconstructed G channel image using a low-computing-power network without resolving the zipper effect. Figure 5 In the figure, the edge of the right image clearly has a zipper effect, while the edge of the middle image does not have a zipper effect. Figure 5 It can be seen that the method for obtaining the second reconstructed G channel image in step 304 of the application example of the present disclosure can effectively eliminate the zipper effect in the second reconstructed G channel image.
[0155] Step 305, performing quarter pixel interpolation processing on the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image;
[0156] In the disclosed embodiment, each pixel point in the target image is traversed to determine the first pixel point coordinates corresponding to the R channel pixel value in the target image and the second pixel point coordinates corresponding to the B channel pixel value in the target image; according to the color difference constancy theory, the pixel points corresponding to the second pixel point coordinates in the R channel image and the pixel points corresponding to the first pixel point coordinates in the B channel image are interpolated.
[0157] In the embodiment of the present disclosure, the pixel point R corresponding to the second pixel point coordinate in the R channel image 2i,2j for:
[0158]
[0159] The pixel points corresponding to the coordinates of the first pixel point in the B channel image are interpolated using the same method, which will not be repeated here.
[0160] Step 306, based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image, using a second channel image reconstruction network to obtain second reconstructed R and B channel images;
[0161] In the embodiment of the present disclosure, in the second channel image reconstruction network based on the convolutional neural network, the following is used
[0162] The two convolutional layers shown in Table 2:
[0163] layer input output kernel stride padding Conv_1 1*3*H*W 1*16*H*W 16*3*3*3 1 1 Conv_2 1*16*H*W 1*2*H*W 2*16*3*3 1 1
[0164] Table 1 Convolutional layers of the first channel image reconstruction network
[0165] In Table 1, i.e., in the first convolution layer Conv_1 and the second convolution layer Conv_2, the sliding stride of the convolution kernel with a scale of 3*3 is 1, and the edge length is 1. H*W represents the size of the input image.
[0166] like Figure 6 As shown, Figure 6 The left image is a reference image of the R / B channel image, the middle image is a second reconstructed R / B channel image without using the second channel image reconstruction network, and the right image is a second reconstructed R / B channel image obtained using step 306 of the example application of the present disclosure. Figure 6 In the figure, the middle image clearly has a pseudo-color effect, while the image on the right does not have a pseudo-color effect. Figure 6 It can be seen that the method for obtaining the second reconstructed R / B channel image by the step 306 of the application example of the present disclosure can effectively eliminate the pseudo-color effect in the second reconstructed R / B channel image.
[0167] Step 307: sort and merge the second reconstructed R, G, and B channel images to obtain an RGB image corresponding to the target image.
[0168] like Figure 7 As shown, Figure 7 In the figure, the upper left image is the reconstructed RGB image, the upper right image is the second reconstructed B channel image, the lower left image is the second reconstructed R channel image, and the lower right image is the second reconstructed G channel image. Figure 7 It can be seen that in the image reconstruction method provided by the application example of the present disclosure, there is no zipper effect and pseudo-color effect in the reconstructed RGB image, and the image quality is high.
[0169] like Figure 8 As shown, the embodiment of the present disclosure further provides an image reconstruction device, the image reconstruction device 900 comprising:
[0170] The channel separation unit 801 is used to perform channel separation on the target image, and obtain an R channel image, a G channel image and a B channel image corresponding to the target image respectively; wherein the target image is a Bayer Pattern image;
[0171] An interpolation unit 802 is used to perform interpolation processing on the G channel image to obtain a first reconstructed G channel image;
[0172] A first reconstruction unit 803 is used to obtain a second reconstructed G channel image by using a first channel image reconstruction network based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image;
[0173] A second reconstruction unit 804 is used to obtain a second reconstructed B channel image and a second reconstructed B channel image using a second channel image reconstruction network based on the second reconstructed G channel image, R channel image and B channel image;
[0174] The merging unit 805 is used to perform channel merging on the second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image to obtain an RGB image corresponding to the target image.
[0175] In one embodiment, the channel separation unit 801 is specifically used for:
[0176] Traverse each pixel point of the target image, obtain the R or G or B pixel value corresponding to each pixel point in the target image, and the coordinate position corresponding to each pixel point;
[0177] Based on the R, G or B pixel value corresponding to each pixel point in the target image and the coordinates corresponding to each pixel point, the R channel image, G channel image and B channel image corresponding to the target image are respectively obtained; wherein the missing pixel values in the R channel image, G channel image and B channel image are filled with 0.
[0178] In one embodiment, the interpolation unit 802 is specifically configured to:
[0179] Determine the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator;
[0180] Based on the edge directions of the pixels that need to be interpolated in the G-channel image, the G-channel image is interpolated to obtain a first reconstructed G-channel image.
[0181] In one embodiment, the interpolation unit 802 is specifically configured to:
[0182] The edge direction of the pixel points that need to be interpolated in the G channel image is determined by a Hamilton edge detection operator.
[0183] In one embodiment, the interpolation unit 802 is specifically configured to:
[0184] If the edge direction of the pixel point to be interpolated in the G channel image is a horizontal edge, interpolation is performed using the pixel values of the pixel points adjacent to the left and right of the pixel point to be interpolated;
[0185] If the edge direction of the pixel point to be interpolated in the G channel image is a vertical edge, the interpolation is performed using the pixel values of the pixel points above and below the pixel point to be interpolated.
[0186] In one embodiment, the first reconstruction unit 803 is specifically configured to:
[0187] The R channel image, the G channel image, the B channel image and the first reconstructed G channel image are taken as input, and a second reconstructed G channel image is obtained by using a first channel image reconstruction network based on a convolutional neural network.
[0188] In one embodiment, the second reconstruction unit 804 is specifically configured to:
[0189] Based on the second reconstructed G channel image, interpolate the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image;
[0190] Based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image, a second reconstructed R channel image and a second reconstructed B channel image are acquired by using a second channel image reconstruction network.
[0191] In one embodiment, the second reconstruction unit 804 is specifically configured to:
[0192] Based on the second reconstructed G-channel image, determining a difference in pixel values of adjacent pixels in the second reconstructed G-channel image;
[0193] Based on the difference in pixel values of adjacent pixels in the second reconstructed G channel image and according to the color difference constancy theory, the R channel image and the B channel image are interpolated to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0194] In one embodiment, the second reconstruction unit 804 is specifically configured to:
[0195] Traversing each pixel point in the target image, determining the first pixel point coordinates corresponding to the R channel pixel value in the target image, and the second pixel point coordinates corresponding to the B channel pixel value in the target image;
[0196] Interpolation processing is performed on the pixel points corresponding to the second pixel point coordinates in the R channel image and the pixel points corresponding to the first pixel point coordinates in the R channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image.
[0197] In one embodiment, the merging unit 805 is specifically configured to:
[0198] The second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image are taken as input, and a second reconstructed R channel image and a second reconstructed B channel image are obtained by using a second channel image reconstruction network based on a convolutional neural network.
[0199] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.
[0200] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0201] Figure 8is a block diagram of an electronic device 900 for implementing the above-mentioned image reconstruction method according to an exemplary embodiment. For example, the electronic device 900 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0202] Reference Fig. 9 , the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .
[0203] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.
[0204] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0205] The power supply component 906 provides power to the various components of the electronic device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.
[0206] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0207] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.
[0208] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0209] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0210] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0211] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0212] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the electronic device 900 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0213] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image reconstruction method described in the above embodiment of the present disclosure.
[0214] The embodiment of the present disclosure also provides a chip, which can be seen in Fig.10 Schematic diagram of the chip structure shown. Fig.10 The chip shown includes a processor 1001 and an interface 1002. The number of processors 1001 may be one or more, and the number of interfaces 1002 may be multiple. The interface 1002 is used to receive a signal from a memory of an electronic device and send the signal to the processor 1001, wherein the signal includes a computer instruction stored in the memory.
[0215] Optionally, the chip further includes a memory 1003, and the memory 1003 is used to store necessary computer instructions and data. When the processor 1001 executes the computer instructions, the electronic device executes the image reconstruction method described in the above embodiment of the present disclosure.
[0216] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0217] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0218] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0219] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0220] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0221] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0222] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0223] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An image reconstruction method, characterized in that: include: Perform channel separation on the target image, and respectively obtain an R channel image, a G channel image, and a B channel image corresponding to the target image; wherein the target image is an image of a Bayer Pattern; Performing interpolation processing on the G channel image to obtain a first reconstructed G channel image; Based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image, obtaining a second reconstructed G channel image using the first channel image reconstruction network; Based on the second reconstructed G channel image, R channel image and B channel image, using a second channel image reconstruction network to obtain a second reconstructed B channel image and a second reconstructed B channel image; The second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image are channel-merged to obtain an RGB image corresponding to the target image.
2. The method according to claim 1, characterized in that The channel separation of the target image to obtain an R channel image, a G channel image and a B channel image corresponding to the target image respectively includes: Traverse each pixel point of the target image, obtain the R or G or B pixel value corresponding to each pixel point in the target image, and the coordinate position corresponding to each pixel point; Based on the R, G or B pixel value corresponding to each pixel point in the target image and the coordinates corresponding to each pixel point, the R channel image, G channel image and B channel image corresponding to the target image are respectively obtained.
3. The method according to claim 1, characterized in that The interpolation process is performed on the G channel image to obtain a first reconstructed G channel image, including: Determine the edge direction of the pixel points that need to be interpolated in the G channel image by using an edge detection operator; Based on the edge directions of the pixels that need to be interpolated in the G-channel image, the G-channel image is interpolated to obtain a first reconstructed G-channel image.
4. The method according to claim 3, characterized in that The interpolation process of the G channel image based on the edge direction of the pixel points that need to be interpolated in the G channel image includes: If the edge direction of the pixel point to be interpolated in the G channel image is a horizontal edge, interpolation is performed using the pixel values of the pixel points adjacent to the left and right of the pixel point to be interpolated; If the edge direction of the pixel point to be interpolated in the G channel image is a vertical edge, the interpolation is performed using the pixel values of the pixel points above and below the pixel point to be interpolated.
5. The method according to claim 1, characterized in that The method of obtaining a second reconstructed G channel image by using a first channel image reconstruction network based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image includes: The R channel image, the G channel image, the B channel image and the first reconstructed G channel image are taken as input, and a second reconstructed G channel image is obtained by using a first channel image reconstruction network based on a convolutional neural network.
6. The method according to claim 1, characterized in that The method of obtaining second reconstructed R and B channel images based on the second reconstructed G channel image, R channel image and B channel image by using a second channel image reconstruction network includes: Based on the second reconstructed G channel image, interpolate the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image; Based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image, a second reconstructed R channel image and a second reconstructed B channel image are acquired by using a second channel image reconstruction network.
7. The method according to claim 6, characterized in that: The interpolation processing is performed on the R channel image and the B channel image based on the second reconstructed G channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image, including: Based on the second reconstructed G-channel image, determining a difference in pixel values of adjacent pixels in the second reconstructed G-channel image; Based on the difference between pixel values of adjacent pixels in the second reconstructed G channel image, the R channel image and the B channel image are interpolated to obtain a first reconstructed R channel image and a first reconstructed B channel image.
8. The method according to claim 7, characterized in that: The interpolation processing is performed on the R channel image and the B channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image, including: Traversing each pixel point in the target image, determining the first pixel point coordinates corresponding to the R channel pixel value in the target image, and the second pixel point coordinates corresponding to the B channel pixel value in the target image; Interpolation processing is performed on the pixel points corresponding to the second pixel point coordinates in the R channel image and the pixel points corresponding to the first pixel point coordinates in the R channel image to obtain a first reconstructed R channel image and a first reconstructed B channel image.
9. The method according to claim 6, characterized in that: The method of obtaining a second reconstructed R channel image and a second reconstructed B channel image by using a second channel image reconstruction network based on the second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image includes: The second reconstructed G channel image, the first reconstructed R channel image and the first reconstructed B channel image are taken as input, and a second reconstructed R channel image and a second reconstructed B channel image are obtained by using a second channel image reconstruction network based on a convolutional neural network.
10. An image reconstruction device, characterized in that: include: A channel separation unit is used to perform channel separation on a target image, and obtain an R channel image, a G channel image and a B channel image corresponding to the target image respectively; wherein the target image is an image of a Bayer Pattern; An interpolation unit, used for performing interpolation processing on the G channel image to obtain a first reconstructed G channel image; A first reconstruction unit is used to obtain a second reconstructed G channel image by using a first channel image reconstruction network based on the R channel image, the G channel image, the B channel image and the first reconstructed G channel image; A second reconstruction unit is used to obtain a second reconstructed R channel image and a second reconstructed B channel image by using a second channel image reconstruction network based on the second reconstructed G channel image, R channel image and B channel image; The merging unit is used to perform channel merging on the second reconstructed R channel image, the second reconstructed G channel image and the second reconstructed B channel image to obtain an RGB image corresponding to the target image.
11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.
13. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 9.
