Image re-mosaic method

Abstract

The invention provides an image re-mosaicing method, which comprises the following steps of: acquiring an original Bayer format image, and interpolating to obtain an RGB (Red, Green, Blue) three-channel image; re-sampling the RGB three-channel image to obtain a multi-unit single-phase format image; performing degradation processing on the multi-unit single-phase format image to obtain a simulation multi-unit four-phase format image; establishing a mapping network structure from a simulation multi-unit four-phase format image to an RGB three-channel image, and training to obtain an optimized offline reasoning model; a real multi-unit four-phase format image collected randomly is preprocessed and then input into the optimized offline reasoning model, an RGB three-channel image is output, and a bayer image is obtained through re-sampling. According to the method, aiming at the structural characteristics of the QPD, based on the texture image degradation strategy, the output quality of the multi-unit four-phase format image is improved by utilizing the difference of the light splitting pixels, the complexity is reduced, and the image quality is good.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image re-mosaic method. Background Art

[0002] The resolution of current mobile phone image sensors is getting higher and higher, and high-pixel products have become the mainstream. The color filter array (CFA) format used by image sensors has gradually evolved from a Bayer single-cell color filter array to a multi-cell (Cell, abbreviated as C) color filter array, such as 4C, 16C, etc., and each primary color array group (containing N*N pixels) of the multi-cell CFA has the same color component, such as Figure 1 In the 16C-QPD multi-unit color filter array shown in the figure, each 4*4 pixel point constitutes a multi-unit (16C) of the primary color array group, in which adjacent 2x2 pixel blocks use the same color filter and share a microlens. This new full-pixel spectroscopic structure is referred to as QPD. The advantages of image sensors using this structure are: (1) Since each pixel carries phase information, it has a great advantage in autofocus performance; (2) According to the brightness of the different shooting environments, the 2x2 pixel blocks under the same microlens can be merged to increase the amount of light entering.

[0003] The Raw data collected by the image sensor needs to be processed by the back-end image signal processor and finally presented to the user. Currently, the general image signal processor only supports Bayer format input. In practical applications, non-Bayer format data must be rearranged into the Bayer format of the same resolution, that is, re-mosaic.

[0004] There are some difficulties in re-mosaicing images in multi-unit-QPD (e.g., 16C-QPD) format into Bayer images: (1) Using traditional algorithms, since the pixels under a microlens of 16C-QPD have spectral characteristics and large brightness differences, there will be defects such as uneven spectral distribution, distortion, and shadows between pixels, resulting in reduced resolution and image quality. In addition, the width of 16 pixels is large, resulting in reduced credibility of pixels restored using image interpolation algorithms. (2) Using deep learning methods, since there is no relevant 16C-QPD dataset, the industry usually uses downsampling combined with super-resolution methods, which has poor results. (3) The high pixel count of 16C-QPD leads to high algorithm complexity and running time: currently, a frame of 200M pixel image takes up to 6-7 seconds.

[0005] A re-mosaic method merges every 4 pixels into 1 pixel, and uses the average value of the 4 pixels as the pixel value of the merged pixel. The difference of the 4 pixels is eliminated through the averaging operation. This method will lose a lot of image details. Especially in the edge area (texture area) of the image, a lot of details will be lost through the averaging operation. Merging pixels will cause the original information to be lost, and additional model computing power is required to supplement it through global information, which greatly increases the complexity. In addition, merging pixels will cause high-frequency textures at the pixel level, and weak textures will change, making them difficult to recover. Summary of the invention

[0006] The object of the present invention is to provide an image re-mosaicing method, which is used for a pixel re-mosaicing method of a multi-unit four-phase (QPD) format with a full-pixel spectroscopic structure. The present invention is based on the structural characteristics of QPD and a texture-based image degradation strategy, which can make good use of the differences in spectroscopic pixels, thereby greatly improving the output quality of the multi-unit four-phase (QPD) format image at the original size. The present invention has lower complexity and better image quality than traditional interpolation methods.

[0007] The present invention provides an image re-mosaic method, comprising:

[0008] S1, using an image sensor to capture and obtain original Bayer format images in multiple scenes; one filtering microlens in the image sensor corresponds to one pixel;

[0009] S2, performing interpolation and white balance processing on the Bayer format image to obtain an RGB three-channel image in a linear domain; the RGB three-channel image is used as a true label of training data;

[0010] S3, resampling the RGB three-channel image to obtain a multi-unit single-phase format image;

[0011] S4, performing degradation processing on the multi-unit single-phase format image to obtain a simulated multi-unit four-phase format image;

[0012] S5, establishing a mapping network structure from the simulated multi-unit four-phase format image to the RGB three-channel image, and performing optimization training on the mapping network structure to obtain an optimized offline inference model;

[0013] S6, preprocessing the randomly collected real multi-unit four-phase format image and inputting it into the optimized offline inference model to output an RGB three-channel image;

[0014] S7, resampling the output RGB three-channel image to obtain a bayer image with the same resolution, and completing re-mosaicing.

[0015] Furthermore, the multi-unit single-phase format includes any one of 4-unit single-phase, 16-unit single-phase and 32-unit single-phase.

[0016] Furthermore, the multi-unit single-phase format image includes a plurality of first filter array groups, one of which includes four primary color array groups: a blue primary color array group, two green primary color array groups and one red primary color array group; each pixel in the primary color array group corresponds to a filtering microlens; in the 4-unit single-phase format image, each primary color array group includes 4 pixels; in the 16-unit single-phase format image, each primary color array group includes 16 pixels; in the 32-unit single-phase format image, each primary color array group includes 32 pixels.

[0017] Furthermore, the simulated multi-unit four-phase format includes: any one of simulated 4-unit four-phase, simulated 16-unit four-phase and simulated 32-unit four-phase.

[0018] Furthermore, the simulated multi-unit four-phase format image includes a plurality of second filter array groups, one of which includes four primary color array groups: one blue primary color array group, two green primary color array groups and one red primary color array group; and every four pixels in the primary color array group share one filter microlens;

[0019] In the simulated 4-unit four-phase format image, each of the primary color array groups includes 4 pixels; in the simulated 16-unit four-phase format image, each of the primary color array groups includes 16 pixels; in the simulated 32-unit four-phase format image, each of the primary color array groups includes 32 pixels.

[0020] Furthermore, step S4 specifically includes: in the multi-unit single-phase format image, taking the 2x2 pixel block of the same channel as the unit, performing texture and gradient detection in the MxN window; the 2x2 pixel block is located in the MxN window; respectively calculating the image variance Var, the horizontal gradient Gh, and the vertical gradient Gv in the current window; setting the image variance threshold to K; the pixel values ​​of the four pixels of the 2x2 pixel block are P1, P2, P3 and P4 respectively.

[0021] Further, in step S4, when the image variance Var in the window is less than the image variance threshold K, the 2x2 pixel block is within the smooth region confidence level; the degradation processing includes step S41, which specifically includes:

[0022] S411, calculating an average value of the four pixels, and using the average pixel value P to replace the original four pixel values ​​respectively, so as to make up the brightness difference of the four pixels;

[0023] S412. Add a random compensation value to each of the four average pixel values ​​P, wherein the random compensation value conforms to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel bit width. Each of the average pixel values ​​P is added to the corresponding random compensation value to obtain four compensated pixel values, which are P1', P2', P3' and P4'.

[0024] Further, in step S4, when the image variance Var in the window> the image variance threshold K, the 2x2 pixel block is within the texture region confidence level; the degradation processing includes step S42, which specifically includes:

[0025] S421, generate four compensation values ​​C P1 , C P2 , C P3 and C P4 , the four compensation values ​​conform to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel bit width;

[0026] S422, determine the gradient direction of the current 2x2 pixel block, when the horizontal gradient Gh> the vertical gradient Gv, it is necessary to constrain the compensation value to conform to C P1 >C P3 , C P2 >C P4 or C P1 <C P3 ,C P2 <C P4 ;

[0027] When the horizontal gradient Gh<the vertical gradient Gv, it is necessary to constrain the compensation value to conform to C P1 >C P2 , C P3 >C P4 or C P1 <C P2 ,C P3 <C P4 ;

[0028] S423, adding the pixel values ​​of the four pixels to their corresponding compensation values ​​to obtain four compensated pixels, which are P1, b 、P2 b 、P3 b 、P4 b ;and

[0029] P1 b =P1+C P1 ;P2 b =P2+CP2 ;P3 b =P3+C P3 ;P4 b =P4+C P4 .

[0030] Furthermore, in step S4, in the multi-unit single-phase format image, all the 2x2 pixel blocks are subjected to degradation processing according to step S41 or step S42 to obtain the simulated multi-unit four-phase format image;

[0031] A first brightness mean of the entire image of the multi-unit single-phase format image before degradation is calculated, a second brightness mean of the entire image of the simulated multi-unit four-phase format image after degradation is calculated, a global brightness compensation value is set, and the second brightness mean is compensated so that the second brightness mean after global compensation is consistent with the first brightness mean, that is, the brightness mean of the image after degradation is consistent with that before degradation.

[0032] Furthermore, in step S6, in the real multi-unit four-phase format image, the 2x2 pixel blocks of the same channel are also taken as units, and all the 2x2 pixel blocks are preprocessed according to step S41 or step S42, and then input into the optimized offline inference model after preprocessing to perform model inference, and finally obtain the output result of the model, that is, the output RGB three-channel image.

[0033] Furthermore, in step S5, the mapping network structure includes: any one of a residual network structure, a densely connected network structure, a U-type encoding and decoding network structure, and a generative adversarial network structure.

[0034] Furthermore, in step S5, the mapping network structure is optimally trained using a supervision function with the minimum absolute error, and the model with the minimum absolute error is saved as the optimal offline inference model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention provides an image re-mosaic method, comprising: S1, using an image sensor to capture and obtain original Bayer format images in multiple scenes; a filtering microlens in the image sensor corresponds to a pixel; S2, interpolating and white balancing the Bayer format images respectively to obtain RGB three-channel images in a linear domain; the RGB three-channel images are used as real labels for training data; S3, pixel resampling the RGB three-channel images to obtain multi-unit single-phase format images; S4, degrading the multi-unit single-phase format images to obtain simulated multi-unit four-phase format images; S5, establishing a mapping network structure from the simulated multi-unit four-phase format image to the RGB three-channel image, and optimizing the mapping network structure to obtain an optimized offline inference model; S6, pre-processing the randomly collected real multi-unit four-phase format images and inputting them into the optimized offline inference model to output the RGB three-channel images; S7, pixel resampling the output RGB three-channel images to obtain a Bayer image with the same resolution, and completing the re-mosaic.

[0037] The present invention is a pixel re-mosaic method for a multi-unit four-phase (QPD) format with a full-pixel spectroscopic structure. The present invention targets the structural characteristics of QPD and a texture-based image degradation strategy, and can make good use of the differences in spectroscopic pixels, thereby greatly improving the output quality of the multi-unit four-phase (QPD) format image at the original size. The present invention has lower complexity and better image quality than traditional interpolation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of re-mosaicing of 16C-QPD image into Bayer format image.

[0039] Figure 2 The figure is a flow chart of an image re-mosaicing method according to an embodiment of the present invention.

[0040] Figure 3 The diagram is a schematic diagram of interpolating a Bayer format image to obtain an RGB three-channel image in the image re-mosaic method according to an embodiment of the present invention.

[0041] Figure 4 It is a schematic diagram of pixel resampling of an RGB three-channel image in the image re-mosaic method according to an embodiment of the present invention.

[0042] Figure 5 Schematic diagram of degradation processing in the image re-mosaicing method according to an embodiment of the present invention.

[0043] Figure 6 It is a schematic diagram of a window and a 2x2 pixel block in the window in the degradation processing of the image re-mosaicing method according to an embodiment of the present invention.

[0044] Figure 7 Schematic diagram of degradation processing of a 2x2 pixel block in a smooth area in an image re-mosaicing method according to an embodiment of the present invention.

[0045] Figure 8 This is a schematic diagram showing that the random compensation values ​​corresponding to four pixels in a 2x2 pixel block in the image re-mosaic method according to an embodiment of the present invention conform to a normal distribution.

[0046] Fig. 9 Schematic diagram of degradation processing of a 2x2 pixel block in a texture area in an image re-mosaicing method according to an embodiment of the present invention.

[0047] Fig.10 It is a schematic diagram comparing the RGB image before degradation and the 16C-QPD image after degradation of the present invention.

[0048] Fig.11 The first example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention.

[0049] Fig.12 The second example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention.

[0050] Fig.13 The third example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention.

[0051] Fig.14 It is a schematic diagram comparing the 16C-QPD diagram of the present invention and the Bayer diagram after heavy mosaic. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0053] In the drawings, the sizes of layers, regions, elements and their relative sizes may be exaggerated for clarity. Like reference numerals denote like elements throughout.

[0054] It should be understood that when an element or layer is referred to as "on ...", "adjacent to ...", "connected to" or "coupled to" other elements or layers, it can be directly on, adjacent to, connected to or coupled to other elements or layers, or there can be intervening elements or layers. On the contrary, when an element is referred to as "directly on ...", "directly adjacent to ...", "directly connected to" or "directly coupled to" other elements or layers, there is no intervening element or layer. It should be understood that although the terms first, second, third, etc. can be used to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the teachings of the present application, the first element, component, region, layer or part discussed below can be represented as the second element, component, region, layer or part. And when the second element, component, region, layer or part is discussed, it does not indicate that the present application necessarily has the first element, component, region, layer or part.

[0055] Spatially relative terms such as "under", "beneath", "below", "under", "above", "above", etc., may be used here for convenience of description to describe the relationship between an element or feature shown in the figure and other elements or features. It should be understood that in addition to the orientation shown in the figure, the spatial relationship terms are intended to also include different orientations of the device in use and operation. For example, if the device in the accompanying drawings is turned over, then the elements or features described as "under other elements" or "under it" or "under it" will be oriented as "on" other elements or features. Therefore, the exemplary terms "under" and "under" may include both upper and lower orientations. The device can be oriented otherwise (rotated 90 degrees or other orientations) and the spatial descriptors used herein are interpreted accordingly.

[0056] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0057] The embodiment of the present invention provides an image re-mosaic method, such as Figure 2 As shown, including:

[0058] S1, using an image sensor to capture and obtain original Bayer format images in multiple scenes; one filtering microlens in the image sensor corresponds to one pixel;

[0059] S2, interpolating and white balancing the Bayer format image to obtain a RGB three-channel image in the linear domain; the RGB three-channel image is used as the true label of the training data;

[0060] S3, resampling the RGB three-channel image to obtain a multi-unit single-phase format image;

[0061] S4, performing degradation processing on the multi-unit single-phase format image to obtain a simulated multi-unit four-phase format image;

[0062] S5, establishing a mapping network structure from a simulated multi-unit four-phase format image to an RGB three-channel image, and performing optimal training on the mapping network structure to obtain an optimized offline inference model;

[0063] S6, preprocessing the randomly collected real multi-unit four-phase format image and inputting it into the optimized offline inference model to output an RGB three-channel image;

[0064] S7, resample the output RGB three-channel image to obtain a bayer image with the same resolution, and complete the re-mosaic.

[0065] The purpose of the present invention is to re-mosaic an image in a multi-unit four-phase (QPD) format into a Bayer image. The multi-unit four-phase includes any one of 4-unit four-phase (4C-QPD), 16-unit four-phase (16C-QPD) and 32-unit four-phase (32C-QPD), as well as other suitable multi-unit four-phases. The multi-unit four-phase format image is the latest image sensor technology developed in recent years, and there is an extreme lack of multi-unit four-phase format image data sets. Steps S1 to S4 are how to simulate and obtain a multi-unit four-phase format image data set. The present invention uses the multi-scene Bayer format image that is easily obtained by the most conventional 1C image sensor (one filter microlens corresponds to one pixel), and then interpolates to obtain an RGB three-channel image (as a true label), and then samples it into a multi-unit single-phase format image, and then degenerates it into a multi-unit four-phase format image (as a corresponding training sample), imitating the characteristics of the multi-unit four-phase format image, and finally obtains a simulated multi-unit four-phase format image through degradation processing. The present invention collects data based on the 1C image sensor and processes it to obtain a simulated multi-unit four-phase format image. 1C image sensor collects data relatively directly using 16C-QPD, which reduces algorithm complexity and running time.

[0066] The simulated multi-unit four-phase format image dataset is used to train the optimized offline inference model for converting multi-unit four-phase format images to RGB three-channel images. After obtaining this optimized offline inference model, a real multi-unit four-phase format image is randomly collected and input into the optimized offline inference model. The output can be an RGB three-channel image, and then the RGB three-channel image is sampled as a bayer image to complete the re-mosaic.

[0067] The steps of the image re-mosaicing method according to the embodiment of the present invention are described in detail below with reference to the accompanying drawings.

[0068] S1. Use an image sensor to capture and obtain raw Bayer format images in multiple scenes; one filtering microlens in the image sensor corresponds to one pixel, also known as a 1C (Cell) image sensor or a full-pixel image sensor. The image sensor collects multiple frames of images in static scenes in several scenes (including but not limited to daytime, nighttime, different color temperatures, different gains, different exposures, etc.) indoors, outdoors, and in the laboratory, and then obtains the corresponding 1C Raw domain Bayer format image by averaging multiple frames. The Bayer format image, for example, includes multiple 2*2 pixel units arranged in any one of RGGB, GRBG, and BGGR, or in other suitable arrangement orders.

[0069] S2, such as Figure 3 As shown, the Bayer format image is interpolated and white balanced to obtain a linear domain RGB three-channel image; the RGB three-channel image is used as the true label of the training data. The general image signal processor is processed by various modules in the raw domain, RGB domain, and YUV domain, which will include complex nonlinear transformations. The re-mosaic process is the Raw domain Bayer format data, so the acquisition of the RGB three-channel image data used in the present invention must use linear domain data, and the public data set is not applicable. White balance processing solves the problem of false color at the edge of the image.

[0070] S3, such as Figure 4As shown, the RGB three-channel image is pixel-resampled to obtain a multi-unit single-phase format image. The multi-unit single-phase format includes, for example, any one of 4-unit single-phase (4C-1), 16-unit single-phase (16C-1) and 32-unit single-phase (32C-1), as well as other suitable multi-unit single-phases. The multi-unit single-phase format image includes a plurality of first filter array groups, and a first filter array group includes four primary color array groups: a blue primary color array group, two green primary color array groups and a red primary color array group. In the 4-unit single-phase (4C-1) format image, each primary color array group includes 4 pixels, and each pixel corresponds to a filter microlens. In the 16-unit single-phase (16C-1) format image, each primary color array group includes 16 pixels, and each pixel corresponds to a filter microlens. In the 32-unit single-phase (32C-1) format image, each primary color array group includes 32 pixels, and each pixel corresponds to a filter microlens.

[0071] S4, such as Figure 5 As shown, the multi-unit single-phase format image is degraded to obtain a simulated multi-unit four-phase format image. The image quality of the multi-unit four-phase format image is reduced (degraded) compared with the corresponding multi-unit single-phase format image, so it is called degradation processing.

[0072] The simulation of multi-unit four-phase includes: any one of the simulation of 4-unit four-phase (4C-QPD), the simulation of 16-unit four-phase (16C-QPD) and the simulation of 32-unit four-phase (32C-QPD), and other suitable simulation of multi-unit four-phase. The simulation of multi-unit four-phase format image is used as the corresponding training sample. The simulation of multi-unit four-phase format image includes a plurality of second filter array groups, and a second filter array group includes four primary color array groups: a blue primary color array group, two green primary color array groups and a red primary color array group; in the simulation of 4-unit four-phase (4C-QPD) format image, each primary color array group includes 4 pixels, and every 4 pixels share a filter microlens. In the simulation of 16-unit four-phase (16C-QPD) format image, each primary color array group includes 16 pixels, and every 4 pixels share a filter microlens. In the simulation of 32-unit four-phase (32C-QPD) format image, each primary color array group includes 32 pixels, and every 4 pixels share a filter microlens.

[0073] Figure 5 Schematic diagram of degradation processing in the image re-mosaicing method according to an embodiment of the present invention. Figure 6 Schematic diagram of a window and a pixel block within the window in degradation processing in an embodiment of the present invention. Figure 7 Schematic diagram of degradation processing of a 2x2 pixel block in a smooth area in an embodiment of the present invention. Figure 8Schematic diagram of a normal distribution of random compensation values ​​corresponding to four pixels in a 2x2 pixel block in an embodiment of the present invention. Fig. 9 Schematic diagram of degradation processing of a 2x2 pixel block in a texture area in an embodiment of the present invention.

[0074] like Figures 5 to 9 As shown, in a multi-unit single-phase format (e.g., 16C-1) image, the 2x2 pixel blocks of the same primary color (same channel) are divided as units, and one of the 2x2 pixel blocks (e.g. Figure 6 As shown in the red box, define an MxN window, which covers and is larger than a 2x2 pixel block area. The MxN window consists of a 2x2 pixel block and pixels located outside the 2x2 pixel block. Perform texture and gradient detection in the MxN window: calculate the image variance Var, horizontal gradient Gh, and vertical gradient Gv in the current window respectively; obtain the window confidence by weighted calculation; set the image variance threshold to K; the pixel values ​​of the four pixels in the 2x2 pixel block are P1, P2, P3, and P4, respectively, and after degradation, the four pixels share a filtering microlens. The window size can be adjusted, for example, including any one of a 16x16 pixel block, a 16x8 pixel block, and a 32x32 pixel block, as well as other suitable sizes.

[0075] When the image variance Var in the window is less than the image variance threshold K, the 2x2 pixel block is within the smooth area confidence level; first, a weighted average is performed on the 2x2 pixel block, and then local compensation is performed on the four pixels, where the compensation value conforms to a normal distribution with a mean of 0. In the smooth area of ​​the image, since the correlation between adjacent pixels is greater, it is easy to obtain information about the other two channel pixels by interpolation. The degradation processing includes step S41, such as Figure 7 and Figure 8 As shown, step S41 specifically includes:

[0076] S411, calculating the average value of the four pixels, and using the average pixel value P to replace the original four pixel values ​​P1, P2, P3 and P4 respectively, to make up the brightness difference of the four pixels;

[0077] S412. Add a random compensation value to each of the four average pixel values ​​P. The random compensation value conforms to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel bit width. Each average pixel value P is added to its corresponding random compensation value to obtain four compensated pixel values, which are P1', P2', P3' and P4'.

[0078] like Fig. 9As shown, when the image variance Var in the window> the image variance threshold K, the 2x2 pixel block is within the texture area confidence; the edge (texture) area of ​​the image, especially in the high-frequency texture, the information of the other two channel pixels interpolated is more susceptible to the influence of the gradient detection accuracy. The degradation processing includes step S42, which specifically includes:

[0079] S421, generate four compensation values ​​C P1 , C P2 , C P3 and C P4 , the four compensation values ​​conform to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel width;

[0080] S422, determine the gradient direction of the current 2x2 pixel block, when the horizontal gradient Gh> vertical gradient Gv, the compensation value needs to be constrained to meet C P1 >C P3 , C P2 >C P4 or C P1 <C P3 ,C P2 <C P4 ;

[0081] When the horizontal gradient Gh < vertical gradient Gv, the compensation value needs to be constrained to meet C P1 >C P2 , C P3 >C P4 or C P1 <C P2 ,C P3 <C P4 ;

[0082] S423, adding the pixel values ​​of the four pixels to their corresponding compensation values ​​to obtain four compensated pixels, which are P1, b 、P2 b 、P3 b 、P4 b ;and

[0083] P1 b =P1+C P1 ;P2 b =P2+C P2 ;P3 b =P3+C P3 ;P4 b =P4+C P4 .

[0084] In step S4, in the multi-unit single-phase format image, all 2x2 pixel blocks are degraded according to step S41 or step S42 to obtain a simulated multi-unit four-phase format image;

[0085] The first brightness mean of the whole image of the multi-unit single-phase format image before degradation is calculated, the second brightness mean of the whole image of the simulated multi-unit four-phase format image after degradation is calculated, a global brightness compensation value is set, and the second brightness mean is compensated so that the second brightness mean after global compensation is consistent with the first brightness mean, that is, the brightness mean of the image after degradation is consistent with that before degradation.

[0086] S5. Establish a mapping network structure from a simulated multi-unit four-phase format image to an RGB three-channel image, and optimize the mapping network structure to obtain an optimized offline inference model. The mapping network structure is used to learn the resampling process from a simulated multi-unit four-phase format image to an RGB three-channel image. Structures with different design complexities can be selected according to different algorithm application platforms. The mapping network structure includes but is not limited to: a simple residual network structure, a densely connected structure, a U-type encoding and decoding structure, and an adversarial generation network structure, etc. The minimum absolute error supervision function is used, and the supervision function can be, for example, an L1 loss function, to optimize the mapping network structure and save the model with the minimum absolute error as the optimized offline inference model.

[0087] S6. Preprocess the randomly collected real multi-unit four-phase format image and input it into the optimized offline reasoning model to output an RGB three-channel image. The real multi-unit four-phase format image is also divided into 2x2 pixel blocks of the same channel. All 2x2 pixel blocks are preprocessed according to step S41 or step S42, and then input into the optimized offline reasoning model after preprocessing to perform model reasoning, and finally obtain the output result of the model, that is, output an RGB three-channel image. The preprocessing in step S6 and the degradation processing in step S4 are the same in processing method, only the processing objects are different, so that the simulated multi-unit four-phase format image and the real multi-unit four-phase format image are more consistent and close. The processing object of step S4 is a multi-unit single-phase format image, and the processing object of step S6 is a randomly collected real multi-unit four-phase format image.

[0088] S7, resample the output RGB three-channel image to obtain a bayer image with the same resolution, and complete the re-mosaic.

[0089] Fig.10 It is a schematic diagram comparing the RGB image before degradation and the 16C-QPD image after degradation of the present invention. Fig.11 The first example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention. Fig.11In the first example, the prior art re-mosaic effect image has color distortion and false color phenomenon (as shown in the red circle), and the present invention adopts a degradation strategy to make the image clear without distortion and false color after re-mosaic. Fig.12 The second example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention. Fig.12 In the second example, the prior art re-mosaic effect image appears distorted and blurred (as shown in the red circle), while the present invention adopts a degradation strategy to make the image clear and undistorted after re-mosaic. Fig.13 The third example is a comparison between the effect image after re-mosaicing using the prior art and the effect image after re-mosaicing using the degradation strategy of the present invention. Fig.13 In the third example, the prior art re-mosaic effect image appears distorted and blurred (as shown in the red circle), and the originally uniform lines are distorted and displayed as lines of different thicknesses; the present invention adopts a degradation strategy to re-mosaic the effect image, and the image is clear, and the lines are uniform and not distorted. Fig.14 It is a schematic diagram comparing the 16C-QPD diagram of the present invention and the Bayer diagram after heavy mosaic.

[0090] In summary, the present invention provides an image re-mosaic method, comprising: S1, using an image sensor to capture and obtain original Bayer format images in multiple scenes; a filtering microlens in the image sensor corresponds to a pixel; S2, interpolating and white balancing the Bayer format images respectively to obtain RGB three-channel images in the linear domain; the RGB three-channel images are used as real labels for training data; S3, pixel resampling the RGB three-channel images to obtain multi-unit single-phase format images; S4, degrading the multi-unit single-phase format images to obtain simulated multi-unit four-phase format images; S5, establishing a mapping network structure from the simulated multi-unit four-phase format image to the RGB three-channel image, and optimizing the mapping network structure to obtain an optimized offline inference model; S6, pre-processing the randomly collected real multi-unit four-phase format images and inputting them into the optimized offline inference model to output the RGB three-channel images; S7, pixel resampling the output RGB three-channel images to obtain bayer images with the same resolution, and completing the re-mosaic.

[0091] The present invention aims at the structural characteristics of QPD and the texture-based image degradation strategy, which can make good use of the difference of the splitting pixels, thereby greatly improving the output quality of the multi-unit four-phase (QPD) format image at the original size. The present invention has lower complexity and better image quality than the traditional interpolation method.

[0092] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0093] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. An image re-mosaic method, characterized in that: include: S1, using an image sensor to capture and obtain original Bayer format images in multiple scenes; one filtering microlens in the image sensor corresponds to one pixel; S2, performing interpolation and white balance processing on the Bayer format image to obtain an RGB three-channel image in a linear domain; The RGB three-channel image is used as the true label of the training data; S3, resampling the RGB three-channel image to obtain a multi-unit single-phase format image; S4, performing degradation processing on the multi-unit single-phase format image to obtain a simulated multi-unit four-phase format image; S5, establishing a mapping network structure from the simulated multi-unit four-phase format image to the RGB three-channel image, and performing optimization training on the mapping network structure to obtain an optimized offline inference model; S6, preprocessing the randomly collected real multi-unit four-phase format image and inputting it into the optimized offline inference model to output an RGB three-channel image; S7, resampling the output RGB three-channel image to obtain a bayer image with the same resolution, and completing re-mosaicing.

2. The image re-mosaic method according to claim 1, characterized in that: The multi-unit single-phase format includes any one of: 4-unit single-phase, 16-unit single-phase and 32-unit single-phase.

3. The image re-mosaic method according to claim 2, characterized in that: The multi-unit single-phase format image includes a plurality of first filter array groups, one of which includes four primary color array groups: one blue primary color array group, two green primary color array groups and one red primary color array group; each pixel in the primary color array group corresponds to a filtering microlens; in the 4-unit single-phase format image, each primary color array group includes 4 pixels; in the 16-unit single-phase format image, each primary color array group includes 16 pixels; in the 32-unit single-phase format image, each primary color array group includes 32 pixels.

4. The image re-mosaic method according to claim 1, characterized in that: The simulated multi-unit four-phase format includes any one of simulated 4-unit four-phase, simulated 16-unit four-phase and simulated 32-unit four-phase.

5. The image re-mosaic method according to claim 4, characterized in that: The simulated multi-unit four-phase format image includes a plurality of second filter array groups, one of which includes four primary color array groups: one blue primary color array group, two green primary color array groups and one red primary color array group; every 4 pixels in the primary color array group share a filter microlens; in the simulated 4-unit four-phase format image, each of the primary color array groups includes 4 pixels; in the simulated 16-unit four-phase format image, each of the primary color array groups includes 16 pixels; in the simulated 32-unit four-phase format image, each of the primary color array groups includes 32 pixels.

6. The image re-mosaic method according to claim 1, characterized in that: Step S4 specifically includes: in the multi-unit single-phase format image, taking a 2x2 pixel block of the same channel as a unit, performing texture and gradient detection in an MxN window; the 2x2 pixel block is located in the MxN window; respectively calculating the image variance Var, the horizontal gradient Gh, and the vertical gradient Gv in the current window; setting the image variance threshold to K; the pixel values ​​of the four pixels of the 2x2 pixel block are P1, P2, P3 and P4 respectively.

7. The image re-mosaic method according to claim 6, characterized in that: In step S4, when the image variance Var in the window is less than the image variance threshold K, the 2x2 pixel block is within the smooth region confidence level; the degradation processing includes step S41, which specifically includes: S411, calculating an average value of the four pixels, and using the average pixel value P to replace the original four pixel values ​​respectively, so as to make up the brightness difference of the four pixels; S412. Add a random compensation value to each of the four average pixel values ​​P, wherein the random compensation value conforms to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel bit width. Each of the average pixel values ​​P is added to the corresponding random compensation value to obtain four compensated pixel values, which are P1', P2', P3' and P4'.

8. The image re-mosaic method according to claim 6, characterized in that: In step S4, when the image variance Var in the window> the image variance threshold K, the 2x2 pixel block is within the texture region confidence level; the degradation processing includes step S42, which specifically includes: S421, generate four compensation values ​​C P1 , C P2 , C P3 and C P4 , the four compensation values ​​conform to a normal distribution with a mean of 0 and a standard deviation of 3 thousandths of the maximum brightness value that can be represented by the pixel bit width; S422, determine the gradient direction of the current 2x2 pixel block, when the horizontal gradient Gh> the vertical gradient Gv, it is necessary to constrain the compensation value to conform to C P1 >C P3 , C P2 >C P4 or C P1 <C P3 ,C P2 <C P4 ; When the horizontal gradient Gh<the vertical gradient Gv, it is necessary to constrain the compensation value to conform to C P1 >C P2 , C P3 >C P4 or C P1 <C P2 ,C P3 <C P4 ; S423, adding the pixel values ​​of the four pixels to their corresponding compensation values ​​to obtain four compensated pixels, which are P1, b 、P2 b 、P3 b 、P4 b ;and P1 b =P1+C P1 ;P2 b =P2+C P2 ;P3 b =P3+C P3 ;P4 b =P4+C P4 。 9. The image re-mosaic method according to any one of claims 7 or 8, characterized in that: In step S4, in the multi-unit single-phase format image, all the 2x2 pixel blocks are subjected to the degradation processing according to step S41 or step S42 to obtain the simulated multi-unit four-phase format image; a first brightness mean of the entire image of the multi-unit single-phase format image before degradation is calculated, and a second brightness mean of the entire image of the simulated multi-unit four-phase format image after degradation is calculated, and a global brightness compensation value is set to compensate the second brightness mean so that the second brightness mean after global compensation is consistent with the first brightness mean, that is, the brightness mean of the image after degradation is consistent with that before degradation.

10. The image re-mosaic method according to any one of claims 7 or 8, characterized in that: In step S6, the real multi-unit four-phase format image is also divided into 2x2 pixel blocks of the same channel, and all the 2x2 pixel blocks are preprocessed according to step S41 or step S42. After the preprocessing, they are input into the optimized offline inference model to perform model inference, and finally the output result of the model, i.e., the output RGB three-channel image, is obtained.

11. The image re-mosaic method according to claim 1, characterized in that: In step S5, the mapping network structure includes: any one of a residual network structure, a densely connected network structure, a U-type encoding and decoding network structure and a generative adversarial network structure.

12. The image re-mosaic method according to claim 1, characterized in that: In step S5, the mapping network structure is optimally trained using the supervision function with the minimum absolute error, and the model with the minimum absolute error is saved as the optimized offline inference model.

Images