Image interpolation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides an image interpolation method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the preprocessing of a to-be-processed image, and obtaining the preset mode distribution of pixels of the to-be-processed image; carrying out RGB interpolation and IR interpolation on each pixel of the to-be-processed image in a plurality of preset directions based on the preset mode distribution, and respectively obtaining a full-size RGB image and a full-size IR image; obtaining a full-size four-channel image based on the full-size RGB image and the full-size IR image, and performing IR separation on the four-channel image to obtain a full-size IR separation image; obtaining a target interpolation direction selected from the interpolation directions of each pixel in the IR separation image; and taking the interpolation result in the target interpolation direction as the interpolation result of the IR separation image. According to the scheme of the embodiment, the influence of the IR component on target interpolation direction selection is reduced, and a technical basis is provided for obtaining an accurate raw image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image interpolation method, device, electronic device and computer-readable storage medium. Background Art

[0002] The RGBIR (red, green, blue, infrared) 4*4 module is a new type of image sensor. Its common pixel distribution is as follows: Figure 1 Compared with the traditional 2*2 Bayer color filter array (CFA) arrangement, the RGBIR filter replaces some filters with IR (infrared) filters. Infrared light is used for fill light at night or in other low-light scenes, thereby realizing night vision technology for some security and monitoring systems, replacing the mechanical device that requires manual switching of IR-CUT (infrared-filter) in traditional solutions, achieving economy and convenience. Summary of the invention

[0003] Embodiments of the present disclosure provide an image interpolation method, an apparatus, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image interpolation method, which may include:

[0005] Performing pre-processing on the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed;

[0006] Based on the preset pattern distribution, red, green, and blue (RGB) interpolation is performed on each pixel of the image to be processed in multiple preset directions to obtain a full-size RGB image;

[0007] Based on the preset pattern distribution, performing infrared IR interpolation on each pixel of the image to be processed in a plurality of preset directions to obtain a full-size IR image;

[0008] Obtaining a full-size four-channel image based on the full-size RGB image and the full-size IR image;

[0009] Performing IR separation on the four-channel image to obtain a full-size IR separation image;

[0010] Acquire a target interpolation direction selected from the interpolation directions of each pixel in the IR separated image;

[0011] The interpolation result in the target interpolation direction is used as the interpolation result of the IR separation image.

[0012] In a second aspect, an embodiment of the present disclosure provides an image interpolation device, including:

[0013] A pre-processing unit, used for pre-processing the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed;

[0014] An interpolation processing unit, configured to perform red, green, and blue (RGB) interpolation on each pixel of the image to be processed in a plurality of preset directions based on the preset mode distribution to obtain a full-size RGB image;

[0015] An IR interpolation processing unit, configured to perform infrared IR interpolation on each pixel of the image to be processed in a plurality of preset directions based on the preset pattern distribution to obtain a full-size IR image;

[0016] A four-channel image acquisition unit, configured to obtain a full-size four-channel image based on the full-size RGB image and the full-size IR image;

[0017] An IR separation processing unit, used for performing IR separation on the four-channel image to obtain a full-size IR separation image;

[0018] a direction selection unit, configured to obtain a target interpolation direction selected from the interpolation directions of each pixel in the IR separated image;

[0019] The interpolation result acquisition unit is used to use the interpolation result in the target interpolation direction as the interpolation result of the IR separation image.

[0020] In a third aspect, an embodiment of the present disclosure provides an electronic device, which may include:

[0021] one or more processors;

[0022] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image interpolation method;

[0023] One or more input / output I / O interfaces are connected between the processor and the memory and are configured to implement information interaction between the processor and the memory.

[0024] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image interpolation method is implemented.

[0025] The disclosed embodiment performs RGB interpolation and IR difference on each pixel of the image to be processed in multiple preset directions to obtain a full-size RGB image and a full-size IR image, and finally obtains a full-size four-channel image based on the full-size RGB image and the full-size IR image. The IR component of the four-channel image is then separated so that the IR component is no longer mixed in the pixel point R, the pixel point B, and the pixel point G. Finally, a target interpolation direction is selected, which reduces the influence of the IR component on the target interpolation direction selection, and provides a technical basis for obtaining an accurate raw image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In the accompanying drawings of the embodiments of the present disclosure:

[0027] Figure 1 A schematic diagram of common pixel distribution in related technologies;

[0028] Figure 2 A flow chart of an image interpolation method provided by an embodiment of the present disclosure;

[0029] Figure 3 A schematic diagram of a preset direction embodiment provided for an embodiment of the present disclosure;

[0030] Figure 4 A flow chart of a method for performing red, green, and blue (RGB) interpolation on each pixel of an image to be processed in multiple preset directions based on a preset mode distribution to obtain a full-size RGB image provided by an embodiment of the present disclosure;

[0031] Figure 5 A flow chart of a method for interpolating corresponding non-G pixels from multiple preset directions based on the first neighborhood of each non-G pixel to obtain a full-size G component of each non-G pixel provided in an embodiment of the present disclosure;

[0032] Figure 6 A schematic diagram of a neighborhood when performing difference calculation for each preset direction of an R pixel provided in an embodiment of the present disclosure;

[0033] Figure 7 A schematic diagram of a neighborhood when performing difference calculation for each preset direction of an IR pixel provided in an embodiment of the present disclosure;

[0034] Figure 8 A schematic diagram of an interpolation plane embodiment provided in an embodiment of the present disclosure;

[0035] Fig. 9 A schematic diagram of a neighborhood embodiment when interpolating the G component of the R(0,0) pixel provided in an embodiment of the present disclosure;

[0036] Fig.10 A schematic diagram of the length direction, width direction, and center pixel of a neighborhood in an image to be processed provided by an embodiment of the present disclosure;

[0037] Fig.11 A flowchart of a method for interpolating a plurality of full-size R components in preset directions for each non-R pixel using a first color difference between each non-R pixel and a corresponding R pixel in a second neighborhood provided in an embodiment of the present disclosure;

[0038] Fig.12 A schematic diagram of a neighborhood embodiment for interpolating a vertical plane with IR as the center pixel provided in an embodiment of the present disclosure;

[0039] Fig.13 A schematic diagram of a neighborhood embodiment for interpolating a horizontal plane with IR as the center pixel provided in an embodiment of the present disclosure;

[0040] Fig.14 A schematic diagram of a neighborhood embodiment provided by the present disclosure when interpolating a 45° oblique plane with IR as the center pixel;

[0041] Fig.15 A schematic diagram of a neighborhood embodiment when interpolating the R component of a horizontal plane with B as the center pixel provided in an embodiment of the present disclosure;

[0042] Fig.16 A flowchart of a method for interpolating a plurality of full-size B components in preset directions for each non-B pixel using the second color difference between each non-B pixel and the corresponding B pixel in the third neighborhood provided in an embodiment of the present disclosure;

[0043] Fig.17 A schematic diagram of a neighborhood embodiment provided by the present disclosure when interpolating the B component on a horizontal plane with R as the center pixel;

[0044] Fig.18 A schematic diagram of an implementation architecture of an image interpolation device provided in an embodiment of the present disclosure;

[0045] Fig.19 A block diagram of the electronic device provided in the embodiment of the present disclosure;

[0046] Fig. 20 A block diagram of the computer-readable storage medium provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the communication perception data processing method and computer-readable storage medium provided by the embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0048] The present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete, and will enable those skilled in the art to fully understand the scope of the present disclosure.

[0049] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used together with the detailed embodiments to explain the present disclosure, and do not constitute a limitation of the present disclosure. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0050] The present disclosure may be described with reference to plan views and / or cross-sectional views by means of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0051] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0052] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in the present disclosure are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in the present disclosure specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0054] The present disclosure is not limited to the embodiments shown in the drawings, but includes modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0055] The RGBIR (red, green, blue, infrared) 4*4 module is a new type of image sensor. Its common pixel distribution is as follows: Figure 1As shown. Among them, RGB is the color mode, R stands for Red, G stands for Green, and B stands for Blue. It is the most widely used color mode. IR is Infrared, which is infrared (or simply infrared). Its wavelength (about 760nm~1mm) is longer than red light and is invisible light. In dark light conditions, use IR cameras to enhance the reality effect. Traditional COMS (Complementary Metal-Oxide-Semiconductor) image sensors use the Bayer format as the color filter array (CFA), and the color information of the pixel is determined according to the specific wavelength of light. RGBIR technology uses RGBIR CFA, based on the Bayer format, to modify some pixels to IR pixels, and IR pixels only allow infrared light to pass through.

[0056] Compared with the traditional 2*2 Bayer color filter array (CFA) arrangement, the RGBIR filter replaces some filters with IR (infrared) filters. Infrared light is used for supplementary lighting at night or in other low-light scenes, thereby realizing night vision technology for some security and monitoring systems, replacing the traditional solution that requires manual switching of the IR-CUT (infrared-filter) mechanical device, achieving economy and convenience.

[0057] However, each filter only allows one basic color of light to pass through. Therefore, when a normal RGB color image needs to be obtained from the RGBIR 4*4 module, how to accurately obtain the other three color components outside the current channel and perform IR separation (wavelength aliasing) becomes a problem that must be solved.

[0058] The related interpolation technology does not take into account the sparse pixel distribution of the RGBIR 4*4 module, resulting in pseudo-color in the edge or narrow object area of ​​the interpolated image. In addition, the current interpolation methods all calculate the directional gradient information before IR separation, which is affected by the IR component and leads to inaccurate calculation of the interpolation direction. The above factors will cause the subjective quality of the final color image to deteriorate and lead to inaccurate interpolated pixel values. In particular, it is necessary to focus on the process of interpolating R at the B position and interpolating B at the IR position.

[0059] The disclosed embodiment provides a full-size multi-directional image interpolation method, which can perform RGB interpolation and IR difference on each pixel of the processed image in multiple preset directions, obtain a full-size RGB image and a full-size IR image, and finally obtain a full-size four-channel image based on the full-size RGB image and the full-size IR image, and then perform IR component separation on the four-channel image, so that the IR component is no longer mixed in the pixel point R, the pixel point B, and the pixel point G, and finally perform target interpolation direction selection, thereby reducing the influence of the IR component on the target interpolation direction selection, and providing a technical basis for obtaining an accurate raw (raw material) image.

[0060] The image interpolation method of the embodiment of the present disclosure can be executed by any electronic device capable of image processing, such as a terminal device or a server. The terminal device may include but is not limited to: vehicle-mounted equipment, user equipment (UE), mobile equipment, computing equipment, wearable equipment, etc., for example, including but not limited to cellular phones, cordless phones, personal digital assistants (PDA), portable computers, etc. The image interpolation method can be implemented by a processor calling computer-readable program instructions stored in a memory, or can be implemented by a server.

[0061] The embodiments of the present disclosure can be applied to the front end of the ISP (image signal processor) image processing field, directly receiving the output data of the RGBIR 4*4 module as input. The common ISP pipeline can use the embodiment of the present disclosure as the first module for data processing, convert it into ordinary bayer pattern data and then input it into other modules for processing.

[0062] The following is a detailed introduction to the embodiments of the present disclosure.

[0063] The present disclosure provides an image interpolation method. Figure 2 As shown, the method may include steps S11-S17:

[0064] S11, performing pre-processing on the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed;

[0065] S12, performing red, green, and blue (RGB) interpolation on each pixel of the image to be processed in multiple preset directions based on a preset mode distribution to obtain a full-size RGB image;

[0066] S13, performing infrared IR interpolation on each pixel of the image to be processed in multiple preset directions based on a preset pattern distribution to obtain a full-size IR image;

[0067] S14, obtaining a full-size four-channel image based on the full-size RGB image and the full-size IR image;

[0068] S15, performing IR separation on the four-channel image to obtain a full-size IR separation image;

[0069] S16, obtaining a target interpolation direction selected from the interpolation directions of each pixel in the IR separation image;

[0070] S17: Using the interpolation result in the target interpolation direction as the interpolation result of the IR separation image.

[0071] In the embodiment of the present disclosure, the pre-processing may include, but is not limited to: obtaining a preset pattern distribution of pixels of the image to be processed and performing black level correction.

[0072] In the embodiment of the present disclosure, the preset pattern distribution may include but is not limited to: RGBIR 4*4Bayer pattern distribution.

[0073] In the embodiments of the present disclosure, Figure 3 As shown, based on the plane image, the multiple preset directions may include but are not limited to: horizontal left direction, horizontal right direction, vertical upward direction, vertical downward direction, upper left oblique first angle (e.g. 45°) direction, upper right oblique second angle (e.g. 45°) direction, lower left oblique third angle (e.g. 45°) direction, lower right oblique fourth angle (e.g. 45°) direction. For the above preset directions, the directions on the plane such as east (expressed as E), west (expressed as W), south (expressed as S), and north (expressed as N) can be indicated, for example, west (i.e. horizontal left direction), east (i.e. horizontal right direction), north (i.e. vertical upward direction), south (i.e. vertical downward direction), northwest (i.e. upper left oblique first angle), northeast (i.e. upper right oblique second angle), southwest (i.e. lower left oblique third angle), and southeast (i.e. lower right oblique fourth angle). Wherein, A is the center pixel of the plane image, and the center pixel is the pixel currently being interpolated.

[0074] The disclosed embodiment can complete the upsampling process of the full-size R, G, and B images through a directional interpolation technology based on the color difference theory, and realize the process of generating ordinary Bayer pattern data (full-size RGB needs to be downsampled) or normal color pictures under a special module. The key point of the disclosed embodiment is to interpolate the full-size G component in different directions; then interpolate the full-size R component and B component based on the color difference constant theory according to the relationship between the G component of the pixel to be interpolated and the G component of the related pixels in the neighborhood; Bicubic (bicubic interpolation) can be used to upsample the full-size IR image, and IR separation can be performed using a certain separation coefficient matrix; finally, the most suitable target interpolation direction is selected based on uniformity judgment, and the interpolation pixels in the target interpolation direction are used as the current R, G, and B values. The purpose of this is to avoid the related technology from selecting the interpolation direction before interpolation, which is easily affected by the IR component, resulting in an error in the interpolation direction selection, causing the occurrence of pseudo-color phenomenon.

[0075] In the embodiment of the present disclosure, the interpolation process of the R and B full-size images is not performed directly based on the color difference relationship of the G component in the neighborhood, but the corresponding relationship between the pixel to be interpolated and the G component in the neighborhood is introduced to perform interpolation restoration. The purpose of this is to improve the accuracy of interpolation. Because the R and B channels of the RGBIR 4*4 module are sparsely distributed, they are affected by drastic changes in pixels in some narrow pixel areas, which are prone to interpolation errors. Inspired by the stability of G component interpolation, the relationship between G components is adopted, and the interpolation weight of the pixel point with higher similarity to the G component at the position to be interpolated is higher within the neighborhood, thereby improving the interpolation stability. The hardware friendliness of the embodiment of the present disclosure is high, easy to implement, and can well improve the user experience of the RGBIR 4*4 module and restore better colors. The following is a detailed introduction to the embodiment of the present disclosure.

[0076] In the embodiments of the present disclosure, Figure 4 As shown, performing red, green, and blue (RGB) interpolation on each pixel of the image to be processed in multiple preset directions based on a preset mode distribution to obtain a full-size RGB image may include steps S21-S24:

[0077] S21. Determine the first neighborhood of each non-G pixel of the image to be processed based on a preset pattern distribution, interpolate the corresponding non-G pixels from multiple preset directions based on the first neighborhood of each non-G pixel, and obtain a full-size G component of each non-G pixel.

[0078] In the embodiments of the present disclosure, Figure 5 As shown, based on the first neighborhood of each non-G pixel, the corresponding non-G pixel is interpolated from multiple preset directions to obtain the full-size G component of each non-G pixel, including steps S31-S33:

[0079] S31 . For each non-G pixel, determine the sum of the G component difference value and the IR component difference value of the non-G pixel in each preset direction in the corresponding first neighborhood as the pixel difference value of the non-G pixel in the corresponding preset direction.

[0080] In the disclosed embodiment, variance calculation can be performed on each non-G pixel in the RAW (raw material) data of the image to be processed in each preset direction to obtain the pixel change degree of each non-G pixel in different preset directions (i.e., the above-mentioned pixel difference value). In order to improve the stability of variance calculation as much as possible, the pixel change difference of the G channel and the IR channel that is relatively stable in each preset direction can be selected as the difference change result of the current non-G pixel in the preset direction.

[0081] In the disclosed embodiment, Var dirtion =Var dir-G +Var dir-IR ;

[0082] Among them, Var dir-G Indicates the pixel difference value of the G component of a non-G pixel in a preset direction, Var dir- Indicates the pixel difference value of the IR component of the non-G pixel in the preset direction, Var dirtion Indicates the difference value of the non-G pixel in the preset direction direction. direction can satisfy: direction∈{N, E, S, W, NE, EW, SE, SW}.

[0083] In the embodiment of the present disclosure, the method of calculating the pixel difference values ​​of different non-G pixels is described below by taking multiple preset directions (such as the aforementioned eight directions) of R pixels and IR pixels as examples.

[0084] In the embodiments of the present disclosure, Figure 6 As shown, it is a neighborhood diagram when the difference calculation is performed for each preset direction of the R pixel, where R (0, 0) is the center pixel, that is, the pixel currently being interpolated.

[0085] In the embodiment of the present disclosure, R (k1, k2) represents the positional relationship between the R pixel position and the center pixel, K1 represents the vertical pixel distance, and when K1 is positive, it represents the vertical upward pixel distance, and when K1 is negative, it represents the vertical downward pixel distance; K2 represents the horizontal pixel distance; and when K2 is positive, it represents the horizontal right pixel distance, and when K2 is negative, it represents the horizontal left pixel distance; for example, Figure 6 Here, R(-2, -2) indicates the R pixel position which is 2 pixels upward vertically and 2 pixels to the left horizontally from the center pixel R(0,0). Figure 6Here, R(-2, 2), R(2, -2), and R(2, 2) have similar meanings and will not be repeated here.

[0086] In the embodiment of the present disclosure, G(k3, k4) represents the positional relationship between the G pixel position and the center pixel, K3 represents the vertical pixel distance, and when K3 is positive, it represents the vertical upward pixel distance, and when K3 is negative, it represents the vertical downward pixel distance; K4 represents the horizontal pixel distance; and when K4 is positive, it represents the horizontal right pixel distance, and when K4 is negative, it represents the horizontal left pixel distance; for example, Figure 6 Here, G(-2, -1) indicates the G pixel position which is 2 pixels upward vertically and 1 pixel to the left horizontally from the center pixel R(0, 0). Figure 6 Among them, G(-2, -1), G(-2, 1), G(-1, -2), G(-1, 0), G(-1, 2), G(0, -1), G(0, 1), G(1, -2), G(1, 0), G(1, 2), G(-2, -2), G(2, -1), and G(2, 1) have similar meanings and are not repeated here.

[0087] In the embodiment of the present disclosure, B(k5, k6) represents the positional relationship between the B pixel position and the center pixel, K5 represents the vertical pixel distance, and when K5 is positive, it represents the vertical upward pixel distance, and when K5 is negative, it represents the vertical downward pixel distance; K6 represents the horizontal pixel distance; and when K6 is positive, it represents the horizontal right pixel distance, and when K6 is negative, it represents the horizontal left pixel distance; for example, Figure 6 Here, B(-2, 0) indicates the B pixel position which is 2 pixels vertically upward and 0 pixels horizontally away from the center pixel R(0, 0). Figure 6 B(0, -2), G(0, 2), and B(2, 0) have similar meanings and will not be repeated here.

[0088] In the embodiment of the present disclosure, IR (k7, k8) represents the positional relationship between the IR pixel position and the center pixel, K7 represents the vertical pixel distance, and when K7 is positive, it represents the vertical upward pixel distance, and when K7 is negative, it represents the vertical downward pixel distance; K8 represents the horizontal pixel distance; and when K8 is positive, it represents the horizontal right pixel distance, and when K8 is negative, it represents the horizontal left pixel distance; for example, Figure 6 IR(-1,-1) indicates the IR pixel position which is 1 pixel upward vertically and 1 pixel to the left horizontally from the center pixel R(0,0). Figure 6 IR(-1, 1), IR(1, -1), and IR(1, 1) have similar meanings and are not repeated here.

[0089] In the embodiment of the present disclosure, the pixel change difference calculation method of the central pixel R (0, 0) in the east, west, south, north, northeast, southwest, northwest, and southeast directions is as follows:

[0090] Pixel change difference in the east direction

[0091]

[0092] Pixel change difference in the west direction

[0093]

[0094] Pixel change difference in north direction

[0095]

[0096] Pixel change difference in the south direction

[0097]

[0098] Pixel change difference in the northeast direction

[0099]

[0100] Pixel change difference in the southwest direction

[0101]

[0102] Pixel change difference in the northwest direction

[0103]

[0104] Pixel change difference in the southeast direction

[0105]

[0106] In the embodiment of the present disclosure, the variance calculation method of B pixels and R pixels in eight directions is the same, which will not be repeated here.

[0107] In the embodiments of the present disclosure, Figure 7 , which is a neighborhood diagram for calculating the difference of each preset direction of the IR pixel, wherein IR(0,0) is the center pixel, that is, the pixel currently being interpolated.

[0108] In the disclosed embodiment, Figure 7Medium IR(-2,-2), G(-2,-1), IR(-2,0), G(-2,1), IR(-2,2), G(-1,-2), R(-1,-1), G(-1,0), B(-1,1), G(-1,2), IR(0,-2), G(0 , -1), G(0,1), IR(0,2), G(1,-2), B(1,-1), G(1,0), R(1,1), G(1,2), IR(2,-2), G(2,-1), IR(2,0), G(2,1), IR(-2,2) and the above Figure 6 The meanings of the related symbols are similar and will not be repeated here.

[0109] In the embodiment of the present disclosure, the pixel change difference calculation method of the central pixel IR (0, 0) in the east, west, south, north, northeast, southwest, northwest, and southeast directions is as follows:

[0110] Pixel change difference in the east direction

[0111]

[0112] Pixel change difference in the west direction

[0113]

[0114] Pixel change difference in north direction

[0115]

[0116] Pixel change difference in the south direction

[0117]

[0118] Pixel change difference in the northeast direction

[0119]

[0120] Pixel change difference in the southwest direction

[0121]

[0122] Pixel change difference in the northwest direction

[0123]

[0124] Pixel change difference in the southeast direction

[0125]

[0126] S32. Determine the weight of each preset direction according to the pixel difference value in each preset direction, and obtain a set of asynchronous weights corresponding to each plane; wherein the weights of two opposite preset directions in the multiple preset directions constitute a pair of asynchronous weights.

[0127] In the embodiment of the present disclosure, there are four pairs of directions in the eight preset directions, which are opposite to each other. Figure 8 As shown, for example, south and north, east and west, northeast and southwest, northwest and southeast, each of the four pairs of directions can determine a corresponding weight, and each of the four pairs of directions can determine an interpolation plane. For example, south and north can determine a vertical plane M1, east and west can determine a horizontal plane M2, northeast and southwest can determine a 45° oblique plane M3, northwest and southeast can determine a 135° oblique plane M4, and the horizontal plane M1, vertical plane M2, 45° oblique plane M3, and 135° oblique plane M4 are respectively an interpolation plane.

[0128] In the embodiment of the present disclosure, the two directions corresponding to each interpolation plane are opposite, the sum of the weights of the two opposite directions is 1, and the weights of the two opposite directions can be called asynchronous weights.

[0129] In the disclosed embodiment, for example, there are a total of eight directions on the above four interpolation planes. Normalizing the pixel change differences corresponding to the two opposite directions can obtain the weights of the two opposite directions, wherein the weight corresponding to the direction with a larger pixel change difference in the two opposite directions is smaller, and the weight corresponding to the direction with a smaller pixel change difference in the two opposite directions is larger. Taking the west direction in the horizontal interpolation plane as an example, the weight delta of the west direction is calculated. W The expression is as follows:

[0130] Among them, Var E is the pixel change difference in the east direction, such as the aforementioned and Var W is the pixel change difference in the west direction, such as the aforementioned and

[0131] The corresponding east weight delta in the horizontal interpolation plane E The expression is as follows:

[0132] delta E =1-deltaW .

[0133] In the embodiment of the present disclosure, similarly, the asynchronous weights of the remaining interpolation planes are calculated in the same manner as described above, and will not be described in detail herein.

[0134] S33. For each non-G pixel, according to each pair of asynchronous weights and the average value of the G component in the preset direction corresponding to the asynchronous weights, determine the G component of the non-G pixel in the interpolation plane corresponding to each pair of asynchronous weights, and obtain the full-size G component; the interpolation plane is a plane determined by two opposite preset directions corresponding to the asynchronous weights.

[0135] In the embodiment of the present disclosure, according to the pixel distribution characteristics of the RGBIR 4*4 Bayer pattern, the R, B, and IR pixel positions are all surrounded by four G pixels on the top, bottom, left, and right sides. Therefore, the interpolation method expression of the G component in each interpolation plane is as follows:

[0136] interpG plane =delta1*G1+delta2*G2;

[0137] Where delta1 and delta2 represent a set of asynchronous weight values ​​of the interpolation plane (for example, the weight of the west and the weight of the east in the horizontal plane), G1 and G2 represent the average value of the G pixels corresponding to each preset direction in the interpolation plane. Depending on the interpolation plane, the average value of the G pixels in the corresponding direction can be selected for calculation. interpG plane Indicates the interpolated G component value of the current non-G pixel, where plane indicates different interpolation planes, for example, it can be any one of a horizontal plane, a vertical plane, a 45° oblique plane, or a 135° oblique plane.

[0138] In the embodiment of the present disclosure, the G interpolation process when the R pixel position is within four interpolation planes is introduced below by taking the R pixel as the center as an example.

[0139] In the embodiments of the present disclosure, Fig. 9 As shown, it is a schematic diagram of a neighborhood implementation example when interpolating the G component with R (0,0) as the central pixel.

[0140] In the disclosed embodiment, Fig. 9 IR(-1,-1), Gir(-1,0), IR(-1,1), G(0,-1), R(0,0), Grb(0,1), IR(1,-1), Gir(1,0), IR(1,1) and the above Figure 6 The meanings of the related symbols are similar and will not be repeated here.

[0141] In the embodiment of the present disclosure, based on Fig. 9 The schematic diagram can be obtained:

[0142] G component of the horizontal plane interpolated G_hor :

[0143] interpolated G_hor =delta E *Grb(0,-1)+delta W *Grb(0,1);delta E is the eastward weight, delta W is the westward weight;

[0144] G component of the vertical plane interpolated G_ver :

[0145] interpolated G_ver =delta N *Gir(-1,0)+delta S *Gir(1,0);delta N is the northbound weight, delta S is the south weight;

[0146] G component of the oblique 45° plane interpolated G_45 :

[0147] interpolated G_45 =delta NE *(Gir(-1,0)+Grb(0,1)) / 2+delta SW *(Gir(1,0)+Grb(0,-1)) / 2; delta NE is the northeast direction weight, delta SW is the weight for the southwest direction;

[0148] G component of the oblique 135° plane interpolated G_135 :

[0149] interpolated G_135 =delta NW *(Gir(-1,0)+Grb(0,-1)) / 2+delta SW *(Gir(1,0)+Grb(0,1)) / 2; delta NW is the northwest direction weight, delta SW is the southeast direction weight.

[0150] In the embodiment of the present disclosure, the G components of all non-G pixel positions in the four interpolation planes can be calculated through the above expression formula.

[0151] In the embodiment of the present disclosure, after the G component interpolation is completed through the above scheme, the R component and the B component interpolation are performed below. S22, based on the preset mode distribution, the second neighborhood of each non-R pixel of the image to be processed is determined, and a first color difference between each non-R pixel and the corresponding R pixel in the second neighborhood is used to interpolate a plurality of full-size R components in preset directions for each non-R pixel.

[0152] In the embodiment of the present disclosure, when performing R component interpolation, an R position that is more similar to the G component of the non-R pixel to be interpolated can be selected from the corresponding second domain, assigned a greater weight, and participates in the full-size interpolation of the R component based on the color difference corresponding to the R position. In the embodiment of the present disclosure, determining the second neighborhood of each non-R pixel of the image to be processed based on the preset pattern distribution may include:

[0153] Determine the interpolation plane currently interpolating each non-R pixel;

[0154] Determine a second neighborhood that matches the shape of the interpolation plane.

[0155] In the disclosed embodiment, when interpolating each non-R pixel, it is necessary to select the corresponding second neighborhood for the non-R pixel on multiple interpolation planes (such as the aforementioned horizontal plane, vertical plane, oblique 45° plane or oblique 135° plane), respectively. The selection rule of the second neighborhood may include but is not limited to: according to the distribution characteristics of the non-R pixels to be interpolated and the different interpolation planes, the selection range of the second neighborhood is also different, wherein the interpolation neighborhood is selected with the non-R pixel currently interpolated as the center pixel, and a certain advantage is given to the corresponding plane direction according to the interpolation plane. For example, for the horizontal plane, more attention will be paid to the pixels in the length direction of the second neighborhood, that is, the number of pixels in the length direction will be more than that in the width direction, such as selecting a 7*3 second neighborhood; similarly, for the vertical plane, more attention will be paid to the pixels in the width direction of the second neighborhood, that is, the number of pixels in the width direction is more than that in the length direction, such as selecting a 3*7 second neighborhood. Similarly, the neighborhood selection criteria for the oblique 45° plane and the oblique 135° plane are consistent.

[0156] In the embodiment of the present disclosure, the length direction, width direction, and center pixel of the second neighborhood in the image to be processed can be as follows: Fig.10 shown.

[0157] In the embodiment of the present disclosure, the second neighborhood may include but is not limited to several formats such as 7*3, 3*7, 5*5, etc., and the range of the corresponding second neighborhood may be different depending on the central pixel.

[0158] In the embodiments of the present disclosure, Fig.11 As shown, using the first color difference between each non-R pixel and the corresponding R pixel in the second neighborhood, a plurality of full-size R components in a preset direction are interpolated for each non-R pixel, including: for each non-R pixel, performing the following interpolation operations S41-S43 on each interpolation plane:

[0159] S41, determining a first G component of the non-R pixel on the interpolation plane, determining a second G component of each R position in the second neighborhood on the interpolation plane, and taking a difference between each second G component and the first G component as a first color difference of each R position on the interpolation plane;

[0160] S42, obtaining the distance between each R position and the non-R pixel, and determining a first color difference fusion coefficient corresponding to the R position according to the distance;

[0161] S43, determining the R component interpolated from the non-R pixel on the interpolation plane according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the R position corresponding to the first color difference, and the first color difference fusion coefficient.

[0162] In the embodiment of the present disclosure, the R component interpolated from the non-R pixel on the interpolation plane is determined according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction where the R position corresponding to the first color difference is located, and the first color difference fusion coefficient, including:

[0163] Multiply each first color difference by the first color difference fusion coefficient corresponding to the R position corresponding to the first color difference, to obtain the first fused color difference corresponding to each R position;

[0164] Add all first fusion color differences in the same preset direction of the interpolation plane, and divide the sum of the first color difference fusion coefficients corresponding to all R positions in the same preset direction to obtain the RG color difference in the same preset direction;

[0165] Multiplying each RG color difference in the same preset direction by the weight corresponding to the same preset direction to obtain the weighted RG color difference in the same preset direction;

[0166] All weighted RG color differences calculated for the interpolation plane and the full-size G component are added together to obtain the R component interpolated from the non-R pixel on the interpolation plane.

[0167] In the embodiment of the present disclosure, according to the pixel distribution characteristics of RGBIR, non-R pixels at different positions are interpolated on any interpolation plane to calculate the R component interpR plane The process expression is as follows:

[0168]

[0169]

[0170]

[0171] In the above formula, interpRG plane It represents the G component interpolation result of the non-R pixel (center pixel) currently interpolated on the calculated interpolation plane. deltaR1 and deltaR2 represent a set of asynchronous weight values ​​of the interpolation plane. and Represents the (RG) color difference (i.e., the first color difference) result of the corresponding direction on the interpolation plane, wherein alphaR (as shown in alphaR1, alphaR2, ..., alphaRn, where n is a positive integer) represents the first color difference fusion coefficient of the color difference result of pixels at different R positions (i.e., the color difference between the pixel at the R position and the center pixel). According to the distance criterion, generally, the closer a certain R position in the second field is to the non-R pixel (center pixel) to be interpolated, the larger the first fusion coefficient assigned to the R position is. On each interpolation plane, the selected color difference interpolation pixels (i.e., R positions, or R pixels) are symmetrically distributed, so the values ​​of the first color difference fusion coefficient alphaR in asynchronous directions on the same interpolation plane are the same. Among them, R (as shown in R1, R2, ..., Rn) and R′ (as shown in R′1, R′2, ..., R′n) represent the pixel value of the R position in the asynchronous direction (as the second G component), and R1-G1, R2-G2, Rn-Gn, R′1-G′1, R′2-G′2, R′nG′n are the first color differences of the corresponding R positions, respectively.

[0172] In the embodiment of the present disclosure, before determining the R component interpolated from the non-R pixel on the interpolation plane according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction where the R position corresponding to the first color difference is located, and the first color difference fusion coefficient, the method may further include:

[0173] Comparing each first color difference with a preset first color difference threshold to obtain a first comparison result;

[0174] It is determined whether to adopt the first color difference to determine the R component interpolated from the non-R pixel on the interpolation plane according to the first comparison result.

[0175] In the disclosed embodiment, the above-mentioned R component interpolation process is used in ordinary scenarios. However, for the interpolation process of R pixels, the first color difference in the second neighborhood is not necessarily adopted. Because in areas such as edge transitions, the image may easily have interpolation pseudo-color due to different selections of the second neighborhood. Therefore, the R pixels in the second neighborhood are screened, and the screening rules mainly include: using the difference between the first G component of the central pixel and the second G component of the R pixel in the second neighborhood (i.e., the first color difference size) to determine.

[0176] In an embodiment of the present disclosure, determining whether to use the first color difference to determine the R component interpolated from the non-R pixel on the interpolation plane according to the first comparison result includes:

[0177] When any one or more first color differences are less than the first color difference threshold, use any one or more first color differences to determine an R component interpolated from a non-R pixel on an interpolation plane;

[0178] When each first color difference is greater than or equal to the first color difference threshold, the average value of the first color differences of the first R positions closest to the non-R pixels in the second neighborhood is used as the first color difference of each first R position, and the first color difference of the first R position is used to determine the R component interpolated from the non-R pixel on the interpolation plane.

[0179] In the embodiment of the present disclosure, when the absolute value of the difference between the second G component of the R pixel in the second neighborhood and the first G component of the central pixel is less than the first color difference threshold threshold1, the above scheme can be directly used to complete the interpolation process of the R component of the current non-R pixel.

[0180] In the embodiment of the present disclosure, when the absolute value of the difference between the second G component of a part of the R pixels in the second neighborhood and the first G component of the central pixel is greater than or equal to the first color difference threshold threshold1, the first color difference of this part of the R pixels will not be used in the second neighborhood to participate in the above-mentioned R component interpolation operation.

[0181] In the embodiment of the present disclosure, when the absolute value of the difference between the second G component of all R pixels in the current interpolation plane and the first G component of the central pixel is greater than or equal to the first color difference threshold threshold1, then for the interpolation plane, the average value of the first color difference of the R pixels closest to the central pixel position can be directly used to participate in the R component interpolation operation of the current non-R pixel.

[0182] In the embodiment of the present disclosure, the above scheme can be used to complete the interpolation of the R component of any non-R pixel in multiple interpolation planes.

[0183] In the embodiments of the present disclosure, Fig.12As shown, it is a schematic diagram of a second neighborhood implementation example when interpolating the R component on a vertical plane with IR as the center pixel (IR at 45° to R), and the domain range of the second neighborhood is 7*3. Among them, Grb refers to the G pixel in the row containing R and B pixels, Gir refers to the G pixel in the row containing IR pixels, B1-G1 refers to the G1 component (i.e. G component) at the position of B1 pixel (i.e. B pixel), R3-G3 refers to the G3 component (i.e. G component) at the position of R1 pixel (i.e. R pixel), R1-G1 refers to the G1 component (i.e. G component) at the position of R1 pixel (i.e. R pixel), and so on. B3-G3, B2-G2, R4-G4, R2-G2, and B4-G4 have similar meanings and will not be repeated here. The following is based on Fig.12 The R component at the interpolation IR position is described.

[0184] In the embodiment of the present disclosure, when the difference between the vertical plane G component at the IR position and the vertical G components of the four R positions in the neighborhood are respectively less than the threshold threshold1, that is, the four RG in the neighborhood can all participate in the interpolation calculation of the vertical plane R component at the IR position. According to the distance criterion, the closer the distance to the center pixel IR, the greater the interpolation weight assigned to it, that is, the greater the first color difference fusion coefficient.

[0185]

[0186]

[0187]

[0188] src ver =Ir_b45_ver_G;

[0189] In the above formula, IR_b45_R_ver is the R pixel value of the vertical plane interpolated at the IR (center pixel) position at 45° to the R position, and src ver is the vertical plane G component at the IR position, Indicates the first color difference value in the north direction on the vertical plane, deltaR N Indicates the weight of the north direction obtained when the IR position is in the vertical plane, alphaR1 and alphaR2 indicate the first color difference fusion coefficient. R1-G1 and R3-G3 are the first color differences of the two R positions in the north direction.

[0190] In the disclosed embodiment, since the positional relationship between IR and R is rather special and spans rows and columns, in some narrow areas or at the edges of sudden brightness changes, the chromatic aberration interpolation may be unstable due to the spanning of rows and columns. Therefore, it is necessary to screen and judge G on the vertical plane, and preferentially select the difference between the G component at the IR position and the G component at the R position of the selected position within the 3*3 neighborhood block centered on the IR pixel for screening.

[0191] In the embodiment of the present disclosure, if abs(RG4_ver-src ver )>threshold1, and abs(RG1_ver-src ver )>threshold1, then:

[0192]

[0193]

[0194] IR_b45_R_ver=src ver +deltaR N *alphaR1*(R3-G3_ver)+(1-deltaR N )*alphaR2*(R2-G2_ver);

[0195] Among them, abs() is the absolute value function, RG4_ver refers to the G component at the R4-G4 position on the vertical plane, (R1-G1) is the first color difference value at the R1-G1 position; RG1_ver refers to the G component at the R1-G1 position on the vertical plane, (R4-G4) is the first color difference value at the R4-G4 position, the first color difference at the R1-G1 position and the R4-G4 position is not available, and only the first color difference at the R2-G2 position and the R3-G3 position is used.

[0196] In the embodiment of the present disclosure, if abs(RG1_ver-src ver )>threshold1, abs(RG2_ver-src ver )>threshold1, abs(RG3_ver-src ver )>threshold1, abs(RG4_ver-src ver )>threshold1, then:

[0197] IR_b45_R_ver=src ver +((R4-G4)+(R1-G1)) / 2;

[0198] Among them, RG2_ver refers to the G component of the R2-G2 position on the vertical plane, RG3_ver refers to the G component of the R3-G3 position on the vertical plane, and ((R4-G4)+(R1-G1)) / 2 is used as the average value here because the R1-G1 position and the R4-G4 position are closest to the center pixel.

[0199] In the embodiment of the present disclosure, at this point, the interpolation calculation of the R component of the vertical plane at the IR center position that is 45° to R is completed.

[0200] In the embodiments of the present disclosure, Fig.13 As shown, it is a schematic diagram of a second neighborhood implementation example when interpolating the R component on the horizontal plane with IR as the center pixel (IR at 45° to R). The area range of the second neighborhood is 3*7. Fig.13 The idea of ​​interpolating the R component at the IR position in the horizontal plane is the same as the idea of ​​interpolating the R component at the IR position in the vertical plane mentioned above, and the principles are the same, which will not be repeated here.

[0201] In the embodiments of the present disclosure, Fig.14 As shown, it is a schematic diagram of a second neighborhood implementation example when interpolating the R component of the oblique 45° plane with lR as the center pixel. The area range of the second neighborhood is 7*7. Fig.14 The R component at the interpolated IR position is illustrated in the oblique 45° plane.

[0202] In the embodiments of the present disclosure, B5-G5, B6-G6, B7-G7, B8-G8, R5-G5, R6-G6, R7-G7, R8-G8 and Fig.12 The Chinese symbols have similar meanings.

[0203] In the embodiment of the present disclosure, when the difference between the oblique 45° plane G component at the IR position and the oblique 45° G components of the four R positions in the neighborhood are respectively less than the threshold threshold1, that is, the four RG in the neighborhood can all participate in the interpolation calculation of the oblique 45° plane R component at the IR position. According to the distance criterion, the closer the distance to the center pixel IR, the greater the interpolation weight assigned to it, that is, the greater the first color difference fusion coefficient.

[0204]

[0205]

[0206]

[0207] src 45 =IR_b45_45_G.

[0208] In the above formula, IR_b45_R_45 is the R pixel value of the oblique 45° plane interpolated at the IR (center pixel) position at 45° to R, and src 45 is the oblique 45° plane G component at the IR position, Indicates the first color difference value in the NE (northeast) direction on the oblique 45° plane. It represents the first color difference value in the SW (southwest) direction on the oblique 45° plane, deltaR_NE represents the weight of the northeast direction obtained when the IR position is on the oblique 45° plane, and alphaR3 and alphaR4 represent the first color difference fusion coefficient. R2-G2_45, R7-G7_45, R5-G5_45, and R6-G6_45 are the first color differences of the four R positions R2-G2, R7-G7, R5-G5, and R6-G6 in the northeast direction. R2-G2_45, R7-G7_45, R3-G3_45, and R4-G4_45 are the first color differences of the four R positions R2-G2, R7-G7, R3-G3, and R4-G4 in the southwest direction. Among them, R2-G2 and R7-G7 are exactly in the middle area, so the first color differences R2-G2_45 and R7-G7_45 of these two positions are calculated in both the northeast and southwest directions. By increasing the calculation data, the calculation accuracy is improved.

[0209] In the disclosed embodiment, since the positional relationship between IR and R is rather special and spans rows and columns, in some narrow areas or at the edges of sudden brightness changes, the chromatic aberration interpolation may be unstable due to the spanning of rows and columns. Therefore, it is necessary to screen and judge G on the vertical plane, and preferentially select the difference between the G component at the IR position and the G component at the R position of the selected position within the 3*3 neighborhood block centered on the IR pixel for screening.

[0210] In the embodiment of the present disclosure, if abs(RG2_45-src 45 )>threshold1, and abs(RG7_45-src 45 )threshold1, then:

[0211]

[0212]

[0213] IR_b45_R_45=src 45 +((R5-G5 45 )+(R6-G6 45 )+(R3-G3_45)+(R4-G4_45)) / 2;

[0214] Among them, abs() is the absolute value function, RG2_45 refers to the G component at the R2-G2 position on the oblique 45° plane, RG7_45 refers to the G component at the R7-G7 position on the oblique 45° plane, the first color difference at the R2-G2 position and the R7-G7 position is not available, and only the first color difference at the R5-G5 position and the R6-G6 position is used.

[0215] In the embodiment of the present disclosure, if abs(RG2_45-src 45 )>threshold1,abs(RG7_45-src 45 )threshold1,abs(RG5_45-src 45 )>threshold1,abs(RG5_45-src 45 )threshold1,abs(RG3_45-src 45 )>threshold1, and abs(RG4_45-src 45 )threshold1, then:

[0216] IR_b45_R_45=src 45 +(R2-G2_45)+(R7-G7_45)) / 2;

[0217] Among them, RG5_45r refers to the G component of the R5-G5 position on the oblique 45° plane, RG6_45 refers to the G component of the R6-G6 position on the oblique 45° plane, RG3_45 refers to the G component of the R3-G3 position on the oblique 45° plane, RG4_45 refers to the G component of the R4-G4 position on the oblique 45° plane, (R2-G2_45)+(R7-G7_45)) / 2 is used here as the average value, because the R2-G2 position and the R7-G7 position are closest to the center pixel.

[0218] In the embodiment of the present disclosure, at this point, the interpolation calculation of the R component of the oblique 45° plane at the IR center position is completed.

[0219] In the embodiment of the present disclosure, the idea of ​​interpolating the R component at the IR position in the oblique 135° plane is the same as the idea of ​​interpolating the R component at the IR position in the oblique 45° plane mentioned above, and the principles are the same, which will not be repeated here.

[0220] In the embodiments of the present disclosure, Fig.15 As shown, it is a schematic diagram of a second neighborhood implementation example when interpolating the R component on the horizontal plane with B as the center pixel. The area range of the second neighborhood is 5*5. Fig.15 The R component at the interpolated IR position is illustrated in the horizontal plane.

[0221] In the embodiment of the present disclosure, when the difference between the horizontal plane G component at the B position and the horizontal plane G component of the four R positions in the neighborhood are respectively less than the threshold threshold1, that is, the four RG in the neighborhood can all participate in the interpolation calculation of the horizontal plane R component at the B position. According to the distance criterion, the closer the distance to the center pixel B, the greater the interpolation weight assigned to it, that is, the greater the first color difference fusion coefficient.

[0222]

[0223]

[0224]

[0225] src hor =B_hor_G;

[0226] In the above formula, R_R_hor is the R pixel value of the horizontal plane interpolated at the R (center pixel) position, src hor (i.e. B_hor_G) is the horizontal plane G component at position B, Represents the first color difference value in the E (east) direction on the horizontal plane, It represents the first color difference value in the W (west) direction on the horizontal plane, and deltaR_E represents the weight of the east direction obtained at the B position on the horizontal plane. R3-G3_hor and R2-G2_hor are the first color differences of the two R positions R3-G3 and R2-G2 in the east and west directions respectively.

[0227] In the embodiment of the present disclosure, if abs(RG2_hor-src hor )>threshold1, then:

[0228] R_R_hot=src hor +R3-G3-hor;

[0229] Wherein, abs() is an absolute value function, RG2_hor refers to the G component at the R2-G2 position on the horizontal plane, the first color difference at the R2-G2 position is not available, and only the first color difference at the R3-G3 position (R3-G3_hor) is used.

[0230] In the embodiment of the present disclosure, if abs(RG2_hor-src hor )>threshold1, abs(RG3_hor-src hor )>threshold1, then:

[0231] R_R_hor=src hor+((R3-G3_hor)+(R2-G2_hor)) / 2;

[0232] Among them, RG3_hot refers to the G component of the R3-G3 position on the horizontal plane, and ((R3-G3_hor)+(R2-G2_hor)) / 2 is taken as the average value here, because the R2-G2 position and the R3-G3 position are at the same distance from the center pixel and are both taken as the closest distance.

[0233] In the embodiment of the present disclosure, at this point, the interpolation calculation of the R component of the horizontal plane at the center position of B is completed.

[0234] In the embodiment of the present disclosure, the idea of ​​interpolating the R component at the B position in the vertical plane is the same as the idea of ​​interpolating the R component at the B position in the horizontal plane mentioned above, and the principles are the same, which will not be repeated here.

[0235] S23, determining a third neighborhood of each non-B pixel of the image to be processed based on a preset pattern distribution, and interpolating a plurality of full-size B components in preset directions for each non-B pixel by using a second color difference between each non-B pixel and a corresponding B pixel in the third neighborhood.

[0236] In an embodiment of the present disclosure, determining the third neighborhood of each non-B pixel of the image to be processed based on a preset pattern distribution includes:

[0237] Determine an interpolation plane for currently interpolating each non-B pixel;

[0238] Determine a third neighbor that matches the shape of the interpolation plane.

[0239] In the embodiment of the present disclosure, when interpolating each non-B pixel, it is necessary to select the corresponding third neighborhood for the non-B pixel on multiple interpolation planes (such as the aforementioned horizontal plane, vertical plane, oblique 45° plane or oblique 135° plane), respectively. The selection rule of the third neighborhood may include but is not limited to: according to the distribution characteristics of the non-B pixels to be interpolated and the different interpolation planes, the selection range of the third neighborhood is also different, wherein the interpolation neighborhood is selected with the current interpolated non-B pixel as the center pixel, and a certain advantage is given to the corresponding plane direction according to the interpolation plane. For example, for the horizontal plane, more attention will be paid to the pixels in the length direction of the third neighborhood, that is, the number of pixels in the length direction will be more than that in the width direction, such as selecting a 7*3 third neighborhood; similarly, for the vertical plane, more attention will be paid to the pixels in the width direction of the third neighborhood, that is, the number of pixels in the width direction is more than that in the length direction, such as selecting a 3*7 third neighborhood. Similarly, the neighborhood selection criteria for the oblique 45° plane and the oblique 135° plane are consistent.

[0240] In the embodiment of the present disclosure, the length direction, width direction, and center pixel of the third neighborhood in the image to be processed can be as follows: Fig.10 shown.

[0241] In the embodiment of the present disclosure, the third neighborhood may include but is not limited to several formats such as 7*3, 3*7, 5*5, etc., and the range of the corresponding third neighborhood may be different depending on the central pixel.

[0242] In the embodiments of the present disclosure, Fig.16 As shown, using the second color difference between each non-B pixel and the corresponding B pixel in the third neighborhood, a plurality of full-size B components in a preset direction are interpolated for each non-B pixel, including: for each non-B pixel, performing the following interpolation operations S51-S53 on each interpolation plane:

[0243] S51, determining a third G component of the non-B pixel on the interpolation plane, determining a fourth G component of each B position in the third neighborhood on the interpolation plane, and taking a difference between each fourth G component and the third G component as a second color difference of each B position on the interpolation plane;

[0244] S52, obtaining the distance between each B position and the non-B pixel, and determining the second color difference fusion coefficient corresponding to the B position according to the distance;

[0245] S53, determining the B component interpolated from the non-B pixel on the interpolation plane according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the B position corresponding to the second color difference, and the second color difference fusion coefficient.

[0246] In the embodiment of the present disclosure, the B component interpolated from the non-B pixel on the interpolation plane is determined according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the B position corresponding to the second color difference, and the second color difference fusion coefficient, including:

[0247] Multiply each second color difference by the second color difference fusion coefficient corresponding to the B position corresponding to the second color difference, to obtain the second fused color difference corresponding to each B position;

[0248] Add all the second fusion color differences in the same preset direction of the interpolation plane, and divide the sum of the second color difference fusion coefficients corresponding to all B positions in the same preset direction to obtain the BG color difference in the same preset direction;

[0249] Multiplying the BG color difference in each same preset direction by the weight corresponding to the same preset direction to obtain the weighted BG color difference in the same preset direction;

[0250] All weighted BG color differences calculated for the interpolation plane and the full-size G component are added together to obtain the B component interpolated from the non-B pixel on the interpolation plane.

[0251] In the embodiment of the present disclosure, according to the pixel distribution characteristics of RGBIR, non-B pixels at different positions are interpolated on any interpolation plane to calculate the B component interpB plane The process expression is as follows:

[0252]

[0253]

[0254]

[0255] In the above formula, interpBG plane It represents the G component interpolation result of the non-B pixel (center pixel) currently interpolated on the calculated interpolation plane. deltaB1 and deltaB2 represent a set of asynchronous weight values ​​of the interpolation plane. and Represents the (BG) color difference (i.e., second color difference) result of the corresponding direction on the interpolation plane, wherein alphaB (as shown in alphaB1, alphaB2, ..., alphaBm, where m is a positive integer) represents the second color difference fusion coefficient of the color difference result of pixels at different B positions (i.e., the color difference between the pixel at the B position and the center pixel). According to the distance criterion, generally, the closer a certain B position in the third field is to the non-B pixel (center pixel) to be interpolated, the larger the first fusion coefficient assigned to the B position is. On each interpolation plane, the selected color difference interpolation pixels (i.e., B positions, or B pixels) are symmetrically distributed, so the values ​​of the second color difference fusion coefficient alphaB in asynchronous directions on the same interpolation plane are the same. Among them, B (as shown in B1, B2, ..., Bm) and B′ (as shown in B′1, B′2, ..., B′m) represent the pixel values ​​of the B position in the asynchronous direction (such as the fourth G component), and B1-G1, B2-G2, Bm-Gm, B′1-G′1, B′2-G′2, B′mG′m are the second color differences of the corresponding B positions, respectively.

[0256] In the embodiment of the present disclosure, before determining the B component interpolated from the non-B pixel on the interpolation plane according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the B position corresponding to the second color difference, and the second color difference fusion coefficient, the method further includes:

[0257] Comparing each second color difference with a preset second color difference threshold to obtain a second comparison result;

[0258] It is determined whether to use the second color difference to determine the B component interpolated from the non-B pixel on the interpolation plane according to the second comparison result.

[0259] In the disclosed embodiment, the above-mentioned B component interpolation process is used in ordinary scenarios. However, for the interpolation process of B pixels, the second color difference in the third neighborhood is not necessarily adopted. Because in areas such as edge transitions, the image may easily have interpolation pseudo-color due to different selections of the third neighborhood. Therefore, the B pixels in the third neighborhood are screened, and the screening rules mainly include: determining by using the difference between the third G component of the central pixel and the fourth G component of the B pixel in the third neighborhood (i.e., the second color difference size).

[0260] In an embodiment of the present disclosure, determining whether to use the second color difference to determine the B component interpolated from the non-B pixel on the interpolation plane according to the second comparison result includes:

[0261] When any one or more second color differences are less than the second color difference threshold, use any one or more second color differences to determine a B component interpolated from a non-B pixel on an interpolation plane;

[0262] When each second color difference is greater than or equal to the second color difference threshold, the average value of the second color differences of the first B positions closest to the non-B pixels in the third neighborhood is used as the second color difference of each first B position, and the second color difference of the first B position is used to determine the B component interpolated from the non-B pixel on the interpolation plane.

[0263] In the embodiment of the present disclosure, when the absolute value of the difference between the fourth G component of the B pixel in the third neighborhood and the third G component of the central pixel is less than the second color difference threshold threshold2, the above scheme can be directly used to complete the interpolation process of the B component of the current non-B pixel.

[0264] In the embodiment of the present disclosure, when the absolute value of the difference between the fourth G component of a part of B pixels in the third neighborhood and the third G component of the central pixel is greater than or equal to the second color difference threshold threshold2, the second color difference of this part of B pixels in the third neighborhood will not be used to participate in the above-mentioned B component interpolation operation.

[0265] In the embodiment of the present disclosure, when the absolute value of the difference between the fourth G component of all B pixels in the current interpolation plane and the third G component of the central pixel is greater than or equal to the second color difference threshold threshold2, then for the interpolation plane, the average value of the second color difference of the B pixels closest to the central pixel position can be directly used to participate in the B component interpolation operation of the current non-B pixel.

[0266] In the embodiment of the present disclosure, the above scheme can be used to complete the interpolation of the B component of any non-B pixel in multiple interpolation planes.

[0267] In the embodiments of the present disclosure, Fig.12 As shown in FIG. 1 , a schematic diagram of a third neighborhood when interpolating the B component on a vertical plane with IR as the center (IR at 45° to B) is shown. The range of the third neighborhood is 7*3. Fig.12 The B component at the interpolation IR position is described.

[0268] In the embodiment of the present disclosure, when the differences between the vertical plane G component at the IR position and the vertical G components of the four B positions in the neighborhood are respectively less than the threshold threshold1, that is, the four BGs in the neighborhood can all participate in the interpolation calculation of the vertical plane B component at the IR position. According to the distance criterion, the closer the distance to the center pixel IR, the greater the interpolation weight assigned to it, that is, the greater the second color difference fusion coefficient.

[0269]

[0270]

[0271]

[0272] src ver =Ir_b45_ver_G;

[0273] In the above formula, IR_b45_B_ver is the B pixel value of the vertical plane interpolated at the IR (center pixel) position at 45° to the B position, src ver is the vertical plane G component at the IR position, Indicates the second color difference value in the north direction on the vertical plane, deltaB N Indicates the weight of the north direction obtained on the vertical plane at the IR position, alphaB1 and alphaB2 represent the first color difference fusion coefficients. B1-G1 and B3-G3 are the second color differences of the two B positions in the north direction.

[0274] In the disclosed embodiment, since the positional relationship between IR and B is rather special and spans rows and columns, in some narrow areas or at the edges of sudden brightness changes, the color difference interpolation may be unstable due to the spanning of rows and columns. Therefore, it is necessary to screen and judge G on the vertical plane, and preferentially select the difference between the G component at the IR position and the G component at the B position of the selected position within the 3*3 neighborhood block centered on the IR pixel for screening.

[0275] In the embodiment of the present disclosure, if abs(BG4_ver-src ver)>threshold1, and abs(BG1_ver-src ver )>threshold1, then:

[0276]

[0277]

[0278] IR_b45_B_ver=src ver +deltaB N *alphaB1*(B3-G3)+(1-deltaB N )*alphaB2*(B2-G2);

[0279] Among them, abs() is the absolute value function, BG4_ver refers to the G component at the R4-G4 position on the vertical plane, and (B1-G1) is the second color difference value at the B1-G1 position; BG1_ver refers to the G component at the R1-G1 position on the vertical plane, and (B4-G4) is the second color difference value at the B4-G4 position. The second color differences at the B1-G1 position and the B4-G4 position are not available, and only the second color differences at the B2-G2 position and the B3-G3 position are used.

[0280] In the embodiment of the present disclosure, if abs(BG1_ver-src ver )>threshold1, abs(BG2_ver-src ver )>threshold1, abs(BG3_ver-src ver )>threshold1, abs(BG4_ver-src ver )>threshold1, then:

[0281] IR_b45_R_ver=src ver +((B2-G2)+(B3-G3)) / 2;

[0282] Among them, BG2_ver refers to the G component of the B2-G2 position on the vertical plane, BG3_ver refers to the G component of the B3-G3 position on the vertical plane, and ((B2-G2)+(B3-G3)) / 2 is used as the average value here because the B2-G2 position and the B3-G3 position are closest to the center pixel.

[0283] In the embodiment of the present disclosure, at this point, the interpolation calculation of the B component of the vertical plane at the IR center position that is 45° to B is completed.

[0284] In the embodiments of the present disclosure, Fig.13As shown, it is a schematic diagram of a third neighborhood implementation example when interpolating the R component on the horizontal plane with IR as the center pixel (IR at 45° to B). The area range of the third neighborhood is 3*7. Fig.13 The idea of ​​interpolating the B component at the IR position in the horizontal plane is the same as the idea of ​​interpolating the B component at the IR position in the vertical plane, and the principles are the same, which will not be repeated here.

[0285] In the embodiments of the present disclosure, Fig.14 As shown, it is a schematic diagram of a third neighborhood implementation example when interpolating the B component on a 45° oblique plane with IR as the center pixel. The area range of the third neighborhood is 7*7. Fig.14 The B component at the interpolated IR position is illustrated in an oblique 45° plane.

[0286] In the embodiment of the present disclosure, when the difference between the oblique 45° plane G component at the IR position and the oblique 45° G components of the four R positions in the neighborhood are respectively less than the threshold threshold1, that is, the four RG in the neighborhood can all participate in the interpolation calculation of the oblique 45° plane B component at the IR position. According to the distance criterion, the closer the distance to the center pixel IR, the greater the interpolation weight assigned to it, that is, the greater the second color difference fusion coefficient.

[0287]

[0288]

[0289]

[0290] src 45 =IR_b45_45_G.

[0291] In the above formula, IR_b45_B_45 is the pixel value of B in the oblique 45° plane obtained by interpolating at the IR (center pixel) position at 45° to B, and src 45 is the oblique 45° plane G component at the IR position, Indicates the second color difference value in the NE (northeast) direction on the oblique 45° plane. Indicates the second color difference value in the SW (southwest) direction on the oblique 45° plane, deltaB_NE indicates the weight of the northeast direction obtained when the IR position is on the oblique 45° plane, alphaB3 and alphaB4 indicate the second color difference fusion coefficient. B5-G5_45 and B6-G6_45 are the second color differences of the two B positions B5-G5 and B6-G6 in the northeast, and B3-G3_45 and B4-G4_45 are the second color differences of the two B positions B3-G3 and B4-G4 in the southwest.

[0292] In the disclosed embodiment, since the positional relationship between IR and R is rather special and spans rows and columns, in some narrow areas or at the edges of sudden brightness changes, the chromatic aberration interpolation may be unstable due to the spanning of rows and columns. Therefore, it is necessary to screen and judge G on the vertical plane, and preferentially select the difference between the G component at the IR position and the G component at the R position of the selected position within the 3*3 neighborhood block centered on the IR pixel for screening.

[0293] In the embodiment of the present disclosure, if abs(BG3_45-src 45 )>threshold1, and abs(BG6_45-src 45 )threshold1, then:

[0294] IR_b45_R_45=src 45 +(B5-G5_45)+(B4-G4_45);

[0295] Among them, abs() is the absolute value function, BG3_45 refers to the G component at the B3-G3 position on the oblique 45° plane, BG6_45 refers to the G component at the B6-G6 position on the oblique 45° plane, the second color difference at the B3-G3 position and the B6-G6 position is not available, and only the second color difference at the B4-G4 position and the B5-G5 position is used.

[0296] In the embodiment of the present disclosure, if abs(BG3_45-src 45 )>threshold1,abs(BG6_45-src 45 )threshold1,abs(BG5_45-src 45 )>threshold1, and abs(BG4_45-src 45 )threshold1, then:

[0297] IR_b45_R_45=src 45 +((B3-G3_45)+(B6-G6_45)) / 2;

[0298] Among them, BG4_45 refers to the G component of the B4-G4 position on the oblique 45° plane, BG5_45 refers to the G component of the B5-G5 position on the oblique 45° plane, and ((B3-G3_45)+(B6-G6_45)) / 2 is used here as the average value because the B3-G3 position and the B6-G6 position are closest to the center pixel.

[0299] In the embodiment of the present disclosure, at this point, the interpolation calculation of the B component of the oblique 45° plane at the IR center position is completed.

[0300] In the embodiment of the present disclosure, the idea of ​​interpolating the B component at the IR position in the oblique 135° plane is the same as the idea of ​​interpolating the B component at the IR position in the oblique 45° plane mentioned above, and the principles are the same, which will not be repeated here.

[0301] In the embodiments of the present disclosure, Fig.17 As shown, it is a schematic diagram of a third neighborhood implementation example when interpolating the B component on the horizontal plane with R as the center pixel. The area range of the third neighborhood is 5*5. Fig.17 The B component at the interpolated R position is described in the horizontal plane.

[0302] In the embodiment of the present disclosure, when the difference between the horizontal plane G component at the R position and the horizontal plane G components of the four B positions in the neighborhood are respectively less than the threshold threshold1, that is, the four BGs in the neighborhood can all participate in the interpolation calculation of the horizontal plane B component at the R position. According to the distance criterion, the closer the distance to the center pixel R, the greater the interpolation weight assigned to it, that is, the greater the second color difference fusion coefficient.

[0303]

[0304]

[0305]

[0306] src hor =R_hor_G;

[0307] In the above formula, R_B_hor is the B pixel value of the horizontal plane interpolated at the R (center pixel) position, src hor (i.e. R_hor_G) is the horizontal plane G component at the R position, Indicates the second color difference value in the E (east) direction on the horizontal plane, Indicates the second color difference value in the W (west) direction on the horizontal plane, deltaB_E indicates the weight of the east direction obtained at the R position on the horizontal plane. B3-G3_hor and B2-G2_hor are the second color differences of the two B positions B3-G3 and B2-G2 in the east and west directions respectively.

[0308] In the embodiment of the present disclosure, if abs(BG2_hor-src hor )>threshold1, then:

[0309] R_B_hor=src hor +B3-G3_hor;

[0310] Among them, abs() is the absolute value function, BG2_hor refers to the G component at the B2-G2 position on the horizontal plane, the second color difference at the B2-G2 position is not available, and only the second color difference at the B3-G3 position (B3-G3_hor) is used.

[0311] In the embodiment of the present disclosure, if abs(BG2_hor-src hor )>threshold1, abs(BG3_hor-src hor )>threshold1, then:

[0312] R_B_hot=src hor +((B3-G3_hor)+(B2-G2-hor)) / 2;

[0313] Among them, BG3_hor refers to the G component of the B3-G3 position on the horizontal plane, and ((B3-G3_hor)+(B2-G2_hor)) / 2 is taken as the average value here, because the B2-G2 position and the B3-G3 position are at the same distance from the center pixel and are both taken as the closest distance.

[0314] In the embodiment of the present disclosure, at this point, the interpolation calculation of the B component of the horizontal plane at the center position of R is completed.

[0315] In the embodiment of the present disclosure, the idea of ​​interpolating the B component at the R position in the vertical plane is the same as the idea of ​​interpolating the B component at the R position in the horizontal plane mentioned above, and the principles are the same, which will not be repeated here.

[0316] S24, taking the image to be processed after the interpolation of the G component, the R component and the B component as an RGB image.

[0317] In the disclosed embodiment, through the above-mentioned G component interpolation, R component interpolation and B component interpolation, the interpolation process of the full-size RGB image of each interpolation plane is completed to obtain the RGB image.

[0318] In the disclosed embodiment, Bicubic upsampling can be directly used to complete the full-size interpolation process for the image to be processed to obtain a full-size IR image.

[0319] In the disclosed embodiment, a full-size four-channel image can be obtained after RGB interpolation and IR interpolation are performed on the image to be processed.

[0320] In the embodiment of the present disclosure, the following calculation formula can be used for the IR separation processing of the four-channel image: R′=a1*R+a2*G+a3*B+a4*IR, G′=b1*R+b2*G+b3*B+b4*IR, B′=c1*R+c2*G+c3*B+c4*IR; wherein R′, B′ and G′ represent the pixel values ​​of the R, G, and B pixel values ​​of the current pixel after the IR separation operation; a1, a2, a3, a4, b1, b2, b3, b4, c1, c2, c3, and c4 are all preset parameters.

[0321] In the disclosed embodiment, the target interpolation direction selection method can be implemented by using the currently available related technologies, for example, the interpolation direction selection can be performed by using the uniformity judgment method. The disclosed embodiment mainly uses this method for the first time in the Remosaic process, and the target interpolation direction selection is performed after IR separation, which effectively improves the accuracy of the interpolation direction judgment and has a good effect on image color restoration.

[0322] In the disclosed embodiment, the full-size RGB image after the target interpolation direction selection is completed can be downsampled according to the Bayer pattern to realize the conversion from the RGBIR 4*4 pattern to the ordinary Bayer 2*2, and then the converted image is input into the ISP pipeline for image processing.

[0323] The disclosed embodiments take into account that the relevant interpolation technology does not take into account the sparse distribution of pixels in the RGBIR 4*4 module, resulting in pseudo-color in the interpolated image at the edge or in the narrow object area, and the influence of the IR component, resulting in inaccurate calculation of the interpolation direction, which will cause the subjective quality of the final color image to decrease. The current interpolation methods all perform calculations on directional gradient information before IR separation, resulting in inaccurate interpolated pixel values, especially focusing on the process of interpolating R at the B position and interpolating B at the IR position. In order to overcome the shortcomings of the related technologies, the idea proposed in the embodiments of the present disclosure is based on the stability of the G component. First, the gradient information in different directions is calculated, and the full-size G components in each direction are interpolated. Then, based on the full-size G components in different directions, the gradient information of the G components and the principle of constant color difference are used to screen the interpolation pixels (R pixels or B pixels) on the interpolation plane using the similarity of the full-size G components in the selected neighborhood. Then, the selected interpolation pixels are assigned interpolation weights (i.e., color difference fusion coefficients) of different proportions according to the distance criterion, and the full-size R and B components in the corresponding directions are restored, thereby improving the interpolation stability of the special RGBIR module and enhancing the accuracy of color restoration. Due to the distribution characteristics of the IR component, it is not difficult to think of using the Bicubic interpolation method to upsample and restore the full-size IR component, and then perform the IR component separation process. According to the full-size RGB images in different directions, the uniformity is judged, and the final RGB pixel value of the most accurate target interpolation direction is selected in the direction of the pixel position as the RGB of the current pixel position. At present, due to the different design of the ISP Pipeline, the RGBIR 4*4 color restoration process is also a little different. The conversion of the RGBIR 4*4 Bayer mode to the ordinary Bayer mode can be realized, and then the RAW domain data processing in the ISP pipeline can be entered; the full-size RGB de-mosaic process can also be realized to prepare for the generation of color pictures.

[0324] The present disclosure also provides an image interpolation device, such as Fig.18 As shown, including:

[0325] A pre-processing unit 101 is used to perform pre-processing on the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed;

[0326] An interpolation processing unit 102 is used to perform red, green, and blue (RGB) interpolation on each pixel of the image to be processed in multiple preset directions based on a preset mode distribution to obtain a full-size RGB image;

[0327] An IR interpolation processing unit 103 is used to perform infrared IR interpolation on each pixel of the image to be processed in multiple preset directions based on a preset mode distribution to obtain a full-size IR image;

[0328] A four-channel image acquisition unit 104, configured to obtain a full-size four-channel image based on the full-size RGB image and the full-size IR image;

[0329] An IR separation processing unit 105 is used to perform IR separation on the four-channel image to obtain a full-size IR separation image;

[0330] a direction selection unit 106, configured to obtain a target interpolation direction selected from the interpolation directions of each pixel in the IR separated image;

[0331] The interpolation result obtaining unit 107 is used to use the interpolation result in the target interpolation direction as the interpolation result of the IR separation image.

[0332] In the embodiments of the present disclosure, Fig.18 As shown, the implementation architecture of the image interpolation device may include but is not limited to: a pre-processing unit 101, an interpolation processing unit 102, an IR interpolation processing unit 103, a four-channel image acquisition unit 104, an IR separation processing unit 105, a direction selection unit 106, an interpolation result acquisition unit 107 and an interpolation pixel selection unit 108.

[0333] In the disclosed embodiment, the pre-processing unit 101 is mainly used to obtain RGBIR 4*4 Bayer pattern distribution and perform black level correction.

[0334] The interpolation processing unit 102 can interpolate a full-size RGB image on multiple interpolation planes, for example, the horizontal plane (east-west direction), the vertical plane (north-south direction), the oblique 45° plane (northeast and southwest directions), and the oblique 135° plane (northwest and southeast directions) can interpolate the full-size RGB of the horizontal plane, the full-size RGB of the vertical plane, the full-size RGB of the oblique 45° plane, and the full-size RGB of the oblique 135° plane, respectively.

[0335] IR interpolation processing unit 103: based on the distribution characteristics of IR pixel positions, a full-size IR image is interpolated based on a Bicubic interpolation algorithm.

[0336] The four-channel image acquisition unit 104 obtains a four-channel image based on the interpolation results of the interpolation processing unit 102 and the IR interpolation processing unit 103 , where the four channels include: an R channel, a G channel, a B channel and an IR channel.

[0337] IR separation processing unit 105: performs IR separation on each of the full-size RGB images of the multiple interpolation planes obtained by the above-mentioned interpolation processing unit 102 to obtain an IR separated image, which is a full-size RGB image in which IR components are no longer mixed in pixel point R, pixel point B, and pixel point G.

[0338] The direction selection unit 106 determines the target interpolation direction at each pixel position one by one for the full-size RGB images of the multiple interpolation planes obtained by the IR separation processing unit 104 .

[0339] The interpolation result acquiring unit 107 uses the interpolation result in the target interpolation direction as the interpolation result of the IR separation image.

[0340] Interpolation pixel selection unit 108: Depending on the mode selected by the user, this step may exist when the user needs to convert the RGBIR 4*4 mode into a common Bayer 2*2 mode, that is, a downsampling operation needs to be performed on the full-size RGB image obtained by the direction selection unit 105 (that is, the interpolation result image obtained by the interpolation result acquisition unit 107) according to the Bayer pattern distribution characteristics.

[0341] In the embodiment of the present disclosure, the interpolation processing unit 102 may include but is not limited to the following parts:

[0342] Variance calculation unit 1021: mainly used to calculate the pixel variation degree of each pixel in the RAW data of the image to be processed in different directions, that is, the pixel difference value.

[0343] Direction weight calculation unit 1022: can perform difference normalization processing on two opposite directions according to multiple preset directions on multiple interpolation planes (such as eight directions of four interpolation planes) to obtain the weight of each preset direction.

[0344] Full-size G interpolation unit 1023: mainly used to obtain the G components of all pixel positions in multiple interpolation planes.

[0345] G component color difference fusion processing unit 1024: It is mainly set up to interpolate the full-size R component and B component, and selects the G component with more similar R position and B position in the neighborhood to give it a greater weight, such as the first color difference fusion coefficient and the second color difference fusion coefficient mentioned above, and participates in the full-size interpolation process of the R component and the B component based on the color difference corresponding to the first color difference fusion coefficient and the second color difference fusion coefficient. After the judgment of the G component color difference fusion processing unit 1024 is completed, the interpolation process of the full-size R component and the B component is performed.

[0346] Full-size R interpolation unit 1025: mainly uses color difference to perform interpolation operation of full-size R component after the aforementioned units interpolate multiple G components of interpolation planes. The interpolated R component adopts the first color difference (RG). Similarly, according to multiple full-size interpolation matrices of the interpolated G component, the interpolation results of the R component on four interpolation planes are re-asynchronously interpolated.

[0347] Full-size B interpolation unit 1026: After the aforementioned units interpolate multiple G components on interpolation planes, the full-size B component is interpolated using color difference. The interpolated B component uses the second color difference (BG). Similarly, based on multiple full-size interpolation matrices of the interpolated G component, the interpolation results of the B component on four interpolation planes are re-asynchronously interpolated.

[0348] The present disclosure also provides an electronic device 200, such as Fig.19 As shown, the electronic device 200 may include:

[0349] One or more processors 201;

[0350] A memory 202 having one or more programs stored thereon, and when the one or more programs are executed by the one or more processors 201, the one or more processors 201 implement the image interpolation method;

[0351] One or more input / output I / O interfaces 203 are connected between the processor 201 and the memory 202 and are configured to implement information interaction between the processor 201 and the memory 202 .

[0352] Among them, the processor 201 is a device with data processing capabilities, including but not limited to a central processing unit (CPU) and the like; the memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 203 is connected between the processor 201 and the memory 202, and can realize information interaction between the processor 201 and the memory 202, including but not limited to a data bus (Bus) and the like.

[0353] In some embodiments, the processor 201 , the memory 202 , and the I / O interface 203 are connected to each other via a bus 204 , and further connected to other components of the computing device.

[0354] The present disclosure also provides a computer-readable storage medium 300, such as Fig. 20 As shown, a computer program is stored on the computer-readable storage medium 300, and the computer program implements the image interpolation method when executed by the processor.

[0355] Those skilled in the art will appreciate that all or some of the functional modules / units disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0356] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0357] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0358] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. An image interpolation method, characterized in that: The method comprises: Performing pre-processing on the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed; Based on the preset pattern distribution, red, green, and blue (RGB) interpolation is performed on each pixel of the image to be processed in multiple preset directions to obtain a full-size RGB image; Based on the preset pattern distribution, performing infrared IR interpolation on each pixel of the image to be processed in a plurality of preset directions to obtain a full-size IR image; Obtaining a full-size four-channel image based on the full-size RGB image and the full-size IR image; Performing IR separation on the four-channel image to obtain a full-size IR separation image; Acquire a target interpolation direction selected from the interpolation directions of each pixel in the IR separated image; The interpolation result in the target interpolation direction is used as the interpolation result of the IR separation image.

2. The image interpolation method according to claim 1, characterized in that: The step of performing red, green, and blue (RGB) interpolation on each pixel of the image to be processed in multiple preset directions based on the preset mode distribution to obtain a full-size RGB image includes: Determine a first neighborhood of each non-G pixel of the image to be processed based on the preset pattern distribution, interpolate corresponding non-G pixels from a plurality of preset directions based on the first neighborhood of each non-G pixel, and obtain a full-size G component of each non-G pixel; Determine a second neighborhood of each non-R pixel of the image to be processed based on the preset pattern distribution, and interpolate a plurality of full-size R components of the preset direction for each non-R pixel using a first color difference between each non-R pixel and a corresponding R pixel in the second neighborhood; Determine a third neighborhood of each non-B pixel of the image to be processed based on the preset pattern distribution, and interpolate a plurality of full-size B components in the preset direction for each non-B pixel using a second color difference between each non-B pixel and a corresponding B pixel in the third neighborhood; The image to be processed after interpolation of the G component, the R component and the B component is used as the RGB image.

3. The image interpolation method according to claim 2, characterized in that: The interpolating the corresponding non-G pixel from the plurality of preset directions based on the first neighborhood of each non-G pixel to obtain a full-size G component of each non-G pixel includes: For each non-G pixel, determine the sum of the G component difference value and the IR component difference value of the non-G pixel in each of the preset directions in the corresponding first neighborhood as the pixel difference value of the non-G pixel in the corresponding preset direction; Determine the weight of each preset direction according to the pixel difference value in each preset direction; wherein the weights of two preset directions opposite to each other in the plurality of preset directions constitute a pair of asynchronous weights; For each non-G pixel, the G component of the non-G pixel in the interpolation plane corresponding to each pair of asynchronous weights is determined respectively according to each pair of asynchronous weights and the average value of the G component in the preset direction corresponding to the asynchronous weights, so as to obtain a full-size G component; the interpolation plane is a plane determined by two opposite preset directions corresponding to the asynchronous weights.

4. The image interpolation method according to claim 3, characterized in that: The determining, based on the preset pattern distribution, a second neighborhood of each non-R pixel of the image to be processed comprises: Determine the interpolation plane currently interpolating each of the non-R pixels; The second neighborhood is determined to match the shape of the interpolation plane.

5. The image interpolation method according to claim 3 or 4, characterized in that: The method of interpolating a plurality of full-size R components in the preset direction for each non-R pixel by using the first color difference between each non-R pixel and the corresponding R pixel in the second neighborhood includes: for each non-R pixel, performing the following interpolation operation on each interpolation plane: Determine a first G component of the non-R pixel on the interpolation plane, determine a second G component of each R position in the second neighborhood on the interpolation plane, and use a difference between each of the second G components and the first G component as a first color difference of each of the R positions on the interpolation plane; Obtaining the distance between each of the R positions and the non-R pixel, and determining a first color difference fusion coefficient corresponding to the R position according to the distance; The R component interpolated by the non-R pixel on the interpolation plane is determined according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the R position corresponding to the first color difference, and the first color difference fusion coefficient.

6. The image interpolation method according to claim 5, characterized in that: The determining, according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction in which the R position corresponding to the first color difference is located, and the first color difference fusion coefficient, the R component interpolated by the non-R pixel on the interpolation plane includes: Multiply each of the first color differences by the first color difference fusion coefficient corresponding to the R position corresponding to the first color difference, to obtain a first fused color difference corresponding to each of the R positions; Add all the first fused color differences in the same preset direction of the interpolation plane, and divide the sum of the first color difference fusion coefficients corresponding to all the R positions in the same preset direction to obtain the RG color difference in the same preset direction; Multiplying each RG color difference in the same preset direction by the weight corresponding to the same preset direction to obtain the weighted RG color difference in the same preset direction; All the weighted RG color differences calculated for the interpolation plane and the full-size G component are added together to obtain the R component interpolated from the non-R pixel on the interpolation plane.

7. The image interpolation method according to claim 5, characterized in that: Before determining the R component interpolated by the non-R pixel on the interpolation plane according to the full-size G component, the first color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the R position corresponding to the first color difference, and the first color difference fusion coefficient, the method further includes: Comparing each of the first color differences with a preset first color difference threshold to obtain a first comparison result; Determine whether to use the first color difference to determine the R component interpolated from the non-R pixel on the interpolation plane according to the first comparison result.

8. The image interpolation method according to claim 7, characterized in that: The determining, according to the first comparison result, whether to use the first color difference to determine the R component interpolated by the non-R pixel on the interpolation plane includes: When any one or more of the first color differences are less than the first color difference threshold, use any one or more of the first color differences to determine an R component interpolated from the non-R pixel on the interpolation plane; When each of the first color differences is greater than or equal to the first color difference threshold, the average value of the first color differences of the first R positions closest to the non-R pixel in the second neighborhood is used as the first color difference of each of the first R positions, and the first color difference of the first R positions is used to determine the R component interpolated by the non-R pixel on the interpolation plane.

9. The image interpolation method according to claim 3, characterized in that: The determining of the third neighborhood of each non-B pixel of the image to be processed based on the preset mode distribution includes: Determine the interpolation plane on which each of the non-B pixels is currently interpolated; The third neighborhood that matches the shape of the interpolation plane is determined.

10. The image interpolation method according to claim 3 or 9, characterized in that: The method of interpolating a plurality of full-size B components in the preset direction for each non-B pixel by using the second color difference between each non-B pixel and the corresponding B pixel in the third neighborhood includes: for each non-B pixel, performing the following interpolation operation on each interpolation plane: Determine a third G component of the non-B pixel on the interpolation plane, determine a fourth G component of each B position in the third neighborhood on the interpolation plane, and use a difference between each fourth G component and the third G component as a second color difference of each B position on the interpolation plane; Obtaining the distance between each of the B positions and the non-B pixel, and determining a second color difference fusion coefficient corresponding to the B position according to the distance; The B component interpolated by the non-B pixel on the interpolation plane is determined according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the B position corresponding to the second color difference, and the second color difference fusion coefficient.

11. The image interpolation method according to claim 10, characterized in that: The determining, according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction in which the B position corresponding to the second color difference is located, and the second color difference fusion coefficient, the B component interpolated by the non-B pixel on the interpolation plane includes: Multiply each of the second color differences by the second color difference fusion coefficient corresponding to the B position corresponding to the second color difference, to obtain a second fused color difference corresponding to each of the B positions; Add all the second fused color differences in the same preset direction of the interpolation plane, and divide the sum of the second color difference fusion coefficients corresponding to all the B positions in the same preset direction to obtain the BG color difference in the same preset direction; Multiplying each BG color difference in the same preset direction by the weight corresponding to the same preset direction to obtain the weighted BG color difference in the same preset direction; All the weighted BG color differences calculated for the interpolation plane and the full-size G component are added together to obtain a B component interpolated from the non-B pixel on the interpolation plane.

12. The image interpolation method according to claim 10, characterized in that: Before determining the B component interpolated from the non-B pixel on the interpolation plane according to the full-size G component, the second color difference in two preset directions corresponding to the interpolation plane, the weight of the preset direction of the B position corresponding to the second color difference, and the second color difference fusion coefficient, the method further includes: Comparing each of the second color differences with a preset second color difference threshold to obtain a second comparison result; Determine whether to use the second color difference to determine a B component interpolated from the non-B pixel on the interpolation plane according to the second comparison result.

13. The image interpolation method according to claim 12, characterized in that: The determining, according to the second comparison result, whether to use the second color difference to determine the B component interpolated by the non-B pixel on the interpolation plane includes: When any one or more of the second color differences are less than the second color difference threshold, use any one or more of the second color differences to determine a B component interpolated from the non-B pixel on the interpolation plane; When each of the second color differences is greater than or equal to the second color difference threshold, the average value of the second color differences of the first B positions closest to the non-B pixel in the third neighborhood is used as the second color difference of each of the first B positions, and the second color difference of the first B positions is used to determine the B component interpolated by the non-B pixel on the interpolation plane.

14. An image interpolation device, characterized in that: include: A pre-processing unit, used for pre-processing the image to be processed to obtain a preset pattern distribution of pixels of the image to be processed; An interpolation processing unit, configured to perform red, green, and blue (RGB) interpolation on each pixel of the image to be processed in a plurality of preset directions based on the preset mode distribution to obtain a full-size RGB image; An IR interpolation processing unit, configured to perform infrared IR interpolation on each pixel of the image to be processed in a plurality of preset directions based on the preset pattern distribution to obtain a full-size IR image; A four-channel image acquisition unit, configured to obtain a full-size four-channel image based on the full-size RGB image and the full-size IR image; An IR separation processing unit, used for performing IR separation on the four-channel image to obtain a full-size IR separation image; a direction selection unit, configured to obtain a target interpolation direction selected from the interpolation directions of each pixel in the IR separated image; The interpolation result acquisition unit is used to use the interpolation result in the target interpolation direction as the interpolation result of the IR separation image.

15. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image interpolation method according to any one of claims 1 to 13; One or more input / output I / O interfaces are connected between the processor and the memory and are configured to implement information interaction between the processor and the memory.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image interpolation method according to any one of claims 1 to 13.

Images