Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium. The method comprises the following steps: determining a first deviation degree between a first chromatic aberration second order derivative and a first chromatic aberration mean value in a first window, and determining a second deviation degree between a second chromatic aberration second order derivative and a second chromatic aberration mean value in the first window; the first window is a window taking an R pixel or a B pixel as a center in the Bayer image to be processed, the first chromatic aberration second order derivative and the first chromatic aberration are determined along a first diagonal direction, and the second chromatic aberration second order derivative and the second chromatic aberration mean value are determined along a second diagonal direction; under the condition that the first deviation degree and / or the second deviation degree are / is not smaller than a preset threshold value, performing weighted summation on the first color difference mean value and the second color difference mean value to obtain first color difference estimation; and carrying out B pixel interpolation or R pixel interpolation on the center pixel position of the first window by utilizing the first chromatic aberration estimation. According to the scheme provided by the invention, the interpolation result accuracy can be improved, and the image quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, electronic device, chip and storage medium. Background Art

[0002] In the related art, data consisting of one R pixel, one B pixel and two G pixels arranged in a fixed pattern is called Bayer data, that is, image data in Bayer format, and the process of converting image data in Bayer format into image data in RGB format is called demosaicing. Summary of the invention

[0003] The present disclosure provides an image processing method, device, electronic device, chip and storage medium to solve the problems in the related art, improve the accuracy of interpolation results, and enhance image quality.

[0004] A first aspect of the present disclosure provides an image processing method, the method comprising:

[0005] Determine a first deviation between a first color difference second-order derivative and a first color difference mean in a first window, and determine a second deviation between a second color difference second-order derivative and a second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in a Bayer image to be processed, the first color difference second-order derivative and the first color difference mean are determined along a first diagonal direction, and the second color difference second-order derivative and the second color difference mean are determined along a second diagonal direction;

[0006] Determining whether the first deviation and the second deviation are less than a preset threshold;

[0007] When the first deviation degree and / or the second deviation degree is not less than a preset threshold, performing weighted summation on the first color difference mean value and the second color difference mean value to obtain a first color difference estimate;

[0008] The first color difference estimation is used to perform B pixel interpolation on the R pixel position at the center of the first window, or the R pixel interpolation is performed on the B pixel position at the center of the first window.

[0009] In some embodiments, determining a first deviation between the second-order derivative of the first color difference in the first window and the first color difference mean, and determining a second deviation between the second-order derivative of the second color difference in the first window and the second color difference mean, comprises:

[0010] Determine a first ratio of the first color difference second derivative to the first color difference mean, and use the absolute value of the first ratio as the first deviation degree;

[0011] A second ratio of the second color difference second derivative to the second color difference mean is determined, and an absolute value of the second ratio is used as the second deviation degree.

[0012] In some embodiments, using the first color difference estimation to perform B pixel interpolation on the R pixel position at the center of the first window, or performing R pixel interpolation on the B pixel position at the center of the first window, includes:

[0013] Subtracting the G pixel value at the R pixel position at the center of the first window from the first color difference estimate to obtain a B pixel interpolation result at the R pixel position at the center of the first window;

[0014] or,

[0015] The G pixel value at the B pixel position at the center of the first window is subtracted from the first color difference estimate to obtain an R pixel interpolation result at the B pixel position at the center of the first window.

[0016] In some embodiments, the method further comprises:

[0017] When both the first deviation and the second deviation are less than a preset threshold, calculating the difference between the first color difference mean and the first color difference second-order derivative to obtain a first difference; calculating the difference between the second color difference mean and the second color difference second-order derivative to obtain a second difference;

[0018] The first color difference estimation is obtained by performing a weighted summation on the first difference and the second difference.

[0019] In some embodiments, the method further comprises:

[0020] Calculating a first first-order derivative and a first second-order derivative along a first diagonal direction in the first window, and calculating a second first-order derivative and a second second-order derivative along a second diagonal direction in the first window;

[0021] Based on the first first-order derivative and the first second-order derivative, calculate first edge indication information along a first diagonal direction in the first window, and based on the second first-order derivative and the second second-order derivative, calculate second edge indication information along a second diagonal direction in the first window;

[0022] Based on the difference between the first edge indication information and the second edge indication information, a first weight of the first color difference mean and a second weight of the second color difference mean are determined; wherein the sum of the first weight and the second weight is 1.

[0023] In some embodiments, the method further comprises:

[0024] Determine a third color difference mean value along the R pixel arrangement direction in the second window, and use the third color difference mean value as a second color difference estimate;

[0025] Determine a fourth color difference mean value along the B pixel arrangement direction in the second window, and use the fourth color difference mean value as a third color difference estimate;

[0026] The second color difference estimation is used to perform R pixel interpolation on the G pixel position at the center of the second window, and the third color difference estimation is used to perform B pixel interpolation on the G pixel position at the center of the second window; wherein,

[0027] The second window is a window centered on the G pixel G0 in the Bayer image, the R pixel arrangement direction is horizontal or vertical, the B pixel arrangement direction is horizontal or vertical, and the R pixel arrangement direction is different from the B pixel arrangement direction.

[0028] In some embodiments, performing R pixel interpolation on the G pixel position at the center of the second window using the second color difference estimation, and performing B pixel interpolation on the G pixel position at the center of the second window using the third color difference estimation, comprises:

[0029] Subtracting the G pixel value at the G pixel position at the center of the second window from the second color difference estimate to obtain an R pixel interpolation result at the G pixel position at the center of the second window;

[0030] The G pixel value at the G pixel position at the center of the second window is subtracted from the third color difference estimate to obtain a B pixel interpolation result at the G pixel position at the center of the second window.

[0031] A second aspect of the present disclosure provides an image processing device, the device comprising:

[0032] A determination unit, used to determine a first deviation between a first color difference second-order derivative and a first color difference mean in a first window, and to determine a second deviation between a second color difference second-order derivative and a second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in a Bayer image to be processed, the first color difference second-order derivative and the first color difference mean are determined along a first diagonal direction, and the second color difference second-order derivative and the second color difference mean are determined along a second diagonal direction;

[0033] A judging unit, configured to judge whether the first deviation and the second deviation are less than a preset threshold;

[0034] a calculation unit, configured to perform weighted summation of the first color difference mean value and the second color difference mean value to obtain a first color difference estimate when the first deviation degree and / or the second deviation degree is not less than a preset threshold;

[0035] An interpolation unit is used to perform B pixel interpolation on the R pixel position at the center of the first window by using the first color difference estimation, or to perform R pixel interpolation on the B pixel position at the center of the first window.

[0036] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0037] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0038] A fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which executes the method described in the first aspect embodiment of the present disclosure when a processor is used to execute the computer program.

[0039] The sixth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0040] In summary, the image processing method, device, electronic device, chip and storage medium proposed in the present disclosure determine a first deviation between the second derivative of the first color difference and the first color difference mean in the first window, and determine a second deviation between the second derivative of the second color difference and the second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in the Bayer image to be processed, the first second derivative of the color difference and the first color difference mean are determined along a first diagonal direction, and the second second derivative of the color difference and the second color difference mean are determined along a second diagonal direction; determine whether the first deviation and the second deviation are less than a preset threshold; when the first deviation and / or the second deviation are not less than a preset threshold, perform weighted summation on the first color difference mean and the second color difference mean to obtain a first color difference estimate; use the first color difference estimate to perform B pixel interpolation on the R pixel position at the center of the first window, or perform R pixel interpolation on the B pixel position at the center of the first window. The solution provided by the present disclosure, in the process of using the pixel value of the G channel to guide the interpolation of the R channel and the B channel, screens out the pixel positions where the color difference of the G channel and the R / B channel is unbalanced by judging the deviation between the second-order derivative of the color difference in the diagonal direction and the color difference mean, and estimates the color difference of the unbalanced color difference position according to the color difference mean, so as to avoid the problem of large deviation of the estimation result caused by the second-order derivative of the color difference dominating the color difference estimation result, improve the accuracy of the color difference estimation result, thereby improving the interpolation accuracy, and further improving the image quality.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0043] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0044] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0045] Figure 2a Schematic diagram of the first diagonal line and the second diagonal line in the embodiment of the present disclosure;

[0046] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0047] Figure 4 is a schematic diagram of a Bayer image in an embodiment of the present disclosure;

[0048] Figure 4a Schematic diagram of the R / B pixel arrangement structure with the G channel as the center of the window in the embodiment of the present disclosure;

[0049] Figure 4b A schematic diagram of the effect of performing RB interpolation on a Bayer image using an LED algorithm in the related art;

[0050] Figure 4c A schematic diagram of the effect of performing RB interpolation on a Bayer image using the method in the application embodiment of the present disclosure;

[0051] Figure 4d A schematic diagram of the effect of performing RB interpolation on another Bayer image using the LED algorithm in the related art;

[0052] Figure 4e A schematic diagram of the effect of performing RB interpolation on another Bayer image using the method in the application embodiment of the present disclosure;

[0053] Figure 5 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0054] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0055] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0057] In the related art, the image sensor may not directly input the complete RGB three-channel data, but collect data from different channels through the color filter array (CFA), and each pixel only outputs one channel component in RGB. Among them, the sampling rate of the R channel and the B channel is 1 / 4, and the sampling rate of the G channel is 1 / 2. This data composed of one R, one B and two G arranged in a fixed pattern is called bayer data. Demosaic processing can convert image data in bayer format into image data in RGB format. It is a basic and key link in the image signal processing pipeline (ISP pipeline). The demosaic processing result will directly determine the final image quality. However, the demosaic algorithm in the related art is based on the consistency of the directions of the three channels of R, G, and B. However, in actual applications, the directions of these three channels are not consistent. When interpolation processing is guided by the direction of the G channel, artifacts are easily generated at the edges or in areas with rich textures. For example, for the highlight red area, the background is black and the foreground is red, and the transition between the foreground and the background is large, which results in the R channel having a large transition, while the G channel is a flat area with a relatively small transition. At this time, if the R / B pixel interpolation is still guided by the G channel direction, the interpolation result will have a large deviation.

[0058] In view of the above problems, in the related art, a low-cost edge detection for high-quality demosaicking (LED) algorithm can be used for demosaic processing. The basic idea of ​​the LED algorithm is: first restore the G channel, and then restore the RB channel based on the G channel; wherein, the RB channel interpolation is first performed by interpolating the color difference, and then the color difference obtained by interpolation is applied to the G pixel at the corresponding position, and finally the interpolation result of the RB channel is obtained; when interpolating the missing B / R pixels at the R / B position, the color difference of the B / R pixels in the diagonal and anti-diagonal directions is weighted. Specifically, the difference between the mean color difference of the diagonal and the second-order derivative of the color difference, and the difference between the mean color difference in the anti-diagonal direction and the second-order derivative of the color difference are first calculated, and then the two differences are weighted and summed to obtain the estimated color difference; when interpolating the missing R / B pixels at the G position, the color difference of the R / B pixels in the horizontal and vertical directions is weighted to obtain the estimated color difference.

[0059] However, in places where the changes in the G channel and the R / B channel are seriously inconsistent, serious color imbalance will occur, and the second-order derivative of the color difference may far exceed the color difference mean. At this time, if the color difference is still estimated by weighting the difference between the color difference mean and the second-order derivative of the color difference, the estimation result will be dominated by the second-order derivative of the color difference, resulting in a large deviation in the interpolation result, causing artifacts and other image quality problems in the processed image.

[0060] In order to solve the problems existing in the related art, the present disclosure proposes an image processing method, device, electronic device, chip and storage medium. In the process of using the pixel value of the G channel to guide the interpolation of the R channel and the B channel, the deviation between the second-order derivative of the color difference in the diagonal direction and the mean color difference is judged to screen out the pixel positions where the color difference of the G channel and the R / B channel is unbalanced, and the color difference of the unbalanced color difference position is estimated according to the mean color difference to avoid the problem of large deviation of the estimation result caused by the second-order derivative of the color difference dominating the color difference estimation result, improve the accuracy of the color difference estimation result, thereby improving the interpolation accuracy, and further improving the image quality.

[0061] Figure 1 FIG. 1 is an application scenario diagram of an image processing method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 104, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires the image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image is sent to the target application. Among them, the electronic device 104 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0062] In some optional embodiments, the electronic device may also be a vehicle-mounted device or an Internet of Vehicles device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.

[0063] A camera can be installed on an electronic device, and images can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of an electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on an electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.

[0064] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in the present disclosure.

[0065] Several applications can be installed in electronic devices. Applications refer to software written for a certain application purpose in electronic devices. Electronic devices can provide services for users through applications. When an application needs to capture an image, it will initiate an image capture instruction. The electronic device will call the camera module according to the image capture instruction to capture the image. Image capture instructions refer to instructions used to trigger image capture operations.

[0066] The electronic device is also provided with a processor, and the image processing module in the processor can perform correction processing on the image captured by the camera module, for example, execute the image processing method provided in the present disclosure.

[0067] The embodiments of the present disclosure can be applied to scenarios that require image processing, for example, image processing in single-frame shooting mode, image processing in multi-frame shooting mode, image processing in preview mode, image processing in video recording mode, image processing in night scene shooting or video mode, and the like. For another example, the embodiments can also be applied to scenarios where the image signal processing (ISP) module is a key module that affects the processor and high-quality edge sharpening control is required during the image processing process of the image pipeline.

[0068] Figure 2 An image processing method is provided in an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0069] Step 201: Determine a first deviation between the second-order derivative of the first color difference and the first color difference mean in the first window, and determine a second deviation between the second-order derivative of the second color difference and the second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in the Bayer image to be processed, the first second-order derivative of the color difference and the first color difference mean are determined along a first diagonal direction, and the second second-order derivative of the color difference and the second color difference mean are determined along a second diagonal direction.

[0070] In some embodiments, Figure 2a As shown by the solid line in the figure, the first diagonal line may be a diagonal line from the upper left corner to the lower right corner of the first window, such as Figure 2a As shown by the middle dotted line, the second diagonal line may be a diagonal line from the upper right corner to the lower left corner of the first window. The first diagonal line may also be called a positive diagonal line, and the second diagonal line may also be called an anti-diagonal line. The embodiment of the present disclosure does not limit this, as long as its function can be achieved.

[0071] In some embodiments, the first deviation degree may be the absolute value of the ratio of the first second-order derivative of the color difference to the first color difference mean, and the second deviation degree may be the absolute value of the ratio of the second second-order derivative of the color difference to the second color difference mean; the first deviation degree and the second deviation degree are used to reflect the degree of difference between the corresponding second-order derivative of the color difference and the corresponding color difference mean, and pixel positions with uneven color difference can be screened out according to the first deviation degree and the second deviation degree.

[0072] Step 202: Determine whether the first deviation and the second deviation are less than a preset threshold.

[0073] In some embodiments, the preset threshold may be an adjustable parameter, and the specific value may be set according to the actual application scenario; in the embodiment of the present disclosure, the preset threshold is preferably set to 1 / 16.

[0074] In some embodiments, the preset thresholds of the first deviation and the second deviation may be the same.

[0075] Step 203: When the first deviation degree and / or the second deviation degree is not less than a preset threshold, weighted sum of the first color difference mean value and the second color difference mean value is performed to obtain a first color difference estimate.

[0076] In some embodiments, when the first deviation degree is greater than or equal to a preset threshold and the second deviation degree is less than the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate; when the first deviation degree is less than the preset threshold and the second deviation degree is greater than or equal to the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate; when the first deviation degree is greater than or equal to the preset threshold and the second deviation degree is greater than or equal to the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate.

[0077] It should be noted that in the related art, the difference between the first color difference mean and the first color difference second-order derivative and the difference between the second color difference mean and the second color difference second-order derivative are weighted and summed to obtain a color difference estimate for B pixel interpolation of the R pixel position or for R pixel interpolation of the B pixel position. If either the first deviation degree or the second deviation degree is greater than or equal to a preset threshold, it means that the color difference second-order derivative far exceeds the corresponding color difference mean. At this time, if the color difference estimate is still calculated based on the difference between the color difference mean and the color difference second-order derivative, the calculation result will be dominated by the second-order derivative, causing discontinuity in the interpolation result of the surrounding pixels, thereby causing artifacts and other image quality problems in the processed image. The disclosed embodiment weights the color difference mean to calculate the color difference estimate, thereby reducing the deviation range of the color difference estimate, thereby improving the accuracy of the interpolation result when the color difference estimate is used for R / B pixel interpolation, thereby improving the image quality.

[0078] Step 204: perform B pixel interpolation on the R pixel position at the center of the first window using the first color difference estimation, or perform R pixel interpolation on the B pixel position at the center of the first window.

[0079] In some embodiments, when the center position of the first window is an R pixel, the first color difference estimate is a color difference estimate of the interpolated B pixel that is missing from the R pixel position at the center of the first window. At this time, the B pixel interpolation result of the R pixel position at the center of the first window can be obtained by subtracting the G pixel value of the R pixel position at the center of the first window from the first color difference estimate; when the center position of the first window is a B pixel, the first color difference estimate is a color difference estimate of the interpolated R pixel that is missing from the B pixel position at the center of the first window. At this time, the R pixel interpolation result of the B pixel position at the center of the first window can be obtained by subtracting the G pixel value of the B pixel position at the center of the first window from the first color difference estimate.

[0080] In some embodiments, the R pixel position may also be referred to as the R channel position, and being centered on the R pixel may also be referred to as being centered on the R channel; the G pixel position may also be referred to as the G channel position, and being centered on the G pixel may also be referred to as being centered on the G channel; the B pixel position may also be referred to as the B channel position, and being centered on the B pixel may also be referred to as being centered on the B channel.

[0081] It should be noted that the execution timing of step 201 to step 204 is after the G channel interpolation is completed, that is, all G pixels in the Bayer image window are known.

[0082] In summary, in the process of using the pixel value of the G channel to guide the interpolation of the R channel and the B channel, the embodiment of the present disclosure screens out the pixel positions where the color difference of the G channel and the R / B channel is unbalanced by judging the deviation between the second-order derivative of the color difference in the diagonal direction and the mean color difference, and estimates the color difference of the unbalanced color difference position according to the mean color difference, so as to avoid the problem of large deviation of the estimation result caused by the second-order derivative of the color difference dominating the color difference estimation result, improve the accuracy of the color difference estimation result, thereby improving the interpolation accuracy, and further improving the image quality.

[0083] based on Figure 2 The embodiment shown, Figure 3 A flowchart of an image processing method proposed in the present disclosure is further shown. Figure 3 As shown, the method includes:

[0084] Step 301: Determine a first deviation between the second-order derivative of the first color difference and the first color difference mean in the first window, and determine a second deviation between the second-order derivative of the second color difference and the second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in the Bayer image to be processed, the first second-order derivative of the color difference and the first color difference mean are determined along a first diagonal direction, and the second second-order derivative of the color difference and the second color difference mean are determined along a second diagonal direction.

[0085] In some embodiments, when determining a first deviation between the second derivative of the first color difference and the first color difference mean in the first window, a first ratio of the second derivative of the first color difference to the first color difference mean can be determined, and the absolute value of the first ratio is used as the first deviation; when determining a second deviation between the second derivative of the second color difference and the second color difference mean in the first window, a second ratio of the second derivative of the second color difference to the second color difference mean can be determined, and the absolute value of the second ratio is used as the second deviation. The first deviation and the second deviation can reflect the degree of difference between the corresponding second derivative of the color difference and the corresponding color difference mean, so that the pixel position with uneven color difference can be screened out according to the first deviation and the second deviation.

[0086] For example, in Figure 4 In the Bayer image shown, M represents an R or B pixel on the bayer plane, wherein when the B pixel is the center of the window, that is, when M_0, M_2, M_4, M_6, M_8 represent B pixels and M_1, M_3, M_5, M_7 represent R pixels, M represents B pixels, and when the R pixel is the center of the window, that is, when M_0, M_2, M_4, M_6, M_8 represent R pixels and M_1, M_3, M_5, M_7 represent B pixels, M represents R pixels; the following is an example of taking the B pixel as the center of the window:

[0087] Mean color difference in the diagonal direction The mean color difference between the diagonal direction and the anti-diagonal direction It can be determined by the following formula:

[0088]

[0089]

[0090] The second derivative of chromatic aberration in the diagonal direction and the second derivative of chromatic aberration in the anti-diagonal direction It can be determined by the following formula:

[0091]

[0092]

[0093] Among them, Δ i is the color difference of the R pixel or B pixel in the direction of Mi (diagonal direction or anti-diagonal direction), i = 0, 1..., 8; here, Δ i It can be expressed by the following formula:

[0094] Δ i =G i -M i .

[0095] The first difference is The second difference is

[0096] Step 302: Determine whether the first deviation and the second deviation are less than a preset threshold.

[0097] In some embodiments, the preset threshold may be an adjustable parameter, and the specific value may be set according to the actual application scenario; in the embodiment of the present disclosure, the preset thresholds of the first difference and the second difference are preferably set to 1 / 16.

[0098] Step 303: When the first deviation degree and / or the second deviation degree is not less than a preset threshold, weighted summation is performed on the first color difference mean value and the second color difference mean value to obtain a first color difference estimate.

[0099] In some embodiments, when the first difference is not less than a preset threshold, or the second difference is not less than a preset threshold, or both the first difference and the second difference are not less than the preset threshold, it indicates that the second-order derivative of the color difference far exceeds the corresponding color difference mean, and the surrounding color difference varies greatly. At this time, by weighting the color difference mean to calculate the color difference estimation, the deviation range of the color difference estimation can be reduced, thereby improving the accuracy of the interpolation result when using the color difference estimation to perform R / B pixel interpolation, thereby improving the image quality.

[0100] In some embodiments, when the first deviation degree is greater than or equal to a preset threshold and the second deviation degree is less than the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate; when the first deviation degree is less than the preset threshold and the second deviation degree is greater than or equal to the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate; when the first deviation degree is greater than or equal to the preset threshold and the second deviation degree is greater than or equal to the preset threshold, the first color difference mean and the second color difference mean may be weightedly summed to obtain a first color difference estimate.

[0101] In actual application, before performing weighted summation on the first color difference mean value and the second color difference mean value, the weights of the first color difference mean value and the second color difference mean value may be determined first.

[0102] Based on this, in some embodiments, the method may further include:

[0103] Calculating a first first-order derivative and a first second-order derivative along a first diagonal direction in the first window, and calculating a second first-order derivative and a second second-order derivative along a second diagonal direction in the first window;

[0104] Based on the first first-order derivative and the first second-order derivative, calculate first edge indication information along a first diagonal direction in the first window, and based on the second first-order derivative and the second second-order derivative, calculate second edge indication information along a second diagonal direction in the first window;

[0105] Based on the difference between the first edge indication information and the second edge indication information, a first weight of the first color difference mean and a second weight of the second color difference mean are determined; wherein the sum of the first weight and the second weight is 1.

[0106] For example, for Figure 4 The Bayer image shown:

[0107] The first and second derivatives in the diagonal direction are calculated using the following formula:

[0108]

[0109]

[0110] The first and second derivatives in the anti-diagonal direction are calculated by the following formula:

[0111]

[0112]

[0113] Then, using the first-order and second-order derivatives in different diagonal directions, the edge indication information v in the diagonal and anti-diagonal directions is calculated by the following formula: d and v a :

[0114]

[0115]

[0116] Finally, the weights w for the diagonal and anti-diagonal directions are calculated d and w a :

[0117]

[0118] ω a =1-ω d .

[0119] In actual application, when the first deviation and the second deviation are both less than the preset threshold, it means that the second-order derivative of the color difference is not much different from the corresponding color difference mean, and the surrounding color difference changes little. At this time, the original LED algorithm method can be used to calculate the first color difference estimate.

[0120] Based on this, in some embodiments, the method may further include:

[0121] When both the first deviation and the second deviation are less than a preset threshold, calculating the difference between the first color difference mean and the first color difference second-order derivative to obtain a first difference; calculating the difference between the second color difference mean and the second color difference second-order derivative to obtain a second difference;

[0122] The first color difference estimation is obtained by performing a weighted summation on the first difference and the second difference.

[0123] It should be noted that the above-mentioned first weight can be used as the weight of the first difference, and the above-mentioned second weight can be used as the weight of the second interpolation. The calculation process of the first weight and the second weight has been described in detail above and will not be repeated here.

[0124] For example, for Figure 4 The Bayer image shown:

[0125] Determine whether the color difference mean and the corresponding color difference second-order derivative meet the conditions If it satisfies, it means that the second-order derivative of the color difference is not much different from the corresponding color difference mean. At this time, the following formula can be used to calculate the first color difference estimate

[0126]

[0127] If it is not satisfied, it means that the second-order derivative of the color difference is significantly different from the corresponding color difference mean. In this case, the following formula can be used to calculate the first color difference estimate:

[0128]

[0129] Step 304: perform B pixel interpolation on the R pixel position at the center of the first window using the first color difference estimation, or perform R pixel interpolation on the B pixel position at the center of the first window.

[0130] In some embodiments, the G pixel value at the R pixel position at the center of the first window may be subtracted from the first color difference estimate to obtain a B pixel interpolation result at the R pixel position at the center of the first window, and the G pixel value at the B pixel position at the center of the first window may be subtracted from the first color difference estimate to obtain a R pixel interpolation result at the B pixel position at the center of the first window.

[0131] For example, the color difference estimation is performed using right Figure 4 The center pixel position of the window shown is interpolated, that is, the color difference estimate is By applying the G pixel value G0 at the center pixel position of the window, the estimated R / B interpolation result E of the pixel position can be obtained, which can be specifically expressed by the following formula:

[0132]

[0133] Step 305: Determine a third color difference mean along the R pixel arrangement direction in the second window, and use the third color difference mean as a second color difference estimate.

[0134] Step 306: Determine a fourth color difference mean along the B pixel arrangement direction in the second window, and use the fourth color difference mean as a third color difference estimate.

[0135] It should be noted that step 305 may be performed before step 306, or after steps 3-6, or simultaneously with step 306, which is not limited in the present embodiment. Steps 305 to 307 are the process of interpolating the actual R / B pixels at the G position, at which point the center of the window is the G channel.

[0136] In practical applications, when the center of the window is the G channel, the arrangement of R / B pixels can include two types: the first type is that the B pixels are arranged along the horizontal direction of the window, and the R pixels are arranged along the vertical direction of the window, that is, Figure 4a In the window shown, M2 and M3 are B pixels, and M1 and M4 are R pixels. In the second case, R pixels are arranged along the horizontal direction of the window, and B pixels are arranged along the vertical direction of the window, that is, Figure 4a In the window shown, M2 and M3 are R pixels, and M1 and M4 are B pixels.

[0137] In some embodiments, the second color difference estimation may also be referred to as R channel color difference estimation, and the third color difference estimation may also be referred to as B channel estimation.

[0138] Exemplarily, when the R / B pixels are arranged in the first arrangement, the R channel color difference estimation and the B channel color difference estimation at the G0 position are respectively:

[0139]

[0140]

[0141] Among them, Δ i is the color difference of the R pixel or B pixel in the direction (horizontal or vertical direction) where Mi is located, where i = 1, 2, 3, 4; here, Δ i It can be expressed by the following formula:

[0142] Δ i =G i -M i .

[0143] Step 307: performing R pixel interpolation on the G pixel position at the center of the second window using the second color difference estimation, and performing B pixel interpolation on the G pixel position at the center of the second window using the third color difference estimation; wherein,

[0144] The second window is a window centered on the G pixel G0 in the Bayer image, the R pixel arrangement direction is horizontal or vertical, the B pixel arrangement direction is horizontal or vertical, and the R pixel arrangement direction is different from the B pixel arrangement direction.

[0145] In some embodiments, the G pixel value at the G pixel position at the center of the second window may be subtracted from the second color difference estimate to obtain an R pixel interpolation result at the G pixel position at the center of the second window, and the G pixel value at the G pixel position at the center of the second window may be subtracted from the third color difference estimate to obtain a B pixel interpolation result at the G pixel position at the center of the second window.

[0146] Exemplarily, using the second color difference estimation or the third color difference estimation Figure 4a The center pixel position of the window shown is interpolated, that is, the second color difference estimation is applied to the G pixel value G0 of the center pixel position of the window, and the estimated R pixel interpolation result or B pixel interpolation result of the pixel position can be obtained; specifically, it can be expressed by the following formula:

[0147]

[0148]

[0149] In summary, in the process of using the pixel value of the G channel to guide the interpolation of the R channel and the B channel, the embodiment of the present disclosure screens out the pixel positions where the color difference of the G channel and the R / B channel is unbalanced by judging the deviation between the second-order derivative of the color difference in the diagonal direction and the mean color difference, and estimates the color difference of the unbalanced color difference position according to the mean color difference, so as to avoid the problem of large deviation of the estimation result caused by the second-order derivative of the color difference dominating the color difference estimation result, improve the accuracy of the color difference estimation result, thereby improving the interpolation accuracy, and further improving the image quality.

[0150] Furthermore, since the color difference mean is directly used to calculate the color difference estimation during the R / B interpolation process of the G position, the window size can be 5*5. Compared with the related technology that requires the weighted summation of the second-order derivative of the color difference in the 11*11 window to calculate the R / B color difference estimation of the G position, the window size is reduced, thereby reducing the amount of calculation and reducing the consumption of computing resources.

[0151] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.

[0152] The present application embodiment provides a RB channel interpolation method based on the LED algorithm, the method comprising the following steps:

[0153] Step 1: Calculate the weights of the diagonal and anti-diagonal directions on the Bayer plane.

[0154] Here, the center of the window is the B pixel, and all the G pixels within the window are known.

[0155] Step 1 may specifically include:

[0156] Step 1.1: Calculate the first and second derivatives of the diagonal and anti-diagonal lines.

[0157] Here, as shown in 4, M, M_0, M_2, M_4, M_6, and M_8 represent B pixels, and M_1, M_3, M_5, and M_7 represent R pixels.

[0158] Compute the first and second derivatives in the diagonal direction:

[0159]

[0160]

[0161] Compute the first and second derivatives in the anti-diagonal direction:

[0162]

[0163]

[0164] Step 1.2: Calculate the edge indication information v in the diagonal and anti-diagonal directions d and v a :

[0165]

[0166]

[0167] Step 1.3: Calculate the weights w for the diagonal and anti-diagonal directions d and w a :

[0168]

[0169] ω a =1-ω d .

[0170] Step 2: Interpolate the missing B / R pixels at the R / B positions.

[0171] Step 2 may specifically include:

[0172] Step 2.1: Calculate the color difference of R / B pixels on the diagonal and anti-diagonal lines △ i , where i=0, 1, 2, 3, 4, 5, 6, 7, 8.

[0173] Color difference i The specific calculation formula is:

[0174] Δ i =G i -M i .

[0175] Step 2.2: Calculate the mean color difference and the second derivative of the color difference in the diagonal and anti-diagonal directions.

[0176] Among them, the color difference mean calculation formula is:

[0177]

[0178]

[0179] The calculation formula of the second derivative of color difference is:

[0180]

[0181]

[0182] Step 2.3: Use different color difference calculation methods according to the changes in color difference.

[0183] If the conditions are met Indicates that the color difference change transition is small, and the color difference can be estimated by the LED method in the relevant technology. At this time, the color difference is estimated according to the following formula:

[0184]

[0185] Wherein, α is adjustable, and in the application embodiment of the present disclosure, α is set to 16;

[0186] If not satisfied Indicates that the color difference changes greatly. At this time, the color difference is estimated according to the following formula:

[0187]

[0188] Step 2.4: Apply the estimated result of the color difference to G0 to obtain the estimated R / B interpolation result E; E can be calculated by the following formula:

[0189]

[0190] Step 3: Interpolate the missing R / B pixels at the G position.

[0191] Step 3 may specifically include:

[0192] Step 3.1: Calculate the color difference in horizontal and vertical directions i , where i = 1, 2, 3, 4; △ i The calculation formula can be expressed as:

[0193] Δ i =G i -M i ;

[0194] Step 3.2: Calculate the color difference of R / B pixels along the horizontal and vertical directions respectively.

[0195] When the center of the window is the G channel, there are two ways to arrange the R / B pixels: First, the B pixels are arranged along the horizontal direction of the window, and the R pixels are arranged along the vertical direction of the window, that is, Figure 4a In the window shown, M2 and M3 are B pixels, and M1 and M4 are R pixels. In the second case, R pixels are arranged along the horizontal direction of the window, and B pixels are arranged along the vertical direction of the window, that is, Figure 4a In the window shown, M2 and M3 are R pixels, and M1 and M4 are B pixels. Different arrangement modes will be introduced below.

[0196] When the first arrangement is adopted, the R channel color difference estimation and the B channel color difference estimation at the G0 position are respectively expressed as:

[0197]

[0198]

[0199] When the second arrangement is adopted, the R channel color difference estimation and the B channel color difference estimation at the G0 position are respectively expressed as:

[0200]

[0201]

[0202] Step 3.3: Apply the estimated result of the color difference to G0 to obtain the estimated R / B interpolation result; it can be specifically expressed by the following formula:

[0203]

[0204]

[0205] The application embodiments of the present disclosure have the following beneficial effects:

[0206] 1. When there is a large difference between the second-order derivative of the color difference and the mean color difference, the color difference mean is used for weighting to estimate the B pixels missing at the R channel position or the R pixels missing at the B channel position. This allows the positions with uneven color difference to be screened, and when uneven color difference occurs, the color difference is estimated by weighting the color difference mean to avoid the situation where the color difference estimation result is dominated by the second-order derivative of the color difference and has a large deviation, thereby improving the accuracy of the color difference estimation result and the interpolation result, and eliminating the image quality problem that is easy to occur in the area with severe color difference imbalance in the RB channel interpolation of the LED algorithm in the related technology.

[0207] 2. In the process of interpolating the missing R / B pixels at the G position, the mean color difference along the horizontal or vertical direction is used as the color difference estimation, instead of performing weighted summation of the mean color difference in the horizontal or vertical direction and the difference of the second-order derivative of the color difference in the related technology to obtain the color difference estimation, thereby reducing the window size from 11*11 to 5*5, reducing the amount of calculation and reducing the consumption of computing resources.

[0208] Figure 4b and Figure 4c The schematic diagrams of demosaic output results obtained by performing RB interpolation processing on the same Bayer image using the LED algorithm in the related art and the method of the application embodiment of the present disclosure are respectively, Figure 4d and Figure 4e The following are schematic diagrams of demosaic output results obtained by performing RB interpolation processing on the same Bayer image using the LED algorithm in the related art and the method of the application embodiment of the present disclosure. Figure 4b to Figure 4e As shown in the box, the interpolation method provided by the application embodiment of the present disclosure can effectively reduce the occurrence of pixel discontinuity in the image and improve image quality problems such as artifacts.

[0209] Figure 5 FIG. 5 is a schematic diagram of the structure of an image processing device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the image processing device comprises:

[0210] A determination unit 501 is used to determine a first deviation between a first color difference second-order derivative and a first color difference mean in a first window, and to determine a second deviation between a second color difference second-order derivative and a second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in a Bayer image to be processed, the first color difference second-order derivative and the first color difference mean are determined along a first diagonal direction, and the second color difference second-order derivative and the second color difference mean are determined along a second diagonal direction;

[0211] A judging unit 502 is used to judge whether the first deviation degree and the second deviation degree are less than a preset threshold;

[0212] A calculation unit 503 is used to perform weighted summation on the first color difference mean and the second color difference mean to obtain a first color difference estimate when the first deviation degree and / or the second deviation degree is not less than a preset threshold;

[0213] The interpolation unit 504 is configured to perform B pixel interpolation on the R pixel position at the center of the first window by using the first color difference estimation, or perform R pixel interpolation on the B pixel position at the center of the first window.

[0214] In some embodiments, the determining unit 501 is specifically configured to:

[0215] Determine a first ratio of the first color difference second derivative to the first color difference mean, and use the absolute value of the first ratio as the first deviation degree;

[0216] A second ratio of the second color difference second derivative to the second color difference mean is determined, and an absolute value of the second ratio is used as the second deviation degree.

[0217] In some embodiments, the interpolation unit 504 is specifically configured to:

[0218] Subtracting the G pixel value at the R pixel position at the center of the first window from the first color difference estimate to obtain a B pixel interpolation result at the R pixel position at the center of the first window;

[0219] or,

[0220] The G pixel value at the B pixel position at the center of the first window is subtracted from the first color difference estimate to obtain an R pixel interpolation result at the B pixel position at the center of the first window.

[0221] In some embodiments, the computing unit 503 may also be used to:

[0222] When both the first deviation and the second deviation are less than a preset threshold, calculating the difference between the first color difference mean and the first color difference second-order derivative to obtain a first difference; calculating the difference between the second color difference mean and the second color difference second-order derivative to obtain a second difference;

[0223] The first color difference estimation is obtained by performing a weighted summation on the first difference and the second difference.

[0224] In some embodiments, the computing unit 503 may also be used to:

[0225] Calculating a first first-order derivative and a first second-order derivative along a first diagonal direction in the first window, and calculating a second first-order derivative and a second second-order derivative along a second diagonal direction in the first window;

[0226] Based on the first first-order derivative and the first second-order derivative, calculate first edge indication information along a first diagonal direction in the first window, and based on the second first-order derivative and the second second-order derivative, calculate second edge indication information along a second diagonal direction in the first window;

[0227] Based on the difference between the first edge indication information and the second edge indication information, a first weight of the first color difference mean and a second weight of the second color difference mean are determined; wherein the sum of the first weight and the second weight is 1.

[0228] In some embodiments, the computing unit 503 may also be used to:

[0229] Determine a third color difference mean along the R pixel arrangement direction in the second window, and use the third color difference mean as the second color difference estimate; and determine a fourth color difference mean along the B pixel arrangement direction in the second window, and use the fourth color difference mean as the third color difference estimate;

[0230] The interpolation unit 504 may also be configured to perform R pixel interpolation on the G pixel position at the center of the second window using the second color difference estimation, and perform B pixel interpolation on the G pixel position at the center of the second window using the third color difference estimation; wherein,

[0231] The second window is a window centered on the G pixel G0 in the Bayer image, the R pixel arrangement direction is horizontal or vertical, the B pixel arrangement direction is horizontal or vertical, and the R pixel arrangement direction is different from the B pixel arrangement direction.

[0232] In some embodiments, the interpolation unit 504 may be specifically used to:

[0233] Subtracting the G pixel value at the G pixel position at the center of the second window from the second color difference estimate to obtain an R pixel interpolation result at the G pixel position at the center of the second window;

[0234] The G pixel value at the G pixel position at the center of the second window is subtracted from the third color difference estimate to obtain a B pixel interpolation result at the G pixel position at the center of the second window.

[0235] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0236] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0237] Figure 6is a block diagram of an electronic device 600 for implementing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0238] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0239] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0240] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0241] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0242] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0243] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0244] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0245] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0246] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0247] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0248] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by the processor 620 of the electronic device 600 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0249] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.

[0250] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the image processing method described in the above embodiment of the present disclosure when a processor is used.

[0251] The embodiment of the present disclosure also provides a chip, which can be seen in Figure 7 Schematic diagram of the chip structure shown. Figure 7 The chip shown includes a processor 701 and an interface circuit 702. The number of the processor 701 can be one or more, and the number of the interface circuit 702 can be one or more.

[0252] Optionally, the chip also includes a memory 703, which is used to store necessary computer programs and data; the interface circuit 702 is used to receive signals from the memory 703 and send signals to the processor 701, and the signals include computer instructions stored in the memory 703. When the processor 701 executes the computer instructions, the electronic device executes the image processing method described in the above embodiments of the present disclosure.

[0253] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0254] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0255] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0256] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0257] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0258] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0259] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0260] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a first deviation between a first color difference second-order derivative and a first color difference mean in a first window, and determine a second deviation between a second color difference second-order derivative and a second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in a Bayer image to be processed, the first color difference second-order derivative and the first color difference mean are determined along a first diagonal direction, and the second color difference second-order derivative and the second color difference mean are determined along a second diagonal direction; Determining whether the first deviation and the second deviation are less than a preset threshold; When the first deviation degree and / or the second deviation degree is not less than a preset threshold, performing weighted summation on the first color difference mean value and the second color difference mean value to obtain a first color difference estimate; Using the first color difference estimation, B pixel interpolation is performed on the R pixel position at the center of the first window, or R pixel interpolation is performed on the B pixel position at the center of the first window.

2. The method according to claim 1, characterized in that The step of determining a first deviation between the second-order derivative of the first color difference in the first window and the first color difference mean, and determining a second deviation between the second-order derivative of the second color difference in the first window and the second color difference mean, comprises: Determine a first ratio of the first color difference second derivative to the first color difference mean, and use the absolute value of the first ratio as the first deviation degree; A second ratio of the second color difference second derivative to the second color difference mean is determined, and an absolute value of the second ratio is used as the second deviation degree.

3. The method according to claim 2, characterized in that The using the first color difference estimation to perform B pixel interpolation on the R pixel position at the center of the first window, or performing R pixel interpolation on the B pixel position at the center of the first window, includes: Subtracting the G pixel value at the R pixel position at the center of the first window from the first color difference estimate to obtain a B pixel interpolation result at the R pixel position at the center of the first window; or, The G pixel value at the B pixel position at the center of the first window is subtracted from the first color difference estimate to obtain an R pixel interpolation result at the B pixel position at the center of the first window.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When both the first deviation and the second deviation are smaller than a preset threshold, calculating the difference between the first color difference mean and the first color difference second-order derivative to obtain a first difference, and calculating the difference between the second color difference mean and the second color difference second-order derivative to obtain a second difference; The first color difference estimation is obtained by performing a weighted summation on the first difference and the second difference.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Calculating a first first-order derivative and a first second-order derivative along a first diagonal direction in the first window, and calculating a second first-order derivative and a second second-order derivative along a second diagonal direction in the first window; Based on the first first-order derivative and the first second-order derivative, calculate first edge indication information along a first diagonal direction in the first window, and based on the second first-order derivative and the second second-order derivative, calculate second edge indication information along a second diagonal direction in the first window; Based on the difference between the first edge indication information and the second edge indication information, a first weight of the first color difference mean and a second weight of the second color difference mean are determined; wherein the sum of the first weight and the second weight is 1.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Determine a third color difference mean value along the R pixel arrangement direction in the second window, and use the third color difference mean value as a second color difference estimate; Determine a fourth color difference mean value along the B pixel arrangement direction in the second window, and use the fourth color difference mean value as a third color difference estimate; The second color difference estimation is used to perform R pixel interpolation on the G pixel position at the center of the second window, and the third color difference estimation is used to perform B pixel interpolation on the G pixel position at the center of the second window; wherein, The second window is a window centered on the G pixel G0 in the Bayer image, the R pixel arrangement direction is horizontal or vertical, the B pixel arrangement direction is horizontal or vertical, and the R pixel arrangement direction is different from the B pixel arrangement direction.

7. The method according to claim 6, characterized in that The method of performing R pixel interpolation on the G pixel position at the center of the second window by using the second color difference estimation, and performing B pixel interpolation on the G pixel position at the center of the second window by using the third color difference estimation, comprises: Subtracting the G pixel value at the G pixel position at the center of the second window from the second color difference estimate to obtain an R pixel interpolation result at the G pixel position at the center of the second window; The G pixel value at the G pixel position at the center of the second window is subtracted from the third color difference estimate to obtain a B pixel interpolation result at the G pixel position at the center of the second window.

8. An image processing device, characterized in that: The device comprises: A determination unit, used to determine a first deviation between a first color difference second-order derivative and a first color difference mean in a first window, and to determine a second deviation between a second color difference second-order derivative and a second color difference mean in the first window; wherein the first window is a window centered on an R pixel or a B pixel in a Bayer image to be processed, the first color difference second-order derivative and the first color difference mean are determined along a first diagonal direction, and the second color difference second-order derivative and the second color difference mean are determined along a second diagonal direction; A judging unit, configured to judge whether the first deviation and the second deviation are less than a preset threshold; a calculation unit, configured to perform weighted summation of the first color difference mean value and the second color difference mean value to obtain a first color difference estimate when the first deviation degree and / or the second deviation degree is not less than a preset threshold; An interpolation unit is used to perform B pixel interpolation on the R pixel position at the center of the first window by using the first color difference estimation, or to perform R pixel interpolation on the B pixel position at the center of the first window.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

12. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.

Images