Image processing method and device, terminal, chip, and chip module

Abstract

An image processing method and device, terminal, chip, and chip module, the method comprising: calculating the color difference of each pixel in the neighborhood of the current pixel in a first image to obtain a color difference map, where the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel; determining unit color difference values ​​and weight values ​​in M ​​directions based on the color difference map, where M is an integer greater than 1, the unit color difference value in each direction is used to represent the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction; reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer; fusing the unit color difference values ​​in each direction based on the weight value to obtain a target color difference value; and obtaining the pixel value of the current pixel in the second channel based on the pixel value of the current pixel in the first channel and the target color difference value. The solution of the present application is conducive to improving the processing effect of demosaic.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device, terminal, chip, and chip module. Background Art

[0002] To save costs, a color filter array (CFA) is typically used during image acquisition to capture light of different spectra. The image's color information is then restored or reconstructed based on the CFA results. This method is also known as the CFA method or demosaicing. With the development of technologies such as automotive, security, and smart terminals, the requirements for image quality are becoming increasingly higher. The demosaicing effect is a significant factor affecting image quality, but the existing demosaicing effect still needs to be improved. Summary of the Invention

[0003] The technical problem to be solved by this application is how to improve the effect of demosaicing.

[0004] To solve the above technical problems, an embodiment of the present application provides an image processing method, which includes: calculating the color difference of each pixel point in the neighborhood of the current pixel point in the first image to obtain a color difference map, where the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel; determining the unit color difference values ​​and weight values ​​in M ​​directions according to the color difference map, where M is an integer greater than 1, the unit color difference value in each direction is used to represent the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction; reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer; fusing the unit color difference values ​​in each direction according to the weight value to obtain a target color difference value; and obtaining the pixel value of the current pixel point in the second channel according to the pixel value of the current pixel point in the first channel and the target color difference value.

[0005] Optionally, the first image is a Bayer format image, and calculating the color difference of each pixel point in the neighborhood of the current pixel point in the first image includes: determining the current pixel point, performing interpolation processing on the pixel points in the neighborhood, so that after the interpolation processing, each pixel point in the neighborhood has the pixel value of the first channel and the second channel; and calculating the color difference based on the pixel value of each pixel point in the neighborhood after the interpolation processing.

[0006] Optionally, reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions includes: determining a reference threshold based on the weight values ​​of the M directions; for any one of the M directions, if the weight value of the direction is less than or equal to the reference threshold, reducing the weight value of the direction.

[0007] Optionally, if the weight value of any direction is less than or equal to the reference threshold, the reduced weight value of the direction is 0.

[0008] Optionally, reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions also includes: for any one of the M directions, if the weight value of the direction is greater than the reference threshold, reducing the weight value of the direction according to the reference threshold.

[0009] Optionally, determining the reference threshold according to the weight values ​​of the M directions includes: determining the reference threshold according to the weight values ​​of the M directions.

[0010] Optionally, M is greater than or equal to 4.

[0011] Optionally, the method also includes: determining the color temperature of the first image; determining a correction matrix based on the color temperature of the first image; performing color correction on the second image based on the correction matrix to obtain a target image, wherein the second image is a full-channel image obtained after traversing each pixel point in the first image.

[0012] Optionally, before calculating the color difference of each pixel in the neighborhood of the current pixel in the first image, the method further includes: acquiring an original image; and performing remosaic processing on the original image to obtain the first image.

[0013] In order to solve the above-mentioned technical features, an embodiment of the present application also provides an image processing device, which includes: a first calculation module, which is used to calculate the color difference of each pixel point in the neighborhood of the current pixel point in the first image to obtain a color difference map, where the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel; a second calculation module, which is used to determine the unit color difference values ​​and weight values ​​in M ​​directions based on the color difference map, where M is an integer greater than 1, the unit color difference value in each direction is used to represent the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction; an adjustment module, which is used to reduce the weight values ​​of the N directions with the smallest weight values ​​among the M directions, 1≤N≤M, where N is an integer; a third calculation module, which is used to fuse the unit color difference values ​​in each direction according to the weight value to obtain a target color difference value; and a fourth calculation module, which is used to obtain the pixel value of the current pixel point in the second channel based on the pixel value of the current pixel point in the first channel and the target color difference value.

[0014] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned image processing method are executed.

[0015] An embodiment of the present application further provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned image processing method when running the computer program.

[0016] An embodiment of the present application also provides a chip, which is used in the steps of the above-mentioned image processing method.

[0017] An embodiment of the present application also provides a chip module, which is used for the steps of the above-mentioned image processing method.

[0018] Compared with the prior art, the technical solution of the embodiment of the present application has the following beneficial effects:

[0019] In the scheme of the embodiment of the present application, the color difference of each pixel point in the neighborhood of the current pixel point is calculated to obtain a color difference map, and the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel; then, based on the color difference map, the unit color difference values ​​and weight values ​​of M directions are determined; the weight values ​​of N directions with the smallest weight values ​​among the M directions are reduced, and then fusion processing is performed based on the weight values ​​and the unit color difference values ​​of each direction to obtain a target color difference value; based on the pixel value of the current pixel point in the first channel and the target color difference value, the pixel value of the current pixel point in the second channel is determined to complete the interpolation of the second channel of the current pixel point.

[0020] Compared with the prior art, in the solution of the embodiment of the present application, the relevant information between the first channel and the second channel in each direction is characterized by the unit color difference value in each direction, and the degree of color difference change in each direction is characterized by the weight value. The more intense the color difference change, the more obvious the texture, and the smaller the weight value. Furthermore, by reducing the weight value of a part of the direction to weaken the relevant information of the secondary direction (for example, the direction with a smaller weight value) and strengthen the relevant information of the main direction (for example, the direction with a larger weight value), it is beneficial to improve the accuracy of interpolation, especially to improve the interpolation accuracy of texture-dense areas, thereby improving the quality of the final full-channel image, thereby improving the processing effect of Demosaic.

[0021] Furthermore, in the embodiment of the present application, if the weight value of any direction is less than or equal to the reference threshold, the weight value of the direction after reduction is 0. This solution is conducive to maximally weakening the relevant information of the secondary direction and improving the accuracy of interpolation.

[0022] Furthermore, in the solution of the embodiment of the present application, the reference threshold is determined according to the weight values ​​of the M directions. With such a solution, the main direction and the secondary direction can be adaptively distinguished, which is conducive to improving the robustness of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of an image processing method in an embodiment of the present application;

[0024] Figure 2 is a structural diagram of an image processing system in an embodiment of the present application;

[0025] Figure 3 It is a structural diagram of an image processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0026] As described in the background art, the existing demosaicing effect still needs to be improved.

[0027] Each pixel in the original image captured by CFA typically only contains the pixel value of a single channel. The demosaicing process requires interpolation to restore the pixel values ​​of the other channels. However, existing solutions are prone to inaccurate interpolation, resulting in suboptimal image quality. One way to improve image quality is to enrich the image's pixel information by adding interpolation directions. However, when there are too many interpolation directions, interpolation inaccuracies are more likely to occur.

[0028] In order to solve the above technical problems, an embodiment of the present application provides an image processing method. In the scheme of the embodiment of the present application, the color difference of each pixel point in the neighborhood of the current pixel point is calculated to obtain a color difference map, and the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel; then, according to the color difference map, the unit color difference values ​​and weight values ​​of M directions are determined; the weight values ​​of the N directions with the smallest weight values ​​among the M directions are reduced, and then fusion processing is performed based on the weight values ​​and the unit color difference values ​​of each direction to obtain a target color difference value; according to the pixel value of the current pixel point in the first channel and the target color difference value, the pixel value of the current pixel point in the second channel is determined to complete the interpolation of the second channel of the current pixel point.

[0029] Compared with the prior art, in the solution of the embodiment of the present application, the relevant information between the first channel and the second channel in each direction is characterized by the unit color difference value in each direction, and the importance of the relevant information in each direction to the pixel value of the current pixel point in the second channel is characterized by the weight value. Furthermore, by reducing the weight value of a part of the direction to weaken the relevant information of the secondary direction (for example, the direction with a smaller weight value) and strengthen the relevant information of the main direction (for example, the direction with a larger weight value), it is beneficial to improve the accuracy of interpolation, especially to improve the interpolation accuracy of texture-dense areas, thereby improving the quality of the final full-channel image, thereby improving the processing effect of Demosaic.

[0030] In order to make the above-mentioned objectives, features and beneficial effects of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0031] Reference Figure 1 , Figure 1 : is a flow chart of an image processing method in an embodiment of the present application. The method can be executed by a terminal, for example, a mobile phone, a computer, an Internet of Things device, a wearable device, a server, etc., but is not limited thereto. The terminal can be configured with a camera to have functions such as shooting and video recording. Alternatively, the terminal can be an edge device, for example, a camera with data processing functions, such as a camera used for monitoring security. Alternatively, the terminal can also be a device that is not configured with a camera, and the terminal can obtain images for processing from the outside, which is not limited in this embodiment. Figure 1 The image processing method shown may include the following steps:

[0032] Step S11: Calculate the color difference of each pixel in the neighborhood of the current pixel in the first image to obtain a color difference map, where the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel;

[0033] Step S12: Determine unit color difference values ​​and weight values ​​in M ​​directions according to the color difference map, where M is an integer greater than 1, the unit color difference value in each direction is used to represent the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction;

[0034] Step S13: reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer;

[0035] Step S14: fusing the unit color difference values ​​in each direction according to the weight value to obtain a target color difference value;

[0036] Step S15: obtaining the pixel value of the current pixel in the second channel according to the pixel value of the current pixel in the first channel and the target color difference value.

[0037] It can be understood that, in a specific implementation, the above method can be implemented in the form of a software program, which runs in a processor integrated inside a chip or chip module; alternatively, the method can be implemented in the form of hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module in combination with a software program; alternatively, the method can also be implemented in the form of hardware.

[0038] Before executing step S11, a first image may be acquired. The first image includes information of multiple channels, and each pixel has only a pixel value of a single channel. In a specific example, the first image may be a Bayer format image.

[0039] It should be noted that the multiple channels in the first image can be a combination of any of the following: R (red) channel, G (green) channel, B (blue) channel, Y (yellow) channel, Cy (Cyan) channel, Or (orange) channel, C (clear) channel, IR (infrared) channel, etc., but is not limited to this.

[0040] In one embodiment of the present application, the first image may be obtained by performing remosaic processing on an original image. The original image may be an image captured by a CFA, and the original image may be an image in RAW format. In other words, remosaic processing may convert a RAW image into a Bayer format image. Remosaic processing may also be referred to as re-mosaicing.

[0041] In a specific implementation, remosaic processing may include preprocessing and data structure splitting and reassembling. More specifically, preprocessing may include one or more of the following: bad pixel removal, black level processing, channel separation, etc.; data structure splitting and reassembling may include converting various Bayer data formats into a Bayer image that meets a preset format. It should be noted that the remosaic processing described above may be any suitable existing remosaic processing method, and this embodiment is not limited thereto.

[0042] In the specific implementation of step S11, the current pixel can be first determined in the first image. The current pixel has a single-channel pixel value, which can be recorded as the first channel. That is, the pixel value of the current pixel in the first channel is not obtained by interpolation. By executing steps S11 to S15 on the pixel value of the current pixel in the first channel, the pixel value of the current pixel in the second channel can be obtained. In other words, the first channel is the existing channel, and the second channel is the channel to be interpolated.

[0043] Furthermore, interpolation processing can be performed on the pixels within the neighborhood of the current pixel to obtain a pre-interpolated neighborhood. The neighborhood can be an area centered on the current pixel in the first image, and the size of the neighborhood can be pre-set. This embodiment does not limit the size of the neighborhood. The pre-interpolated neighborhood refers to the neighborhood after interpolation processing, and each pixel in the interpolated neighborhood has a pixel value of the first channel and a pixel value of the second channel.

[0044] It should be noted that the interpolation processing of the pixel points in the neighborhood may include the interpolation processing of the current pixel point. After the interpolation processing, the pixel value of the current pixel point in the second channel is not the pixel value of the second channel finally obtained in this embodiment, but is only an intermediate value in the algorithm, which is mainly used to calculate the color difference map.

[0045] Specifically, the neighborhood may include traversed pixels and untraversed pixels. The traversed pixels may be pixels that have been interpolated as the current pixel, and the untraversed pixels may be pixels that have not yet been used as the current pixel. In other words, the traversed pixels have pixel values ​​for all channels, while the untraversed pixels only have pixel values ​​for a single channel.

[0046] More specifically, interpolating pixels within the neighborhood of the current pixel may include: first channel pre-interpolation and second channel pre-interpolation, wherein the first channel pre-interpolation may be interpolating pixels within the neighborhood that do not have pixel values ​​of the first channel to obtain the pixel values ​​of the pixels in the first channel. The second channel pre-interpolation may be interpolating pixels within the neighborhood that do not have pixel values ​​of the second channel to obtain the pixel values ​​of the pixels in the second channel. Thus, after pre-interpolation, each pixel within the neighborhood has pixel values ​​of the first channel and the second channel.

[0047] The specific method for interpolating the pixels in the neighborhood may be any appropriate interpolation method, and the embodiments of the present application do not limit this.

[0048] Furthermore, a color difference map may be calculated based on the pixel values ​​of the first channel and the second channel of each pixel point in the pre-interpolated neighborhood.

[0049] The size of the color difference map is the same as the size of the neighborhood, the pixels of the color difference map correspond to the pixels of the neighborhood one-to-one, the value of each pixel in the color difference map is the color difference of the pixel in the neighborhood, and the color difference can be used to represent the difference between the pixel value of the same pixel in the first channel and the pixel value of the second channel. In a specific example, the color difference is the difference between the pixel value of the first channel and the pixel value of the second channel. For example, the color difference can be the result of subtracting the pixel value of the second channel from the pixel value of the first channel, or the color difference can also be the result of subtracting the pixel value of the first channel from the pixel value of the second channel.

[0050] In the specific implementation of step S12, on the one hand, the unit color difference values ​​in M ​​directions are calculated; on the other hand, the weight values ​​in the M directions are calculated. The "direction" referred to in this embodiment refers to the interpolation direction. The specific value of M can be preset, and M is an integer greater than 1. When M=2, the M directions may include up and down; when M=4, the M directions may include up, down, left, and right. In a specific example, M≥4. In a non-limiting example, M=8.

[0051] First, the unit color difference value in each direction is calculated. The unit color difference value in each direction can be used to represent the average value of the color difference in that direction.

[0052] In a specific implementation, the unit color difference value can be calculated using the following formula:

[0053]

[0054] Among them, CD average (i) represents the unit color difference value in the i-th direction, Q(i) represents the number of pixels in the i-th direction of the color difference image, CD i Represents the color difference of each pixel in the i-th direction in the color difference image, sum represents the sum calculation, 1≤i≤M, and i is a positive integer.

[0055] In other implementations, other methods may be used to calculate the unit color difference value in each direction. For example, a weighted average of the color differences of each pixel in the i-th direction may be performed to obtain the unit color difference value in the i-th direction. The farther away from the current pixel, the smaller the weight value of the pixel.

[0056] Secondly, weights are calculated for each direction. Each weight represents the severity of the color change in that direction. The more dramatic the color change and the more pronounced the texture, the smaller the weight. This reduces the problem of poor image quality caused by interpolation in densely textured areas.

[0057] Specifically, the weight value can be expressed as follows:

[0058]

[0059] Where w(i) represents the weight value in the i-th direction, CD_D(i) represents the gradient of the color difference in the i-th direction, and ∝ represents a positive correlation. In other words, the weight value in the i-th direction is negatively correlated with the gradient of the color difference in the i-th direction.

[0060] This embodiment does not limit the method for calculating the color difference gradient; any suitable existing gradient calculation method may be used. The color difference gradient CD_D(i) can represent the severity of the color difference change in the i-th direction. The larger the color difference gradient, the more severe the color difference change. In a specific example, the color difference gradient in the i-th direction can be the sum of the absolute values ​​of the color difference differences between all two adjacent pixels in the i-th direction.

[0061] Thus, the unit color difference value and weight value in each direction can be obtained.

[0062] In the specific implementation of step S13, the weight value w(i) obtained in step S12 may be adjusted to obtain an adjusted w'(i).

[0063] Specifically, the weight values ​​of N directions with the smallest weight values ​​among the M directions are reduced, where 1≤N≤M, and N is an integer.

[0064] In one embodiment, the weight values ​​of each direction can be compared with a reference threshold. If the weight value of any direction is less than or equal to the reference threshold, the weight value of that direction can be reduced; if the weight value is greater than the reference threshold, the weight value of that direction remains unchanged.

[0065] In other words, the weight value w(i) can be adjusted according to the following formula to obtain the adjusted weight value w'(i):

[0066]

[0067] Where threshold is the reference threshold, x is the reduction amplitude, x>0, w'(i)≥0.

[0068] In a specific implementation, the reduction amplitude x can be related to a reference threshold threshold. In one specific example, x = threshold. In another example, the reduction amplitude x = k × threshold, where k is an adjustment coefficient. The value of k can be preset or determined based on a color difference map.

[0069] In another embodiment, the weight values ​​of the M directions may be reduced, wherein the reduction magnitude of the weight values ​​greater than the reference threshold is smaller than the reduction magnitude of the weight values ​​not exceeding the reference threshold, wherein “not exceeding” may mean less than or equal to.

[0070] In other words, the weight value w(i) can be adjusted according to the following formula to obtain the adjusted weight value w'(i):

[0071]

[0072] Where x1 is the first reduction amplitude, and x2 is the second reduction amplitude. In a specific implementation, x1 and / or x2 can be fixed values, or x1 and / or x2 can be related to a reference threshold value, threshold. For example, x1 = k1 × threshold, x2 = k2 × threshold, where k1 < k2. For details about k1 and k2, refer to the above description of the adjustment coefficient k and are not repeated here.

[0073] In a non-limiting example, if the weight value in any direction is less than or equal to a reference threshold, the reduced weight value may be 0; if the weight value in any direction is greater than the reference threshold, the reduced weight value is the result of subtracting the reference threshold from the current weight value.

[0074] Through the adjustment process described above, the relevant information of the main direction can be strengthened and the relevant information of the secondary direction can be weakened, wherein the main direction can be screened out based on the reference threshold.

[0075] In a specific implementation, the reference threshold may be determined according to the weight values ​​of the M directions, thereby enabling adaptive selection of the main direction in practical applications.

[0076] In a specific example, the reference threshold may be determined based on the maximum and / or minimum value of the weight values ​​in the M directions. For example, the average of the maximum weight value and the minimum weight value may be used as the reference threshold.

[0077] In other embodiments, the reference threshold may also be determined based on a statistical method, for example, the average value of the weight values ​​of M directions, or the median value of the weight values ​​of M directions, but is not limited thereto.

[0078] In the specific implementation of step S14, the unit color difference values ​​in each direction are fused according to the weight value to obtain the target color difference value. It should be noted that the weight value in step S14 refers to the weight value after executing step S13, that is, the adjusted weight value w'(i).

[0079] In a specific example, the following formula can be used for fusion processing:

[0080]

[0081] Among them, CD target (i) represents the target color difference value in the i-th direction, w'(i) represents the adjusted weight value in the i-th direction, CD avreage (i) represents the unit color difference value in the i-th direction, and sum represents the summation calculation.

[0082] It should be noted that, in other embodiments, other existing appropriate methods may also be used for fusion processing, and this embodiment does not limit this.

[0083] In a specific implementation of step S15 , the pixel value of the current pixel point in the second channel may be obtained according to the pixel value of the current pixel point in the first channel and the target color difference value.

[0084] Specifically, the result of subtracting the target color difference value from the pixel value of the current pixel in the first channel can be used as the pixel value of the current pixel in the second channel, or the sum of the pixel value of the current pixel in the first channel and the target color difference value can be used as the pixel value of the current pixel in the second channel.

[0085] More specifically, if the color difference in step S11 is calculated by subtracting the pixel value of the second channel from the pixel value of the first channel, then the pixel value of the current pixel in the second channel in step S15 is the result of subtracting the target color difference value from the pixel value of the current pixel in the first channel. If the color difference in step S11 is calculated by subtracting the pixel value of the first channel from the pixel value of the second channel, then the pixel value of the current pixel in the second channel in step S15 is the sum of the pixel value of the current pixel in the first channel and the target color difference value. In this way, the pixel value of the current pixel in the second channel can be obtained, that is, the interpolation of the second channel of the current pixel is completed.

[0086] Furthermore, after executing step S15, if the current pixel point still has a channel to be interpolated, the channel to be interpolated can be used as the second channel, and return to step S11, and repeat steps S11 to S15 until the full-channel pixel value of the current pixel point is obtained.

[0087] Furthermore, the current pixel point may be updated, and the above steps may be repeated until all the pixel points in the first image are traversed, thereby obtaining the second image.

[0088] Therefore, in the embodiment of the present application, each pixel in the first image is sequentially used as the current pixel, and the above scheme is executed on the current pixel to obtain the full-channel pixel value of the current pixel. After traversing each pixel in the first image, a full-channel image can be obtained, which is recorded as the second image. In other words, each pixel in the second image has the pixel value of each channel in the first image.

[0089] Furthermore, color correction may be performed on the second image.

[0090] Specifically, the correction matrix may be determined according to the color temperature of the first image.

[0091] In a specific implementation, multiple correction matrix sets corresponding to color temperature scenes may be pre-set, where there is a one-to-one correspondence between color temperature scenes and correction matrix sets. Each correction matrix set corresponding to each color temperature scene may include multiple correction matrices, and the sizes of the multiple correction matrices in the same correction matrix set may vary. Each correction matrix in each correction matrix set may be pre-calibrated.

[0092] In a specific implementation, the current color temperature scene can be determined based on the color temperature of the first image, and a correction matrix set can be determined based on the current color temperature scene. Then, a correction matrix can be selected from the correction matrix set based on the number of channels of the first image. For example, if the first image includes five channels (R, G, B, Y, and Cy), the size of the selected correction matrix is ​​5×5; if the first image includes three channels (R, G, and B), the size of the selected correction matrix is ​​3×3.

[0093] Furthermore, color correction can be performed on the second image according to the correction matrix to obtain a target image.

[0094] For more details about color correction of the first image according to the correction matrix of the first image, reference may be made to existing color correction methods, and this embodiment does not limit this.

[0095] From the above, high-quality multi-channel images can be obtained through the solution of the embodiment of the present application.

[0096] Reference Figure 2 , Figure 2 It is a structural diagram of an image processing system in an embodiment of the present application. Figure 2 The image processing system shown may include: a data acquisition module 21, a data augmentation module 22, a calibration module 23, a first processing module 24, a second processing module 25, a correction module 26 and an output module 27. Figure 2 The directed arrows between modules are used to indicate the flow of data. Figure 2 The image processing system shown can be applied to application scenarios such as testing and verification of image processing methods, and this embodiment does not limit this.

[0097] Below Figure 2 Each module in the image processing system shown is described in detail.

[0098] The data acquisition module 21 can be used to collect multispectral data. Multispectral data can include single-channel images of multiple spectra, where a single-channel image means that the pixel values ​​of each pixel in the image are the pixel values ​​of the same channel. For example, multispectral data includes, but is not limited to, R channel images, G channel images, B channel images, Y channel images, Cy channel images, Or channel images, C channel images, and IR channel images.

[0099] More specifically, the multispectral data may include a single-channel image for interpolation and a color chart image for calibration, wherein the color chart image is also a single-channel image.

[0100] The data augmentation module 22 can be used to acquire multispectral data from the data acquisition module 21 and reconstruct the single-channel images of each spectrum to obtain Bayer images in different formats. As a result, the data output by the data augmentation module is an image in Bayer format. By amplifying the acquired multispectral data, the data augmentation module 22 can provide more calibration data and test data for subsequent modules (e.g., the calibration module 23). For ease of description, the image output by the data augmentation module 22 will be referred to as the base image below.

[0101] The calibration module 23 can be used to obtain a base image from the data augmentation module 22 and calibrate parameters based on the base image to obtain parameters used by the image processing algorithm in actual applications. For example, the calibrated parameters may include, but are not limited to, white balance correction parameters, color correction parameters, and distortion correction parameters. The white balance correction parameters and distortion correction parameters can be transmitted to the first processing module 24 for processing by the first processing module 24, and the color correction parameters can be transmitted to the correction module 26 for color correction by the correction module 26.

[0102] The first processing module 24 can be used to obtain the base image from the data augmentation module 22 and perform remosaic processing on the base image. The remosaic processing may include white balance correction of the base image using white balance correction parameters and distortion characterization of the base image using distortion correction parameters. For details about the remosaic processing, please refer to the above details and will not be repeated here.

[0103] The second processing module 25 can be used to obtain the Bayer format image from the first processing module 24. The second processing module 25 can perform Figure 1 The image processing method shown is used to obtain the full channel image corresponding to the basic image. Figure 1 The relevant description will not be repeated here.

[0104] The correction module 26 may be configured to perform color correction on the full-channel image obtained by the second processing module 25 . The color correction parameters used in the color correction process may be obtained from the calibration module 23 .

[0105] The output module 27 can be used to convert and output the color-corrected image, so as to convert the obtained image into a certain output format, for example, RGB format, YUV format, etc.

[0106] Reference Figure 3 , Figure 3 is a structural diagram of an image processing device in an embodiment of the present application, Figure 3 The apparatus shown may include:

[0107] A first calculation module 31 is configured to calculate the color difference of each pixel in a neighborhood of a current pixel in the first image to obtain a color difference map, wherein the color difference is used to represent the difference between the pixel value of the first channel and the pixel value of the second channel;

[0108] a second calculation module 32, configured to determine, based on the color difference map, unit color difference values ​​and weight values ​​in M ​​directions, where M is an integer greater than 1, the unit color difference value in each direction represents the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction;

[0109] An adjustment module 33 is configured to reduce the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer;

[0110] A third calculation module 34 is configured to perform a fusion process on the unit color difference values ​​in each direction according to the weight value to obtain a target color difference value;

[0111] The fourth calculation module 35 is configured to obtain the pixel value of the current pixel in the second channel according to the pixel value of the current pixel in the first channel and the target color difference value.

[0112] For more information about the working principle, working method, beneficial effects, etc. of the image processing device in the embodiment of the present application, please refer to the above description of the image processing method, which will not be repeated here.

[0113] In specific implementation, Figure 3 The image processing device shown may correspond to a chip with an image processing function in a terminal, or to a chip with an image processing function, such as an image signal processing (ISP) chip, etc.; or to a chip module with an image processing function, or to a terminal.

[0114] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program performs the steps of the above-mentioned image processing method. The storage medium may include ROM, RAM, a magnetic disk, or an optical disk. The storage medium may also include non-volatile memory or non-transitory memory.

[0115] The present application also provides a terminal comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the steps of the above-described image processing method when executing the computer program. The terminal includes, but is not limited to, mobile phones, computers, tablet computers, and other terminal devices.

[0116] The embodiment of the present application further provides a chip, which can be used to execute the above-mentioned image processing method. Alternatively, the chip may include Figure 3 In a specific example, the chip may be an image signal processing (ISP) chip.

[0117] The embodiment of the present application further provides a chip module, which can be used to execute the above-mentioned image processing method. Alternatively, the chip module may include Figure 3 An image processing apparatus is shown.

[0118] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0119] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0120] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units. For example, for various devices and products applied to or integrated into a chip, the various modules / units contained therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated into a chip module, the various modules / units contained therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units may be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units may be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module. The element can be implemented in the form of a software program, which runs on the processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on the processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0123] It should be understood that the term "and / or" as used herein simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " as used herein indicates that the related objects are in an "or" relationship.

[0124] The "multiple" appearing in the embodiments of the present application refers to two or more. The first, second, etc. descriptions appearing in the embodiments of the present application are only for illustration and distinction of the description objects. There is no order, nor does it represent a special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application. Although the present application is disclosed as above, the present application is not limited to this. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims.

[0125] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Calculating a color difference of each pixel in a neighborhood of a current pixel in the first image to obtain a color difference map, where the color difference represents a difference between a pixel value of a first channel and a pixel value of a second channel, wherein the current pixel has a pixel value of a single channel, the single channel being denoted as the first channel, the first channel being an existing channel, and the second channel being a channel to be interpolated; Determine, according to the color difference map, unit color difference values ​​and weight values ​​in M ​​directions, where M is an integer greater than 1, the unit color difference value in each direction is used to represent the average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction; Reduce the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer; The unit color difference values ​​in each direction are fused according to the weight value to obtain the target color difference value; The pixel value of the current pixel point in the second channel is obtained according to the pixel value of the current pixel point in the first channel and the target color difference value.

2. The image processing method according to claim 1, wherein: The first image is a Bayer format image, and calculating the color difference of each pixel in a neighborhood of a current pixel in the first image includes: Determine the current pixel point; Performing interpolation processing on the pixel points in the neighborhood, so that after the interpolation processing, each pixel point in the neighborhood has a pixel value of the first channel and the second channel; The color difference is calculated according to the pixel value of each pixel point in the neighborhood after the interpolation processing.

3. The image processing method according to claim 1, wherein: Reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions includes: Determining a reference threshold according to the weight values ​​of the M directions; For any one direction among the M directions, if the weight value of the direction is less than or equal to the reference threshold, the weight value of the direction is reduced.

4. The image processing method according to claim 3, wherein: If the weight value of any direction is less than or equal to the reference threshold, the reduced weight value of the direction is 0.

5. The image processing method according to claim 4, characterized in that Reducing the weight values ​​of the N directions with the smallest weight values ​​among the M directions further includes: For any one direction among the M directions, if the weight value of the direction is greater than the reference threshold, the weight value of the direction is reduced according to the reference threshold.

6. The image processing method according to claim 3, wherein: Determining the reference threshold according to the weight values ​​of the M directions includes: The reference threshold is determined according to the maximum value and / or the minimum value of the weight values ​​in the M directions.

7. The image processing method according to claim 1, wherein: The M is greater than or equal to 4.

8. The image processing method according to claim 1, wherein: The method further comprises: determining a color temperature of the first image; determining a correction matrix according to the color temperature of the first image; According to the correction matrix, color correction is performed on the second image to obtain a target image, wherein the second image is a full-channel image obtained by traversing each pixel point in the first image.

9. The image processing method according to claim 1, wherein: Before calculating the color difference of each pixel in the neighborhood of the current pixel in the first image, the method further includes: Get the original image; Remosaic processing is performed on the original image to obtain the first image.

10. An image processing device, characterized in that: The device comprises: a first calculation module, configured to calculate a color difference of each pixel in a neighborhood of a current pixel in the first image to obtain a color difference map, wherein the color difference is used to represent a difference between a pixel value of a first channel and a pixel value of a second channel, wherein the current pixel has a pixel value of a single channel, the single channel being denoted as the first channel, the first channel being an existing channel, and the second channel being a channel to be interpolated; a second calculation module, configured to determine, based on the color difference map, unit color difference values ​​and weight values ​​in M ​​directions, where M is an integer greater than 1, the unit color difference value in each direction represents an average value of the color difference in that direction, and the weight value in each direction is negatively correlated with the gradient of the color difference in that direction; an adjustment module, configured to reduce the weight values ​​of the N directions with the smallest weight values ​​among the M directions, where 1≤N≤M, and N is an integer; A third calculation module is used to fuse the unit color difference values ​​in each direction according to the weight value to obtain a target color difference value; A fourth calculation module is used to obtain the pixel value of the current pixel point in the second channel according to the pixel value of the current pixel point in the first channel and the target color difference value.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 9 are performed.

12. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor runs the computer program, the processor performs the steps of the image processing method according to any one of claims 1 to 9.

13. A chip, characterized in that: The chip is used to execute the steps of the image processing method according to any one of claims 1 to 9.

14. A chip module, characterized in that: The chip module is used to execute the steps of the image processing method according to any one of claims 1 to 9.

Images