Image processing device and method and electronic equipment

Abstract

The invention discloses an image processing device and method and electronic equipment, and belongs to the technical field of display. The device comprises a receiving interface and an image signal processor. The receiving interface is used for acquiring a first image; the image signal processor is used for determining base layer data and detail layer data according to any pair of adjacent pixel points in the first image; and the image signal processor is also used for processing the first image to obtain a second image according to the base layer data and the detail layer data corresponding to the plurality of pairs of adjacent pixel points. The base layer data and the detail layer data are only calculated once for a pair of adjacent pixel points, the calculation amount is reduced, and the image processing efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of display technology, and in particular to an image processing device, method and electronic device. Background Art

[0002] In the field of display technology, image quality can be improved by processing images to improve image contrast, hue, brightness, texture details, etc. Based on this, how to process images has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides an image processing device, method and electronic device, which can be used to solve the problems in the related art. The technical solution includes the following contents.

[0004] In a first aspect, an image processing device is provided, the device comprising a receiving interface and an image signal processor:

[0005] The receiving interface is used to obtain a first image;

[0006] The image signal processor is used to determine base layer data and detail layer data for any pair of adjacent pixel points in the first image;

[0007] The image signal processor is further used to process the first image to obtain a second image according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points.

[0008] In a possible implementation, one pixel in the first image corresponds to one channel, and the first image corresponds to multiple channels; determining the base layer data and the detail layer data for any pair of adjacent pixels in the first image includes:

[0009] Determine a plurality of designated pixels based on the first image and any pair of adjacent pixels, wherein the designated pixels correspond to designated channels among the plurality of channels, and the designated channels are channels corresponding to each pair of adjacent pixels;

[0010] The base layer data and the detail layer data are determined based on the multiple designated pixel points.

[0011] In a possible implementation manner, determining a plurality of designated pixel points based on the first image and any pair of adjacent pixel points includes:

[0012] Determine, in the first image, an image area centered around any pair of adjacent pixels;

[0013] A plurality of designated pixel points are determined based on the image area.

[0014] In a possible implementation manner, determining the base layer data and the detail layer data based on the multiple designated pixel points includes:

[0015] Performing data conversion on the pixel values ​​of the plurality of designated pixel points to obtain converted pixel values ​​of the plurality of designated pixel points;

[0016] The base layer data and the detail layer data are determined based on the converted pixel values ​​of the plurality of designated pixel points.

[0017] In a possible implementation, determining the base layer data and the detail layer data based on the converted pixel values ​​of the plurality of designated pixel points includes:

[0018] Determine statistical data based on the converted pixel values ​​of the plurality of designated pixel points, the statistical data representing fluctuations of the converted pixel values ​​of the plurality of designated pixel points;

[0019] Determining grassroots data based on the statistical data;

[0020] Detail layer data is determined based on the base layer data.

[0021] In a possible implementation manner, determining the basic data based on the statistical data includes:

[0022] Determining a weight coefficient according to the statistical data;

[0023] The base layer data is determined according to the converted pixel value of any designated pixel point, the weight coefficient and the statistical data.

[0024] In a possible implementation manner, any pair of adjacent pixel points corresponds to a plurality of base layer data and a plurality of detail layer data; and determining the detail layer data based on the base layer data includes:

[0025] For any pair of adjacent pixels, the corresponding detail layer data is determined based on any corresponding base layer data and the converted pixel value of any designated pixel, or the corresponding detail layer data is determined based on at least two corresponding base layer data.

[0026] In a possible implementation, the first image includes pixel values ​​of each pixel, the any pair of adjacent pixels includes a first pixel and a second pixel, the first pixel corresponds to the designated channel, and the second pixel corresponds to any channel of the multiple channels except the designated channel; and the processing of the first image to obtain the second image according to the base layer data and the detail layer data corresponding to the multiple pairs of adjacent pixels includes:

[0027] For any pair of adjacent pixels, obtain a first brightness of a first pixel and a first brightness of a second pixel, and determine a reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel;

[0028] For any pixel in any pair of adjacent pixels, processing a pixel value of the any pixel according to the reference brightness and the first brightness of the any pixel to obtain processing data of the any pixel;

[0029] A second image is determined according to the processed data of each pixel.

[0030] In a possible implementation manner, determining a reference brightness according to the base layer data, the detail layer data, and the first brightness of the first pixel includes:

[0031] Determine a base layer brightness according to the base layer data and a first brightness of the first pixel;

[0032] A reference brightness is determined according to the base layer brightness and the detail layer data.

[0033] In a second aspect, an image processing method is provided, the method comprising:

[0034] acquiring a first image;

[0035] Determine base layer data and detail layer data for any pair of adjacent pixel points in the first image;

[0036] The first image is processed according to the base layer data and detail layer data corresponding to a plurality of pairs of adjacent pixel points to obtain a second image.

[0037] In a possible implementation, one pixel in the first image corresponds to one channel, and the first image corresponds to multiple channels; determining the base layer data and the detail layer data for any pair of adjacent pixels in the first image includes:

[0038] Determine a plurality of designated pixels based on the first image and any pair of adjacent pixels, wherein the designated pixels correspond to designated channels among the plurality of channels, and the designated channels are channels corresponding to each pair of adjacent pixels;

[0039] The base layer data and the detail layer data are determined based on the multiple designated pixel points.

[0040] In a possible implementation manner, determining a plurality of designated pixel points based on the first image and any pair of adjacent pixel points includes:

[0041] Determine, in the first image, an image area centered around any pair of adjacent pixels;

[0042] A plurality of designated pixel points are determined based on the image area.

[0043] In a possible implementation manner, determining the base layer data and the detail layer data based on the multiple designated pixel points includes:

[0044] Performing data conversion on the pixel values ​​of the plurality of designated pixel points to obtain converted pixel values ​​of the plurality of designated pixel points;

[0045] The base layer data and the detail layer data are determined based on the converted pixel values ​​of the plurality of designated pixel points.

[0046] In a possible implementation, determining the base layer data and the detail layer data based on the converted pixel values ​​of the plurality of designated pixel points includes:

[0047] Determine statistical data based on the converted pixel values ​​of the plurality of designated pixel points, the statistical data representing fluctuations of the converted pixel values ​​of the plurality of designated pixel points;

[0048] Determining grassroots data based on the statistical data;

[0049] Detail layer data is determined based on the base layer data.

[0050] In a possible implementation manner, determining the basic data based on the statistical data includes:

[0051] Determining a weight coefficient according to the statistical data;

[0052] The base layer data is determined according to the converted pixel value of any designated pixel point, the weight coefficient and the statistical data.

[0053] In a possible implementation manner, any pair of adjacent pixel points corresponds to a plurality of base layer data and a plurality of detail layer data; and determining the detail layer data based on the base layer data includes:

[0054] For any pair of adjacent pixels, the corresponding detail layer data is determined based on any corresponding base layer data and the converted pixel value of any designated pixel, or the corresponding detail layer data is determined based on at least two corresponding base layer data.

[0055] In a possible implementation, the first image includes pixel values ​​of each pixel, the any pair of adjacent pixels includes a first pixel and a second pixel, the first pixel corresponds to the designated channel, and the second pixel corresponds to any channel of the multiple channels except the designated channel; and the processing of the first image to obtain the second image according to the base layer data and the detail layer data corresponding to the multiple pairs of adjacent pixels includes:

[0056] For any pair of adjacent pixels, obtain a first brightness of a first pixel and a first brightness of a second pixel, and determine a reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel;

[0057] For any pixel in any pair of adjacent pixels, processing a pixel value of the any pixel according to the reference brightness and the first brightness of the any pixel to obtain processing data of the any pixel;

[0058] A second image is determined according to the processed data of each pixel.

[0059] In a possible implementation manner, determining a reference brightness according to the base layer data, the detail layer data, and the first brightness of the first pixel includes:

[0060] Determine a base layer brightness according to the base layer data and a first brightness of the first pixel;

[0061] A reference brightness is determined according to the base layer brightness and the detail layer data.

[0062] In a third aspect, a chip is provided, wherein the chip includes any image processing device shown in the first aspect.

[0063] According to a fourth aspect, an electronic device is provided, wherein the electronic device comprises the chip shown in the third aspect.

[0064] The technical solution provided by this application brings at least the following beneficial effects:

[0065] The image signal processor in the technical solution provided by the present application calculates the base layer data and the detail layer data for a pair of adjacent pixel points. Since a pair of adjacent pixel points is calculated only once, the amount of calculation is reduced, and the resource occupation is reduced, so that when the first image is subsequently processed according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points, the image processing efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0068] Figure 2 is a schematic diagram of a RAW image provided in an embodiment of the present application;

[0069] Figure 3 is a schematic diagram for determining a designated pixel point provided in an embodiment of the present application;

[0070] Figure 4 It is a statistical diagram of data provided by an embodiment of the present application;

[0071] Figure 5 is a schematic diagram of a weight coefficient provided in an embodiment of the present application;

[0072] Figure 6 is an interpolation schematic diagram provided in an embodiment of the present application;

[0073] Figure 7 is a schematic diagram of brightness contrast provided in an embodiment of the present application;

[0074] Figure 8 is a schematic diagram of a mapping curve provided in an embodiment of the present application;

[0075] Fig. 9 It is a flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0077] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] Figure 1 The schematic diagram of the structure of an image processing device provided in an embodiment of the present application includes a receiving interface 10 and an image signal processor (ISP) 20. The black circle in the structural diagram is used to mark the input end, and the white circle is used to mark the output end. Figure 1 As shown, the output end of the receiving interface 10 is connected to the input end of the image signal processor 20 .

[0079] In the embodiment of the present application, the receiving interface 10 is used to acquire a first image.

[0080] The user may input the first image into the electronic device, or the electronic device may obtain the first image through the Internet, or the electronic device may obtain the first image through an image acquisition device, etc. The electronic device includes a receiving interface 10, and the electronic device obtains the first image. When the first image needs to be processed, the first image is obtained through the receiving interface 10.

[0081] The embodiment of the present application does not limit the type of the first image. For example, the first image can be a color image or a grayscale image. Optionally, any image that meets the following conditions can be used as the first image. The conditions are: the first image corresponds to multiple channels, but the pixels in the first image correspond to a single channel, and there are multiple pixels in the first image corresponding to the same channel. For example, Figure 2 As shown, Figure 2 A RAW image including 9×9 pixels (i.e., 9 rows and 9 columns) is shown. It should be noted that the RAW image corresponds to three channels of RGB (Red-Green-Blue), and the pixels in the RAW image correspond to the R channel, the G channel, or the B channel. Therefore, the RAW image can be used as the first image.

[0082] The multiple channels include a designated channel, which is a channel that is useful for processing an image. For example, image tone mapping processing processes an image according to the brightness of the image, and the G channel is the main source of brightness. Therefore, the G channel can be a designated channel. The data of any pixel point in the first image is called a pixel value.

[0083] The image signal processor 20 is used to determine base layer data and detail layer data for any pair of adjacent pixel points in the first image.

[0084] A pair of adjacent pixels is two pixels that are adjacent to each other in the first image. Figure 2 In the first image shown, the R pixel and the G pixel with a gray background are adjacent to each other in the first image, so these two pixel points are a pair of adjacent pixel points.

[0085] In this example, a pair of adjacent pixels share the same base layer data and detail layer data. The base layer data contains the low-frequency features of the specified pixel, and the detail layer data contains the high-frequency features of the specified pixel. In the field of images, low frequency refers to the change of information such as color, grayscale, brightness, etc. is less than the threshold, and high frequency refers to the change of information greater than the threshold. Based on this, it can be seen that low-frequency features can reflect the changes in brightness or color of the image on a large scale, corresponding to large areas with slow changes in the image, and high-frequency features can reflect the details of the image on a small scale, corresponding to the edge, texture and other information of the image.

[0086] The embodiment of the present application does not limit the method of determining the base layer data and the detail layer data for any pair of adjacent pixels. It is understandable that the method of determining the base layer data and the detail layer data varies depending on the type of the first image. A possible implementation is shown below.

[0087] In this example, one pixel in the first image corresponds to one channel, and the first image corresponds to multiple channels. For example, the first image is a RAW image. Determining the base layer data and detail layer data for any pair of adjacent pixel points in the first image includes: determining multiple designated pixel points based on the first image and any pair of adjacent pixel points, the designated pixel points corresponding to designated channels in the multiple channels, and the designated channels are channels corresponding to each pair of adjacent pixel points; determining the base layer data and detail layer data based on the multiple designated pixel points.

[0088] For a pair of adjacent pixels, one pixel corresponds to a designated channel, which can be called a first pixel, and the other pixel corresponds to a non-designated channel, which can be called a second pixel. Figure 2 As shown, in the RAW image, the G pixel is adjacent to the R pixel or the B pixel, the G channel is the designated channel, and the non-designated channels include the R channel and the B channel. Based on this, the G pixel with a gray background is the first pixel, and the R pixel with a gray background is the second pixel.

[0089] Among them, the pixel points of the designated channel are designated pixel points. Therefore, the first pixel point belongs to the designated pixel point. For any pair of adjacent pixel points, multiple designated pixel points can be determined based on the first image, and the determination method is not limited here. For example, the multiple designated pixel points include designated pixel points in the row or column where any pair of adjacent pixel points in the first image are located.

[0090] Alternatively, determining a plurality of designated pixels based on the first image and any pair of adjacent pixel points includes: determining an image area centered on any pair of adjacent pixel points in the first image; and determining a plurality of designated pixel points based on the image area.

[0091] The image region may include a region in the first image, and may also include other regions determined based on the region, for example, the other regions are mirror-symmetrical with the region (or a part of the region). Figure 3 As shown in (1), a pair of adjacent pixels includes an R pixel and a G pixel shown in gray background. With this pair of pixels as the center, the following is determined in the first image: Figure 3 If a pair of adjacent pixels is located in the edge region of the first image, the image can be expanded by mirror-flipping the first image, so that the image region is determined in the expanded image with the pair of pixels as the center.

[0092] In practical applications, the size of the image region can be flexibly changed according to the situation, as long as the size of the image region is smaller than the size of the first image. For example, the size of the image region can be 7×8 or 11×12, etc.

[0093] The image signal processor 20 can extract a plurality of designated pixel points from the image region, and these designated pixel points are all or part of the image region. Figure 3 As shown, in Figure 3 In the image area shown in (1), by taking G pixels in alternate rows and columns, the following Figure 3 The G pixel shown in the gray background in (2) is used to obtain the designated pixel points of 5 rows and 5 columns (i.e., 5×5). Optionally, the designated channel is the main source of the image brightness. By first determining the image area and then determining the designated pixel point, the brightness around the adjacent pixel points is extracted to process the image based on the brightness and improve the image quality.

[0094] Afterwards, the image signal processor 20 determines the base layer data and the detail layer data based on a plurality of designated pixel points, and the determination method is not limited here. Exemplarily, a filtering operation can be performed based on the pixel values ​​of a plurality of designated pixel points through a guided filter to obtain the base layer data, and a layering operation can be performed based on the base layer data to obtain the detail layer data. Among them, the input of the guided filter includes a guiding image and an input image, and the input image is guided by the guiding image to obtain the output image. When the input image is consistent with the guiding image, the output image is the base layer image. The input image includes a plurality of designated pixel points, and the data in the base layer image is the base layer data. According to the input image and the base layer data, the detail layer data can be determined.

[0095] In a possible implementation, the first image includes pixel values ​​of each pixel point. Determining the base layer data and detail layer data based on multiple designated pixel points includes: performing data conversion on the pixel values ​​of the multiple designated pixel points to obtain the converted pixel values ​​of the multiple designated pixel points; and determining the base layer data and detail layer data based on the converted pixel values ​​of the multiple designated pixel points.

[0096] A data conversion algorithm can be used to perform data conversion on the pixel value of a specified pixel point to obtain the pixel value after conversion of the specified pixel point. The embodiment of the present application does not limit the data conversion algorithm. For example, the data conversion algorithm is a linear function y=ax, where y represents the pixel value after conversion, x represents the pixel value, and a is a coefficient less than 0.5. Alternatively, the data conversion algorithm is a logarithmic function y=log a x, y represent the converted pixel value, x represents the pixel value, a is the base of the logarithm, and the commonly used bases are 2, 10 or natural exponent e. By converting data through logarithmic functions, multiplication and division processing can be converted into addition and subtraction processing, reducing the difficulty of subsequent data processing.

[0097] Next, based on the pixel values ​​after conversion of multiple specified pixel points, the base layer data and the detail layer data are determined. The determination method is not limited here. For example, a guided filter can be used to perform a filtering operation based on the pixel values ​​after conversion of multiple specified pixel points to obtain the base layer data, and a layering operation can be performed based on the base layer data to obtain the detail layer data.

[0098] In general, the larger the bit width of a pixel value, the larger the brightness or chromaticity range corresponding to the pixel value. For example, when the bit width of a pixel value is 8 bits, the brightness range or chromaticity range is 0 to 255, and when the bit width of a pixel value is 10 bits, the brightness range or chromaticity range is 0 to 1023. If the first image is a high dynamic range image, the difference in bit width between pixels in the first image is too large, and it is difficult to accurately extract high-frequency features. If filtering and layering operations are performed directly based on pixel values, the accuracy of the detail layer data is poor. Based on this, by performing data conversion on pixel values ​​to obtain converted pixel values, reducing the difference between data, and compressing high-frequency information, filtering and layering operations are performed based on converted pixel values, more accurate detail layer data can be obtained, thereby improving image processing effects.

[0099] Optionally, determining the base layer data and detail layer data based on the pixel values ​​after conversion of multiple specified pixel points includes: determining statistical data based on the pixel values ​​after conversion of multiple specified pixel points, the statistical data characterizing the fluctuation of the pixel values ​​after conversion of multiple specified pixel points; determining the base layer data based on the statistical data; determining the detail layer data based on the base layer data. It is understandable that in actual applications, the base layer data and detail layer data can be determined based on the pixel values ​​of multiple specified pixel points according to the implementation principle of this step, and the determination process will not be repeated here.

[0100] In this example, the statistical data includes at least one of the following: a maximum value, a minimum value, an average value, a variance, a standard deviation, etc. determined based on the pixel values ​​after the conversion of multiple specified pixel points. Taking the statistical data including the maximum value, the minimum value and the average value as an example, assuming that the pixel values ​​after the conversion of multiple specified pixel points are sequentially represented as: G0, G1, G2, ..., then according to G max = max(G0, max(G1, max(G2, ...))), determine the maximum value G from these converted pixel values max , according to G min = min(G0, min(G1, min(G2, ...))), determine the minimum value G from these converted pixel values min ,according to Determine the mean value G from these converted pixel values mean , M is the number of converted pixel values.

[0101] It is understandable that the variance of an image block can reflect the degree of fluctuation of the image block. The smaller the variance of an image block, the flatter the image block; the larger the variance of an image block, the closer the image block is to an area with rich textures and edges. However, the process of calculating the variance is very hardware-intensive. The image block can be regarded as a set of data, such as Figure 4 As shown in (1) in , if the difference between the maximum and minimum values ​​in this set of data is small, then this set of data can be considered to be basically equal, that is, the image block is relatively flat; Figure 4 As shown in (2), if the gap between the maximum and minimum values ​​in this set of data is large, then the fluctuation of this set of data is relatively large, and it can be considered that the image block is close to an area with rich textures and edges. In addition, the average value of a set of data can reflect the centralization trend of this set of data. When applied to the image field, the average value of the pixel value can reflect a large range of the image. Based on this, the embodiment of the present application does not need to calculate the variance of multiple converted pixel values, but determines the minimum, maximum and average values ​​of multiple converted pixel values, so as to perform filtering operations and layering operations through the minimum, maximum and average values, thereby improving the accuracy of the base layer data and the detail layer data, reducing the amount of calculation of the operation, and reducing the consumption of hardware resources.

[0102] In an exemplary embodiment, determining the base layer data based on statistical data includes: determining a weight coefficient according to the statistical data; and determining the base layer data according to the converted pixel value of any specified pixel point, the weight coefficient and the statistical data.

[0103] The following uses the example of statistical data including maximum value, minimum value and average value to explain the method of determining the basic data. This is similar to the method of determining the basic data when the statistical data includes other data, and will not be described in detail in the embodiments of the present application.

[0104] Assume the maximum value is Gmax , the minimum value is G min , then the difference can be expressed as G max -G mi2 Then, based on the difference G max -G min Determine the weight coefficient, wherein the method for determining the weight coefficient is not limited herein, for example, a linear function is pre-established in which the weight coefficient is positively correlated with the difference, and the weight coefficient is determined based on the linear function and the difference.

[0105] Optionally, the statistical data includes a maximum value and a minimum value; determining the weight coefficient based on the statistical data includes: determining a first coefficient based on a difference between the maximum value and the minimum value; and determining the minimum coefficient between the coefficient threshold and the first coefficient as the weight coefficient.

[0106] In the embodiment of the present application, the first coefficient is positively correlated with the difference. max -G min Determining the standard deviation Multiply the standard deviation by the slope k to get the first coefficient k×simp_stand. And the variance simp_var=(simp_stand) 2 In comparison, the standard deviation can save a multiplier, and the standard deviation is consistent with the monotonicity of the variance, so the two can achieve the same effect. By setting the slope k, the standard deviation simp_stand is mapped to the first coefficient. In practical applications, the slope k can be multiple. Under the same standard deviation simp_stand, different first coefficients can be obtained by setting different slopes k, so as to facilitate the execution of filtering of different scales.

[0107] In addition, the electronic device may obtain a coefficient threshold. Optionally, the coefficient threshold is data input by a user, or the coefficient threshold is data set based on manual experience. Assume that the coefficient threshold is w max , then the weight coefficient satisfies: w = min (w max ,k×simp_stand). Coefficient threshold w max The value range of is [0,1], such as Figure 5 As shown, by setting the coefficient threshold, the weight coefficient has an upper limit, which reduces the subsequent over-filtering situation, allows some average values ​​to be mixed in the filtering result, improves the accuracy of the base layer data, and thus improves the image processing effect.

[0108] Next, the image signal processor 20 determines the base data according to the pixel value, weight coefficient and average value after the conversion of any specified pixel point. The method for determining the base data is not limited here. For example, the output result of the filter can be regarded as the weighting of the original pixel and the average value. Based on this, a weighted calculation is performed according to the weight coefficient, the average value and the pixel value after the conversion of any specified pixel point to obtain the base data. Alternatively, according to B=w(G center -G mean )+G mean , determine the base data B, where w is the weight coefficient, G center is the pixel value of the specified pixel point located at the center among multiple specified pixels after conversion, G mean is the average value.

[0109] It can be understood that the smaller the standard deviation simp_stand, the smaller the weight coefficient w, and the closer the base data B is to the average value G mean , the filter is mainly used to smooth the image; the larger the standard deviation simp_stand, the larger the weight coefficient w, and the closer the base data B is to the converted pixel value G center ,The filter is mainly used to preserve texture details.

[0110] As mentioned above, the difference between the maximum and minimum values ​​can reflect the degree of fluctuation of the image block, while the average value can reflect the large-scale area of ​​the image block. In this example, the weight coefficient is first determined based on the difference between the maximum and minimum values, and then the converted pixel values ​​are filtered according to the weight coefficient and the average value, so that a relatively suitable filter is selected according to the degree of fluctuation and the large-scale area of ​​the image block, so that the base layer data obtained by filtering the converted pixel values ​​through the filter has a higher accuracy, and the detail layer data determined based on the base layer data also has a higher accuracy. Among them, the method of determining the detail layer data based on the base layer data is not limited here.

[0111] Optionally, any pair of adjacent pixel points corresponds to multiple base layer data and multiple detail layer data; determining the detail layer data based on the base layer data includes: for any pair of adjacent pixel points, determining any corresponding detail layer data based on any corresponding base layer data and the converted pixel value of any specified pixel point, or determining any corresponding detail layer data based on at least two corresponding base layer data.

[0112] In this example, multiple first coefficients are obtained by setting multiple slopes, and / or multiple coefficient thresholds are set. Multiple weight coefficients are determined by multiple first coefficients and / or multiple coefficient thresholds, so that a multiscale filter is determined to obtain multiple base layer data by multiscale filtering through the multiscale filter. Subsequently, multiple base layer data can be multi-scale layered to obtain multiple detail layer data.

[0113] For example, the weight coefficients are w1 and w2, respectively, according to B1=w1(G center -G mean )+G mean Determine a base data, according to B2 = w2 (G cenyer -G mean )+G mean Determine another base layer data. The pixel value after the specified pixel point is converted can be subtracted from the base layer data to obtain the detail layer data. For example, according to D1 g =G center -B1 determines the detail layer data D1 g The difference between at least two base layer data may also be determined as detail layer data, for example, according to D2 g =B2-B1 determines the detail layer data D2 g .

[0114] The image signal processor 20 is further configured to process the first image to obtain a second image according to the base layer data and detail layer data corresponding to a plurality of pairs of adjacent pixel points.

[0115] Each pair of adjacent pixels can determine at least one base layer data and at least one detail layer data, and tone mapping is performed on the first image according to each base layer data and each detail layer data to obtain a second image. Tone mapping is to perform a large contrast attenuation to transform the scene brightness to a displayable range while maintaining image details and color information, so that the tone mapped image matches the perception of the real scene.

[0116] In a possible implementation, the first image includes pixel values ​​of each pixel point, and any pair of adjacent pixels includes a first pixel point and a second pixel point, the first pixel point corresponds to a specified channel, and the second pixel point corresponds to any channel of multiple channels except the specified channel; the first image is processed according to the base layer data and detail layer data corresponding to the multiple pairs of adjacent pixels to obtain the second image, including: for any pair of adjacent pixels, obtaining the first brightness of the first pixel point and the first brightness of the second pixel point, and determining the reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel point; for any pixel point in any pair of adjacent pixels, processing the pixel value of any pixel point according to the reference brightness and the first brightness of any pixel point to obtain the processed data of any pixel point; and determining the second image according to the processed data of each pixel point.

[0117] Adjacent pixels include pixels corresponding to a specified channel and pixels corresponding to other channels. For ease of description, the pixels corresponding to the specified channel are referred to as first pixels, and the pixels corresponding to other channels are referred to as second pixels. For example, in a RAW image, the G channel pixel is referred to as the first pixel, and the B channel pixel or the R channel pixel is referred to as the second pixel.

[0118] In this example, the first brightness of the first pixel is determined according to the pixel value of the first pixel or the converted pixel value. Optionally, the pixel value of the first pixel or the converted pixel value is the data of the first pixel in a specified channel. According to the pixel value or the converted pixel value of an adjacent pixel (adjacent to the first pixel) of any other channel, the data of the first pixel in the other channel is interpolated. According to the data of the first pixel in multiple channels, the first brightness of the first pixel is determined.

[0119] Specifically, taking a RAW image as an example, the first pixel is a G channel pixel. Figure 2 As shown, assume that the pixel value or converted pixel value of the pixel point in the i-th row and j-th column is recorded as M ij , M represents the channel of the pixel, i and j are both positive numbers. For the pixel in the first row and second column, the pixel corresponds to the G channel, and the data of the pixel in the G channel satisfies: G = G 12 The data of this pixel in the R channel satisfies: The data of the pixel in the B channel satisfies: Afterwards, the first brightness lum of the pixel is determined according to lum=αR+βG+γB, where α, β, and γ are three weights that can be flexibly set according to actual needs. For example, α=0.229, β=0.587, and γ=0.114.

[0120] Similarly, the pixel value or converted pixel value of the second pixel is the data of the second pixel in another channel. According to the pixel value or converted pixel value of an adjacent pixel (adjacent to the second pixel) in another channel, the data of the second pixel in the other channel is interpolated. According to the pixel value or converted pixel value of an adjacent pixel (adjacent to the second pixel) in a specified channel, the data of the second pixel in a specified channel is interpolated. According to the data of the second pixel in multiple channels, the first brightness of the second pixel is determined.

[0121] Specifically, still taking the RAW image as an example, the second pixel point is an R channel pixel point or a B channel pixel point. Figure 2 As shown, assume that the pixel value or converted pixel value of the pixel point in the i-th row and j-th column is recorded as M ij, M represents the channel of the pixel, i and j are both positive numbers. For the pixel in the first row and first column, the pixel corresponds to the B channel, and the data of the pixel in the B channel satisfies: B = B 11 The data of this pixel in the R channel satisfies: The data of this pixel in the G channel satisfies: That is: if abs(G 01 -G 21 ) is greater than abs(G 10 -G 12 ),but If abs(G 01 -G 21 ) is not greater than abs(G 10 -G 12 ),but Afterwards, the first brightness of the pixel is determined according to lum=αR+βG+γB.

[0122] According to the interpolation method of multiple channel data described above, such as Figure 6 As shown, for a pixel window of 9 rows and 9 columns, data of the R channel (as shown by reference numeral 701), data of the G channel (as shown by reference numeral 702), and data of the B channel (as shown by reference numeral 703) of the pixel points of 7 rows and 7 columns in the window (i.e., the pixel points shown by the white background) can be interpolated. In practical applications, the pixel window can be expanded by mirror filling to interpolate data of the R channel, G channel, and B channel of the pixel points in the edge area of ​​the window (i.e., the pixel points shown by the gray background) according to the difference method described above.

[0123] In this example, the brightness information in the image mainly comes from the specified channel, and other channels contribute less to the brightness. Therefore, the first brightness of the first pixel can be used to calculate the reference brightness, which not only reduces the amount of calculation, but also ensures the accuracy of the reference brightness and improves the image processing quality.

[0124] like Figure 7 As shown in the figure, in any RAW image, a 99×100 area is selected, and the area is interpolated to obtain the data of each pixel in the area in the R, G, and B channels. The brightness of the pixel is determined based on the data of the pixel in the R, G, and B channels. Figure 7 The image shown in (2) in . Figure 7 In (2), x represents the horizontal coordinate of the pixel in the 99×100 area, y represents the vertical coordinate of the pixel in the 99×100 area, that is, the position of any pixel in the 99×100 area can be expressed as (x, y), and z represents the brightness of the pixel.

[0125] Extract G channel pixels from the 99×100 area in alternate rows and columns to obtain a 50×50 area. Based on the brightness of each G channel pixel in the area, determine Figure 7 The image shown in (1) in . Figure 7 In (1), x represents the horizontal coordinate of the G channel pixel in the 50×50 area, and y represents the vertical coordinate of the G channel pixel in the 50×50 area. That is to say, the position of any G channel pixel in the 50×50 area can be expressed as (x, y), and z represents the brightness of the G channel pixel.

[0126] Figure 7 (1) shows the spatial fluctuation of the G channel data in the RAW image. Figure 7 (2) in the figure shows the fluctuation of the brightness of the RAW image in space. The fluctuations of the two are basically the same, except that the pixels of the G channel are extracted in alternate rows and columns. Therefore, Figure 7 The fluctuation shown in (1) is equivalent to Figure 7 The downsampling of the fluctuations shown in (2) in the figure. That is to say, in the RAW image, the pixels of the G channel are twice as many as the pixels of the R channel or the B channel, and the photosensitivity of the G channel pixels is better than that of the R channel pixels or the B channel pixels. Therefore, most of the brightness of the RAW image comes from the G channel pixels. Therefore, it is reasonable to replace the brightness of the adjacent pixels with the brightness of the G channel pixels. Based on this, the embodiment of the present application determines the reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel.

[0127] In an exemplary embodiment, the reference brightness is determined according to the base layer data, the detail layer data and the first brightness of the first pixel, including: determining the base layer brightness according to the base layer data and the first brightness of the first pixel; determining the reference brightness according to the base layer brightness and the detail layer data.

[0128] In this example, the first sub-brightness can be determined according to the base layer data, the first brightness of the first pixel, and the pixel value or the converted pixel value of any specified pixel. Optionally, any specified pixel is the first pixel. The maximum value of the first sub-brightness and the preset second sub-brightness is determined as the base layer brightness. For example, according to B lum =max(0,lum g -(G center -B g )) Determine the base brightness B lum , where the second sub-brightness is 0 and the first sub-brightness is lum g -(G center -B g ),lum g Represents the first brightness of the first pixel, G centerrepresents the pixel value of the first pixel or the converted pixel value, B g Represents the underlying data.

[0129] Next, the base layer brightness and detail layer data are weighted to obtain a reference brightness, wherein the detail layer data is at least one. For example, according to T = k1 × B lum +k2×D1 g +k3×D2 g , determine the reference brightness T, k1, k2, k3 are three weights, B lum Characterizes the brightness of the base layer, D1 g 、D2 g Characterizing two detail layer data. It can be understood that different detail layer data correspond to texture details of different scales. By setting different weights for each detail layer data, multi-scale enhancement of texture details of different scales can be achieved, which is conducive to improving image processing effects and thus improving image quality.

[0130] Afterwards, for any pixel in any pair of adjacent pixels, the image signal processor 20 processes the pixel value of the pixel according to the reference brightness and the first brightness of the pixel to obtain the processed data of the pixel. In practical applications, if the pixel value is first converted to obtain the converted pixel value, the reference brightness can be inversely converted to restore the data structure, and the first image is then processed based on the reference brightness after inverse data conversion. a For example, when performing data conversion on pixel values, the exponential function T can be used for inverse data conversion. l =a T Perform an inverse data conversion on the reference brightness T, where a is the base of the logarithm and also the base of the exponent.

[0131] The reference brightness or the inverse data converted reference brightness can be normalized. Since each pair of adjacent pixels corresponds to a reference brightness, the maximum value can be determined from each reference brightness or each inverse data converted reference brightness, and the reference brightness or the inverse data converted reference brightness is divided by the maximum value to obtain the normalized reference brightness. Then, according to the mapping curve, the normalized reference brightness is mapped to the reference contrast. Optionally, as Figure 8 As shown, the mapping curve is an S-shaped curve between normalized brightness and contrast.

[0132] For example, according to Determine the reference contrast L out , CE characterization mapping curve, T l Characterizes the reference brightness after inverse data conversion, Characterizes the maximum value of the reference brightness after each inverse data conversion.

[0133] Afterwards, according to the reference contrast and the first brightness of any pixel, the pixel value of the pixel is subjected to tone processing to obtain the processed data of the pixel. Determine the processed data out of the pixel c Among them, (RAW) c Characterizes the pixel value of the pixel, (lum) c Characterizes the first brightness of the pixel, L out Characterize the reference contrast. In this way, the pixel value after conversion of each pixel point is determined, thereby determining the second image.

[0134] In practical applications, when processing the first image, the image signal processor 20 can process at least one pixel in one clock cycle. For example, in one clock cycle, the image signal processor 20 determines a 9×10 image area based on the first image for a pair of adjacent pixels. The 9×10 image area can be regarded as two 9×9 image areas, and the central pixels of the two image areas are the pair of adjacent pixels. The 9×10 image area includes 5×5 designated pixels, and the pair of adjacent pixels can be processed according to the 5×5 designated pixels. Since a pair of adjacent pixels is processed in one clock cycle, the data of this clock cycle does not need to be saved. For another example, in one clock cycle, the image signal processor 20 determines a 9×9 image area based on the first image for a first pixel. The 9×9 image area includes 5×5 designated pixels, and the first pixel can be processed according to the 5×5 designated pixels. In the next clock cycle, the image signal processor 20 uses the 5×5 designated pixels determined in the previous clock cycle to process the second pixel adjacent to the first pixel. Since a pair of adjacent pixels are processed within two clock cycles, the data of the previous clock cycle needs to be saved for use in the next clock cycle.

[0135] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions. For example, the first image involved in this application was obtained with full authorization.

[0136] The image signal processor in the above device calculates the base layer data and detail layer data for a pair of adjacent pixel points. Since a pair of adjacent pixel points is calculated only once, the amount of calculation is reduced, and resource usage is reduced, so that when the first image is subsequently processed according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points, the image processing efficiency can be improved.

[0137] Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application is provided. The method is executed by an image processing device and includes the following steps.

[0138] Step 901, acquiring a first image.

[0139] Step 902: determine base layer data and detail layer data for any pair of adjacent pixel points in the first image.

[0140] Step 903: Process the first image to obtain a second image according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points.

[0141] In a possible implementation, a pixel in the first image corresponds to a channel, and the first image corresponds to multiple channels; the base layer data and the detail layer data are determined for any pair of adjacent pixel points in the first image, including: determining multiple specified pixel points based on the first image and any pair of adjacent pixel points, the specified pixel points corresponding to specified channels in the multiple channels, and the specified channels are channels corresponding to each pair of adjacent pixel points; determining the base layer data and the detail layer data based on the multiple specified pixel points.

[0142] In a possible implementation manner, determining a plurality of designated pixel points based on the first image and any pair of adjacent pixel points includes:

[0143] Determine an image region centered around any pair of adjacent pixels in the first image;

[0144] A plurality of designated pixel points are determined based on the image area.

[0145] In a possible implementation, the first image includes pixel values ​​of each pixel point; and determining the base layer data and the detail layer data based on a plurality of designated pixel points includes:

[0146] Performing data conversion on pixel values ​​of multiple specified pixel points to obtain converted pixel values ​​of the multiple specified pixel points;

[0147] The base layer data and the detail layer data are determined based on the converted pixel values ​​of a plurality of designated pixel points.

[0148] In a possible implementation, determining base layer data and detail layer data based on pixel values ​​converted from a plurality of designated pixel points includes:

[0149] Determine statistical data based on the converted pixel values ​​of the plurality of designated pixel points, the statistical data representing fluctuations of the converted pixel values ​​of the plurality of designated pixel points;

[0150] Determine grassroots data based on statistical data;

[0151] Determine detail layer data based on base layer data.

[0152] In a possible implementation, determining the basic data based on the statistical data includes:

[0153] Determine the weight coefficient based on statistical data;

[0154] Determine the base layer data based on the converted pixel value, weight coefficient and statistical data of any specified pixel point.

[0155] In a possible implementation, any pair of adjacent pixels corresponds to a plurality of base layer data and a plurality of detail layer data; and determining the detail layer data based on the base layer data includes:

[0156] For any pair of adjacent pixels, the corresponding detail layer data is determined based on any corresponding base layer data and the converted pixel value of any designated pixel, or the corresponding detail layer data is determined based on at least two corresponding base layer data.

[0157] In a possible implementation, the first image includes pixel values ​​of each pixel, any pair of adjacent pixels includes a first pixel and a second pixel, the first pixel corresponds to a specified channel, and the second pixel corresponds to any channel of multiple channels except the specified channel; processing the first image to obtain the second image according to base layer data and detail layer data corresponding to multiple pairs of adjacent pixels includes:

[0158] For any pair of adjacent pixels, obtain the first brightness of the first pixel and the first brightness of the second pixel, and determine the reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel;

[0159] For any pixel in any pair of adjacent pixels, a pixel value of any pixel is processed according to a reference brightness and a first brightness of any pixel to obtain processed data of any pixel;

[0160] A second image is determined according to the processed data of each pixel.

[0161] In a possible implementation manner, determining a reference brightness according to base layer data, detail layer data, and a first brightness of a first pixel includes:

[0162] Determine the base layer brightness according to the base layer data and the first brightness of the first pixel;

[0163] Determine the reference brightness based on the base layer brightness and detail layer data.

[0164] The above method calculates the base layer data and detail layer data for a pair of adjacent pixel points. Since a pair of adjacent pixel points is calculated only once, the amount of calculation is reduced, and resource usage is reduced, so that when the first image is subsequently processed according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points, the image processing efficiency can be improved.

[0165] It should be understood that the above method embodiment and the device embodiment belong to the same concept, and the specific implementation process is detailed in the device embodiment, which will not be repeated here.

[0166] In an exemplary embodiment, a chip is provided, the chip comprising the above-mentioned image processing device. Optionally, the chip is a chip including an ISP.

[0167] In an exemplary embodiment, an electronic device is provided. The electronic device includes the above chip.

[0168] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0169] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0170] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing device, characterized in that: The device comprises a receiving interface and an image signal processor: The receiving interface is used to obtain a first image; The image signal processor is used to determine base layer data and detail layer data for any pair of adjacent pixel points in the first image; The image signal processor is further used to process the first image to obtain a second image according to the base layer data and detail layer data corresponding to multiple pairs of adjacent pixel points.

2. The device according to claim 1, characterized in that One pixel in the first image corresponds to one channel, and the first image corresponds to multiple channels; determining the base layer data and the detail layer data for any pair of adjacent pixels in the first image includes: Determine a plurality of designated pixels based on the first image and any pair of adjacent pixels, wherein the designated pixels correspond to designated channels among the plurality of channels, and the designated channels are channels corresponding to each pair of adjacent pixels; The base layer data and the detail layer data are determined based on the multiple designated pixel points.

3. The device according to claim 2, characterized in that The determining a plurality of designated pixel points based on the first image and any pair of adjacent pixel points comprises: Determine, in the first image, an image area centered around any pair of adjacent pixels; A plurality of designated pixel points are determined based on the image area.

4. The device according to claim 2, characterized in that The determining of the base layer data and the detail layer data based on the plurality of designated pixel points comprises: Performing data conversion on the pixel values ​​of the plurality of designated pixel points to obtain converted pixel values ​​of the plurality of designated pixel points; The base layer data and the detail layer data are determined based on the converted pixel values ​​of the plurality of designated pixel points.

5. The device according to claim 4, characterized in that The step of determining the base layer data and the detail layer data based on the pixel values ​​converted from the plurality of designated pixel points comprises: Determine statistical data based on the converted pixel values ​​of the plurality of designated pixel points, the statistical data representing fluctuations of the converted pixel values ​​of the plurality of designated pixel points; Determining grassroots data based on the statistical data; Detail layer data is determined based on the base layer data.

6. The device according to claim 5, characterized in that The determining of the basic data based on the statistical data comprises: Determining a weight coefficient according to the statistical data; The base layer data is determined according to the converted pixel value of any designated pixel point, the weight coefficient and the statistical data.

7. The device according to claim 5, characterized in that Any pair of adjacent pixel points corresponds to a plurality of base layer data and a plurality of detail layer data; and determining the detail layer data based on the base layer data comprises: For any pair of adjacent pixels, the corresponding detail layer data is determined based on any corresponding base layer data and the converted pixel value of any designated pixel, or the corresponding detail layer data is determined based on at least two corresponding base layer data.

8. The device according to any one of claims 2 to 7, characterized in that The first image includes pixel values ​​of each pixel, and any pair of adjacent pixels includes a first pixel and a second pixel, the first pixel corresponds to the designated channel, and the second pixel corresponds to any channel among the multiple channels except the designated channel; The step of processing the first image to obtain the second image according to the base layer data and detail layer data corresponding to the plurality of pairs of adjacent pixel points comprises: For any pair of adjacent pixels, obtain a first brightness of a first pixel and a first brightness of a second pixel, and determine a reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel; For any pixel in any pair of adjacent pixels, processing a pixel value of the any pixel according to the reference brightness and the first brightness of the any pixel to obtain processing data of the any pixel; A second image is determined according to the processed data of each pixel.

9. The device according to claim 8, characterized in that The determining the reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel includes: Determine a base layer brightness according to the base layer data and a first brightness of the first pixel; A reference brightness is determined according to the base layer brightness and the detail layer data.

10. An image processing method, characterized in that: The method comprises: acquiring a first image; Determine base layer data and detail layer data for any pair of adjacent pixel points in the first image; The first image is processed according to the base layer data and detail layer data corresponding to a plurality of pairs of adjacent pixel points to obtain a second image.

11. The method according to claim 10, characterized in that One pixel in the first image corresponds to one channel, and the first image corresponds to multiple channels; determining the base layer data and the detail layer data for any pair of adjacent pixels in the first image includes: Determine a plurality of designated pixels based on the first image and any pair of adjacent pixels, wherein the designated pixels correspond to designated channels among the plurality of channels, and the designated channels are channels corresponding to each pair of adjacent pixels; The base layer data and the detail layer data are determined based on the multiple designated pixel points.

12. The method according to claim 11, characterized in that The determining of the base layer data and the detail layer data based on the plurality of designated pixel points comprises: Performing data conversion on the pixel values ​​of the plurality of designated pixel points to obtain converted pixel values ​​of the plurality of designated pixel points; The base layer data and the detail layer data are determined based on the converted pixel values ​​of the plurality of designated pixel points.

13. The method according to claim 12, characterized in that The step of determining the base layer data and the detail layer data based on the pixel values ​​converted from the plurality of designated pixel points comprises: Determine statistical data based on the converted pixel values ​​of the plurality of designated pixel points, the statistical data representing fluctuations of the converted pixel values ​​of the plurality of designated pixel points; Determining grassroots data based on the statistical data; Detail layer data is determined based on the base layer data.

14. The method according to claim 13, characterized in that The determining of the basic data based on the statistical data comprises: Determining a weight coefficient according to the statistical data; The base layer data is determined according to the converted pixel value of any designated pixel point, the weight coefficient and the statistical data.

15. The method according to claim 13, characterized in that Any pair of adjacent pixel points corresponds to a plurality of base layer data and a plurality of detail layer data; and determining the detail layer data based on the base layer data comprises: For any pair of adjacent pixels, the corresponding detail layer data is determined based on any corresponding base layer data and the converted pixel value of any designated pixel, or the corresponding detail layer data is determined based on at least two corresponding base layer data.

16. The method according to any one of claims 11 to 15, characterized in that The first image includes pixel values ​​of each pixel, and any pair of adjacent pixels includes a first pixel and a second pixel, the first pixel corresponds to the designated channel, and the second pixel corresponds to any channel among the multiple channels except the designated channel; The step of processing the first image to obtain the second image according to the base layer data and detail layer data corresponding to the plurality of pairs of adjacent pixel points comprises: For any pair of adjacent pixels, obtain a first brightness of a first pixel and a first brightness of a second pixel, and determine a reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel; For any pixel in any pair of adjacent pixels, processing a pixel value of the any pixel according to the reference brightness and the first brightness of the any pixel to obtain processing data of the any pixel; A second image is determined according to the processed data of each pixel.

17. The method according to claim 16, characterized in that The determining the reference brightness according to the base layer data, the detail layer data and the first brightness of the first pixel includes: Determine a base layer brightness according to the base layer data and a first brightness of the first pixel; A reference brightness is determined according to the base layer brightness and the detail layer data.

18. A chip, characterized in that: The chip includes the image processing device as described in any one of claims 1 to 9.

19. An electronic device, characterized in that: The electronic device comprises the image processing device as described in any one of claims 1 to 9.

Images