ISP chip, image processing method and device and storage medium
By performing color channel interpolation and directional filtering through the de-mosaicing module in the ISP chip, the problems of increased hardware complexity and area of the ISP chip are solved, the functional reuse of de-mosaicing, noise reduction and sharpening is realized, and the hardware cost is reduced.
Patent Information
- Application Number
- CN202510946903.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-10
AI Technical Summary
The integration of multiple functional modules in existing ISP chips increases the complexity and area of the hardware structure, making it difficult to meet high-quality imaging requirements.
The de-mosaicing module in the ISP chip performs color channel interpolation and directional filtering to achieve the reuse of de-mosaicing, noise reduction, and sharpening functions, reducing the number of hardware modules.
The hardware complexity and area of the ISP chip are reduced, the hardware cost is reduced, and the functions of demosaicing, noise reduction and sharpening are realized.
Smart Images
Figure CN120765469A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of chip technology, and in particular to an ISP chip, an image processing method, a device, and a storage medium. Background Art
[0002] With the rapid development of image processing technology, image signal processor (ISP) chips need to integrate multiple functional modules, such as demosaicing, noise reduction, and sharpening, to meet the needs of high-quality imaging. During the image processing process, the demosaicing module, noise reduction module, and sharpening module operate independently. For example, demosaicing is performed first, followed by noise suppression (noise reduction), and finally edge detail enhancement (sharpening).
[0003] However, integrating multiple modules into the ISP chip greatly increases the hardware structure complexity and hardware area of the ISP chip. Summary of the Invention
[0004] The present invention provides an ISP chip, image processing method, device, and storage medium that can reuse hardware structures and reduce hardware area. The technical solution is as follows:
[0005] According to a first aspect of an embodiment of the present application, a demosaicing chip is provided, including:
[0006] The demosaicing module is used to: obtain an initial image and determine the edge direction of each initial pixel in the initial image; the initial pixel that lacks the first color channel is the first pixel to be interpolated;
[0007] performing interpolation of the first color channel on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the first intermediate image includes a plurality of first intermediate pixels; the first intermediate pixels that are missing the second color channel are second pixels to be interpolated;
[0008] performing second color channel interpolation on each of the second pixels to be interpolated based on the edge direction of each of the second pixels to be interpolated to obtain a second intermediate image; the second intermediate image includes a plurality of second intermediate pixels; the second intermediate pixels that are missing the third color channel are third pixels to be interpolated;
[0009] performing the third color channel interpolation on each of the third pixel points to be interpolated based on the edge direction of each of the third pixel points to be interpolated, to obtain a third intermediate image;
[0010] Performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value for each of the first intermediate pixel points;
[0011] The target filter value of each of the first intermediate pixels is superimposed on the third intermediate image to obtain a target image.
[0012] In a possible implementation, the first color channel is a green channel, the second color channel is a red channel, and the third color channel is a blue channel; or
[0013] The first color channel is the green channel, the second color channel is the blue channel, and the third color channel is the red channel.
[0014] In a possible implementation, acquiring an initial image and determining an edge direction of each initial pixel in the initial image includes:
[0015] For each initial pixel point, obtaining a first adjacent pixel point and a second adjacent pixel point; the first adjacent pixel point and the second adjacent pixel point are the initial pixel point adjacent to the initial pixel point in a first direction; performing a first filtering on the initial pixel point, the first adjacent pixel point, and the second adjacent pixel point to obtain a first filtered value of the initial pixel point;
[0016] Obtaining a third adjacent pixel point and a fourth adjacent pixel point, where the third adjacent pixel point and the fourth adjacent pixel point are the initial pixel point adjacent to the initial pixel point in the second direction; performing a second filtering on the initial pixel point, the third adjacent pixel point, and the fourth adjacent pixel point to obtain a second filtered value of the initial pixel point;
[0017] The edge direction of the initial pixel point is determined based on the first filter value and the second filter value.
[0018] In a possible implementation, the first filtering is high-pass filtering, and the second filtering is high-pass filtering.
[0019] In a possible implementation, determining the edge direction of the initial pixel based on the first filter value and the second filter value includes:
[0020] Calculating the first filter value, the pixel value of the first adjacent pixel point, and the pixel value of the second adjacent pixel point to obtain a first value;
[0021] Calculating the second filtered value, the pixel value of the third adjacent pixel point, and the pixel value of the fourth adjacent pixel point to obtain a second value;
[0022] The edge direction of the initial pixel is determined based on the first value and the second value.
[0023] In a possible implementation, performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value of each of the first intermediate pixel points includes:
[0024] For each first intermediate pixel point, when the edge direction matched with the first intermediate pixel point is the first direction, obtaining a first adjacent intermediate pixel point and a second adjacent intermediate pixel point, where the first adjacent intermediate pixel point and the second adjacent intermediate pixel point are the first intermediate pixel points adjacent to the first intermediate pixel point in the first direction;
[0025] performing a third filtering on the green channel value of the first middle pixel point, the green channel value of the first adjacent middle pixel point, and the green channel value of the second adjacent middle pixel point to obtain an intermediate filtered value;
[0026] Acquire a third adjacent middle pixel point and a fourth adjacent middle pixel point, where the first adjacent middle pixel point and the second adjacent middle pixel point are the first middle pixel points adjacent to the first middle pixel point in the second direction;
[0027] A fourth filtering is performed on the intermediate filtered value, the green channel value of the third adjacent intermediate pixel point, and the green channel value of the fourth adjacent intermediate pixel point to obtain the target filtered value of the first intermediate pixel point.
[0028] In a possible implementation, the third filtering is a low-pass filtering, and the fourth filtering is a high-pass filtering.
[0029] According to a second aspect of an embodiment of the present application, there is provided an image processing method, comprising:
[0030] Acquire an initial image, and determine the edge direction of each initial pixel in the initial image; the initial pixel that is missing the first color channel is the first pixel to be interpolated;
[0031] performing interpolation of the first color channel on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the first intermediate image includes a plurality of first intermediate pixels; the first intermediate pixels that are missing the second color channel are second pixels to be interpolated;
[0032] performing second color channel interpolation on each of the second pixels to be interpolated based on the edge direction of each of the second pixels to be interpolated to obtain a second intermediate image; the second intermediate image includes a plurality of second intermediate pixels; the second intermediate pixels that are missing the third color channel are third pixels to be interpolated;
[0033] performing the third color channel interpolation on each of the third pixel points to be interpolated based on the edge direction of each of the third pixel points to be interpolated, to obtain a third intermediate image;
[0034] Performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value for each of the first intermediate pixel points;
[0035] The target filter value of each of the first intermediate pixels is superimposed on the third intermediate image to obtain a target image.
[0036] According to a third aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the image processing method.
[0037] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the image processing method.
[0038] In an embodiment of the present application, a de-mosaicing module in an ISP chip performs first color channel interpolation on each first pixel to be interpolated in the initial image to obtain a first intermediate image; performs second color channel interpolation on each second pixel to be interpolated in the first intermediate image to obtain a second intermediate image; performs third color channel interpolation on each third pixel to be interpolated in the second intermediate image to obtain a third intermediate image; performs directional filtering on each first intermediate pixel in the first intermediate image to obtain a target filter value for each first intermediate pixel; and superimposes the target filter value of each first intermediate pixel with the third intermediate image to obtain a target image. The technical solution of the present application not only implements de-mosaicing through the de-mosaicing module, but also implements noise reduction and sharpening, thereby reusing the de-mosaicing module and eliminating the need to integrate a noise reduction module and a sharpening module in the ISP chip, thereby greatly reducing the hardware complexity of the ISP chip, reducing the area of the ISP chip, and reducing the hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0040] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;
[0041] Figure 2 This is a schematic diagram of the structure of an ISP chip provided according to an embodiment of the present application;
[0042] Figure 3 is a structural diagram of a demosaicing module provided according to an embodiment of the present application;
[0043] Figure 4 2 is a schematic diagram of the principle of a demosaicing module provided according to an embodiment of the present application;
[0044] Figure 5 This is a schematic diagram of a principle for determining edge direction according to an embodiment of the present application;
[0045] Figure 6 is a flowchart of an image processing method provided according to an embodiment of the present application;
[0046] Figure 7 is a schematic structural diagram of a terminal provided according to an embodiment of the present application;
[0047] Figure 8 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0048] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0049] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.
[0050] In this application, the terms "first," "second," and the like are used to distinguish identical or similar items having substantially the same role and function. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms.
[0051] These terms are simply used to distinguish one element from another. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.
[0052] Here, at least one refers to one or more than one. For example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple refers to two or more than two. For example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.
[0053] Figure 1 10 is a schematic diagram of an implementation environment provided according to an embodiment of the present application, which may include a terminal 101 and a server 102.
[0054] Each terminal 101 is equipped with an ISP chip. A de-mosaicing module is integrated into the ISP chip. For example, the terminal 101 may be a smartphone, wearable device, personal computer, laptop, tablet computer, smart TV, or in-vehicle terminal equipped with an ISP chip.
[0055] The server 102 may be a single server, a server cluster consisting of multiple servers, or a cloud processing center.
[0056] The terminal 101 is connected to the server 102 via a wired or wireless network.
[0057] In some embodiments, the wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.
[0058] In related technologies, different functional modules in chip or system design require independent hardware configurations to meet performance requirements. For example, an ISP chip integrates multiple functional modules, each requiring independent cache, registers, and circuits. Therefore, integrating a demosaicing module, a noise reduction module, and a sharpening module into an ISP chip significantly increases the hardware complexity and area of the ISP chip.
[0059] In order to solve the above technical problems, an embodiment of the present application provides an ISP chip. Compared with the ISP chip in the related technology, the ISP chip reduces the number of internal integrated modules while being able to de-mosaic, denoise and sharpen the image, thereby reducing the complexity of the hardware structure and the hardware area.
[0060] Figure 2 2 is a schematic diagram of the structure of an ISP chip 200 provided according to an embodiment of the present application. The ISP chip 200 includes a demosaicing module 201. A high-pass filter and a low-pass filter are provided in the demosaicing module 201. Noise reduction can be achieved through the low-pass filter, and sharpening can be achieved through the high-pass filter. Therefore, the high-pass filter in the demosaicing module 201 can be reused for sharpening, and the low-pass filter in the demosaicing module 201 can be reused for noise reduction. This not only achieves hardware structure reuse, but also eliminates the need to integrate the noise reduction module and the sharpening module in the ISP chip 200, thereby greatly reducing hardware complexity and hardware area.
[0061] Figure 3 2 is a schematic diagram of the structure of a demosaicing module 201 provided according to an embodiment of the present application. The demosaicing module 201 includes a direction unit 2011, an interpolation unit 2012, a correction unit 2013, and an overlay unit 2014. The direction unit 2011 is used to determine edge direction, the interpolation unit 2012 is used for demosaicing, the correction unit 2013 is used for noise reduction and sharpening, and the overlay unit 2014 is used to obtain a target image. It should be noted that the direction unit 2011, the interpolation unit 2012, the correction unit 2013, and the overlay unit 2014 can be a single physical unit or multiple different units.
[0062] Figure 4 This is a schematic diagram of the principle of a demosaicing module provided according to an embodiment of the present application.
[0063] The following combination Figure 4 , the functions of the direction unit, interpolation unit, correction unit and superposition unit are exemplarily explained.
[0064] In some embodiments, in order to improve the accuracy of demosaicing and facilitate interpolation based on edge direction, in an embodiment of the present application, a direction unit obtains an initial image and determines the edge direction of each initial pixel in the initial image.
[0065] Figure 5 This is a schematic diagram of the principle of determining edge direction provided according to an embodiment of the present application.
[0066] The following combination Figure 5 right Figure 4 The determination of the edge direction in is exemplified.
[0067] In some examples, in order to determine the edge direction of each initial pixel point, an embodiment of the present application filters each initial pixel point in different directions to obtain a first filtering value and a second filtering value of each initial pixel point; based on the first filtering value and the second filtering value of each initial pixel point, the edge direction of each initial pixel point is determined.
[0068] Exemplarily, in order to obtain information of each initial pixel point in the first direction, in an embodiment of the present application, for each initial pixel point, the direction unit obtains a first adjacent pixel point and a second adjacent pixel point; the first adjacent pixel point and the second adjacent pixel point are initial pixel points adjacent to the initial pixel point in the first direction; the initial pixel point, the first adjacent pixel point and the second adjacent pixel point are subjected to a first filtering to obtain a first filtered value of the initial pixel point.
[0069] Exemplarily, in order to obtain the first adjacent pixel point and the second adjacent pixel point corresponding to the initial pixel point, in an embodiment of the present application, each initial pixel point corresponds to a unique preset coordinate in the coordinate system where the initial image is located, and the direction unit obtains the initial pixel point, the first adjacent pixel point and the second adjacent pixel point based on the preset coordinate.
[0070] For example, since the high-frequency information includes a sudden change signal for determining the edge direction, in order to determine the edge direction of each initial pixel point, it is necessary to obtain the high-frequency information of each initial pixel point in the first direction and the high-frequency information of each initial pixel point in the second direction. To obtain the high-frequency information of each initial pixel point in the first direction, an embodiment of the present application, for each initial pixel point, high-pass filtering is performed on the initial pixel point, the first adjacent pixel point, and the second adjacent pixel point based on a high-pass filter to obtain a first filtered value of the initial pixel point.
[0071] For example, the initial image is a RAW image. The arrangement of R, G, and B in the RAW image is RGGB. R represents red, G represents green, and B represents blue. For another example, the first direction is a direction parallel to the x-axis direction in the coordinate system where the RAW image is located. It can be understood that each initial pixel in the RAW image has only one known color channel. The arrangement of R, G, and B in the RAW image is determined based on the filter distribution rule of the Bayer array. The 2×2 pixel unit of the Bayer array includes two green filters, one red filter, and one blue filter.
[0072] In some examples, in order to obtain information of each initial pixel point in the second direction, in an embodiment of the present application, for each initial pixel point, the direction unit obtains the third adjacent pixel point and the fourth adjacent pixel point, where the third adjacent pixel point and the fourth adjacent pixel point are the initial pixel points adjacent to the initial pixel point in the second direction; the initial pixel point, the third adjacent pixel point, and the fourth adjacent pixel point are subjected to a second filtering to obtain a second filtering value of the initial pixel point; and the edge direction of the initial pixel point is determined based on the first filtering value and the second filtering value.
[0073] Exemplarily, in order to obtain the high-frequency information of each initial pixel point in the second direction, an embodiment of the present application, for each initial pixel point, high-pass filtering is performed on the initial pixel point, the third adjacent pixel point, and the fourth adjacent pixel point based on a high-pass filter to obtain the second filtered value of the initial pixel point.
[0074] Exemplarily, in order to determine the edge direction of each initial pixel point based on the first filter value and the second filter value of each initial pixel point, an embodiment of the present application, for each initial pixel point, calculates the first filter value, the pixel value of the first adjacent pixel point, and the pixel value of the second adjacent pixel point to obtain a first value; calculates the second filter value, the pixel value of the third adjacent pixel point, and the pixel value of the fourth adjacent pixel point to obtain a second value; and determines the edge direction of the initial pixel point based on the first value and the second value.
[0075] For example, in order to obtain the first filter value and the second filter value, the embodiment of the present application sets a first preset coefficient, a second preset coefficient, and a third preset coefficient, wherein the first preset coefficient, the second preset coefficient, and the third preset coefficient can be determined according to actual needs.
[0076] For example, the first filter value and the second filter value are obtained by the following formula:
[0077] The first filter value=the pixel value of the first adjacent pixel point×the first preset coefficient+the first filter value×the second preset coefficient+the pixel value of the second adjacent pixel point×the third preset coefficient.
[0078] The second filter value=the pixel value of the third adjacent pixel point×the first preset coefficient+the second filter value×the second preset coefficient+the pixel value of the fourth adjacent pixel point×the third preset coefficient.
[0079] For another example, the second direction is a direction parallel to the y-axis direction in the coordinate system where the RAW image is located.
[0080] Exemplarily, in order to determine the edge direction by the first value and the second value, so as to facilitate accurate interpolation. In the embodiment of the present application, for each initial pixel point, the first filter value, the pixel value of the first adjacent pixel point and the pixel value of the second adjacent pixel point are differentially calculated to obtain the first value; the second filter value, the pixel value of the third adjacent pixel point and the pixel value of the fourth adjacent pixel point are differentially calculated to obtain the second value. Compare the first value and the second value. If the first value is greater than the second value, the second direction is taken as the edge direction; or if the first value is less than the second value, the first direction is taken as the edge direction. It should be noted that the method of differential calculation can be directly obtained from the relevant technology, and the embodiments of the present application will not be repeated.
[0081] In some embodiments, in order to demosaic the initial image, embodiments of the present application interpolate the initial image based on edge directions to obtain a color image.
[0082] In some examples, the interpolation unit performs first color channel interpolation on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the first intermediate image includes multiple first intermediate pixels; the first intermediate pixel that is missing the second color channel is the second pixel to be interpolated.
[0083] In some examples, the interpolation unit performs second color channel interpolation on each second pixel to be interpolated based on an edge direction of each second pixel to be interpolated to obtain a second intermediate image; the second intermediate image includes multiple second intermediate pixels; the second intermediate pixels that are missing the third color channel are third pixels to be interpolated. Based on the edge direction of each third pixel to be interpolated, the interpolation unit performs third color channel interpolation on each third pixel to be interpolated to obtain a third intermediate image. The third intermediate image is a color image, such as an RGB image.
[0084] It's understandable that, because the R, G, and B colors in a RAW image are arranged in the RGGB format, the raw image contains the largest number of green channel pixels, which also contain the most detail and high-frequency information. Therefore, interpolating the green channel first creates a more accurate first intermediate image, providing a more precise luminance basis.
[0085] In some examples, the first color channel is a green channel, the second color channel is a red channel, and the third color channel is a blue channel.
[0086] Exemplarily, the interpolation unit performs green channel interpolation on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the initial pixel that is missing the green channel is the first pixel to be interpolated, and the first intermediate pixel that is missing the red channel is the second pixel to be interpolated. Based on the edge direction of each second pixel to be interpolated, the red channel is interpolated on each second pixel to be interpolated to obtain a second intermediate image; the second intermediate pixel that is missing the blue channel is the third pixel to be interpolated. Based on the edge direction of each third pixel to be interpolated, the blue channel is interpolated on each third pixel to be interpolated to obtain a third intermediate image.
[0087] In some examples, the first color channel is a green channel, the second color channel is a blue channel, and the third color channel is a red channel.
[0088] Exemplarily, the interpolation unit performs green channel interpolation on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the initial pixel that is missing the green channel is the first pixel to be interpolated, and the first intermediate pixel that is missing the blue channel is the second pixel to be interpolated. Based on the edge direction of each second pixel to be interpolated, the blue channel interpolation is performed on each second pixel to be interpolated to obtain a second intermediate image; the second intermediate pixel that is missing the red channel is the third pixel to be interpolated. Based on the edge direction of each third pixel to be interpolated, the red channel interpolation is performed on each third pixel to be interpolated to obtain a third intermediate image.
[0089] In some embodiments, since the human eye is most sensitive to green and rich information can be obtained through green, the embodiments of the present application only perform noise reduction and sharpening on the green channel of each first intermediate pixel point in the first intermediate image. In order to achieve noise reduction and sharpening of the green channel of each first intermediate pixel point in the first intermediate image to obtain the target image, the correction unit performs directional filtering on each first intermediate pixel point to obtain the target filter value of each first intermediate pixel point. It should be noted that after obtaining the third intermediate image, the correction unit performs directional filtering on each first intermediate pixel point to obtain the target filter value of each first intermediate pixel point is only one of the implementation methods. In addition, before obtaining the third intermediate image, the correction unit can also perform directional filtering on each first intermediate pixel point to obtain the target filter value of each first intermediate pixel point.
[0090] In some examples, since much noise exhibits high-frequency characteristics in the horizontal direction, and to avoid amplifying noise during the sharpening process, it is necessary to first reduce noise in the horizontal direction before sharpening. To achieve horizontal noise reduction in the first intermediate image, in an embodiment of the present application, for each first intermediate pixel, when the edge direction matching the first intermediate pixel is horizontal, a first adjacent intermediate pixel and a second adjacent intermediate pixel are obtained. The first adjacent intermediate pixel and the second adjacent intermediate pixel are first intermediate pixels horizontally adjacent to the first intermediate pixel. A third filtering is performed on the green channel value of the first intermediate pixel, the green channel value of the first adjacent intermediate pixel, and the green channel value of the second adjacent intermediate pixel to obtain an intermediate filtered value. Since each initial pixel corresponds to a unique edge direction, the initial pixel and the first intermediate pixel have a one-to-one correspondence. The edge direction corresponding to the initial pixel is used as the edge direction of the first intermediate pixel corresponding to the initial pixel. In other words, each first intermediate pixel has a unique matching edge direction. By reducing noise in the horizontal direction, noise can be effectively smoothed.
[0091] Exemplarily, for each first intermediate pixel point, the green channel value of the first intermediate pixel point, the green channel value of the first adjacent intermediate pixel point, and the green channel value of the second adjacent intermediate pixel point are low-pass filtered based on a low-pass filter to obtain an intermediate filtered value.
[0092] In some examples, in order to sharpen the first intermediate image, an embodiment of the present application obtains a third adjacent intermediate pixel point and a fourth adjacent intermediate pixel point, where the first adjacent intermediate pixel point and the second adjacent intermediate pixel point are the first intermediate pixel points adjacent to the first intermediate pixel point in the second direction; the intermediate filter value, the green channel value of the third adjacent intermediate pixel point, and the green channel value of the fourth adjacent intermediate pixel point are subjected to a fourth filter to obtain a target filter value of the first intermediate pixel point.
[0093] For example, for each first intermediate pixel, a high-pass filter is performed on the intermediate filter value, the green channel value of the third adjacent intermediate pixel, and the green channel value of the fourth adjacent intermediate pixel using a high-pass filter to obtain a target filter value for the first intermediate pixel. The target filter value is high-frequency detail information of the green channel of the corresponding first intermediate pixel; the high-frequency detail information may include edge information and texture information, etc.
[0094] In some embodiments, in order to add edge information and texture information to the third intermediate image, thereby making the third intermediate image clearer, the superposition unit superimposes the target filter value of each first intermediate pixel point with the third intermediate image to obtain a target image.
[0095] In some examples, for each target filter value of the first intermediate pixel point, the target filter value is superimposed on the pixel value of the corresponding third intermediate pixel point to obtain the target pixel point.
[0096] Exemplarily, for the target filter value of each first intermediate pixel point, the target filter value is superimposed with the green channel of the corresponding third intermediate pixel point to obtain the target green channel; the target filter value is superimposed with the red channel of the corresponding third intermediate pixel point to obtain the target red channel; the target filter value is superimposed with the blue channel of the corresponding third intermediate pixel point to obtain the target blue channel.
[0097] For example, the target green channel, target red channel, and target blue channel are obtained by the following formula:
[0098] Target green channel = a × target filter value + the green channel of the corresponding third middle pixel;
[0099] Target red channel = b × target filter value + the red channel of the corresponding third middle pixel;
[0100] Target blue channel=c×target filter value+corresponding blue channel of the third middle pixel; wherein a, b, and c can be determined according to actual needs.
[0101] In an embodiment of the present application, a de-mosaicing module in an ISP chip performs first color channel interpolation on each first pixel to be interpolated in an initial image to obtain a first intermediate image; performs second color channel interpolation on each second pixel to be interpolated in the first intermediate image to obtain a second intermediate image; performs third color channel interpolation on each third pixel to be interpolated in the second intermediate image to obtain a third intermediate image; performs directional filtering on each first intermediate pixel in the first intermediate image to obtain a target filter value for each first intermediate pixel; and superimposes the target filter value of each first intermediate pixel with the third intermediate image to obtain a target image. The technical solution of the present application not only implements de-mosaicing through the de-mosaicing module, but also implements noise reduction and sharpening, thereby reusing the de-mosaicing module and eliminating the need to integrate a noise reduction module and a sharpening module in the ISP chip, thereby greatly reducing the hardware complexity of the ISP chip, reducing the area of the ISP chip, and reducing the hardware cost.
[0102] Figure 6 is a flow chart of an image processing method provided according to an embodiment of the present application, such as Figure 6 As shown, in the embodiment of the present application, the method is described by taking the application on a terminal with an ISP chip having an integrated demosaicing module as an example. The method includes the following steps:
[0103] In step S601, the demosaicing module obtains an initial image and determines the edge direction of each initial pixel in the initial image; the initial pixel that is missing the first color channel is the first pixel to be interpolated.
[0104] In step S602 , the demosaicing module performs first color channel interpolation on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image.
[0105] The first intermediate image includes a plurality of first intermediate pixels; the first intermediate pixel that is missing the second color channel is the second pixel to be interpolated;
[0106] In step S603 , the demosaicing module performs second color channel interpolation on each second pixel to be interpolated based on the edge direction of each second pixel to be interpolated to obtain a second intermediate image.
[0107] The second intermediate image includes a plurality of second intermediate pixel points; the second intermediate pixel point that lacks the third color channel is the third pixel point to be interpolated.
[0108] In step S604 , the demosaicing module performs third color channel interpolation on each third pixel to be interpolated based on the edge direction of each third pixel to be interpolated to obtain a third intermediate image.
[0109] In step S605 , the demosaicing module performs directional filtering on each first intermediate pixel point to obtain a target filtering value of each first intermediate pixel point.
[0110] In step S606 , the demosaicing module superimposes the target filter value of each first intermediate pixel with the third intermediate image to obtain a target image.
[0111] In some embodiments, the first color channel is a green channel, the second color channel is a red channel, and the third color channel is a blue channel; or
[0112] The first color channel is the green channel, the second color channel is the blue channel, and the third color channel is the red channel.
[0113] In some embodiments, obtaining an initial image and determining an edge direction of each initial pixel in the initial image includes:
[0114] For each initial pixel point, obtaining a first adjacent pixel point and a second adjacent pixel point; the first adjacent pixel point and the second adjacent pixel point are initial pixel points adjacent to the initial pixel point in a first direction; performing a first filtering on the initial pixel point, the first adjacent pixel point, and the second adjacent pixel point to obtain a first filtered value of the initial pixel point;
[0115] Obtaining a third adjacent pixel point and a fourth adjacent pixel point, where the third adjacent pixel point and the fourth adjacent pixel point are initial pixel points adjacent to the initial pixel point in the second direction; performing a second filtering on the initial pixel point, the first adjacent pixel point, and the second adjacent pixel point to obtain a second filtered value of the initial pixel point;
[0116] Based on the first filter value and the second filter value, an edge direction of the initial pixel point is determined.
[0117] In some embodiments, the first filtering is a high-pass filtering, and the second filtering is a high-pass filtering.
[0118] In some embodiments, determining the edge direction of the initial pixel based on the first filter value and the second filter value includes:
[0119] Calculating a first filter value, a pixel value of a first adjacent pixel point, and a pixel value of a second adjacent pixel point to obtain a first value;
[0120] Calculating the second filtered value, the pixel value of the third adjacent pixel point, and the pixel value of the fourth adjacent pixel point to obtain a second value;
[0121] Based on the first value and the second value, an edge direction of the initial pixel point is determined.
[0122] In some embodiments, performing directional filtering on each first intermediate pixel point to obtain a target filtering value for each first intermediate pixel point includes:
[0123] For each first intermediate pixel point, when the edge direction matched with the first intermediate pixel point is the first direction, obtaining a first adjacent intermediate pixel point and a second adjacent intermediate pixel point, where the first adjacent intermediate pixel point and the second adjacent intermediate pixel point are first intermediate pixel points adjacent to the first intermediate pixel point in the first direction;
[0124] Performing a third filtering on the green channel value of the first middle pixel, the green channel value of the first adjacent middle pixel, and the green channel value of the second adjacent middle pixel to obtain an intermediate filtered value;
[0125] Acquire a third adjacent middle pixel point and a fourth adjacent middle pixel point, where the first adjacent middle pixel point and the second adjacent middle pixel point are first middle pixel points adjacent to the first middle pixel point in the second direction;
[0126] A fourth filtering is performed on the middle filtered value, the green channel value of the third adjacent middle pixel point, and the green channel value of the fourth adjacent middle pixel point to obtain a target filtered value of the first middle pixel point.
[0127] In some embodiments, the third filtering is a low-pass filtering, and the fourth filtering is a high-pass filtering.
[0128] In addition, the image processing method provided in the above embodiment and the ISP chip embodiment have the same concept. The specific implementation process is detailed in the device embodiment and will not be repeated here.
[0129] In an embodiment of the present application, a de-mosaicing module in an ISP chip performs first color channel interpolation on each first pixel to be interpolated in the initial image to obtain a first intermediate image; performs second color channel interpolation on each second pixel to be interpolated in the first intermediate image to obtain a second intermediate image; performs third color channel interpolation on each third pixel to be interpolated in the second intermediate image to obtain a third intermediate image; performs directional filtering on each first intermediate pixel in the first intermediate image to obtain a target filter value for each first intermediate pixel; and superimposes the target filter value of each first intermediate pixel with the third intermediate image to obtain a target image. The technical solution of the present application not only realizes de-mosaicing through the de-mosaicing module, but also realizes noise reduction and sharpening, thereby reusing the de-mosaicing module and eliminating the need to integrate a noise reduction module and a sharpening module in the ISP chip, thereby greatly reducing the hardware complexity of the ISP chip, reducing the area of the ISP chip, and reducing the hardware cost.
[0130] An embodiment of the present application further provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the above method when executing the computer program.
[0131] Taking computer equipment as the terminal as an example, Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application, see Figure 7 Terminal 700 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 700 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.
[0132] Typically, the terminal 700 includes a processor 701 and a memory 702 .
[0133] The processor 701 may include one or more processing cores, such as a 4-core processor, a 5-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0134] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the process for terminal execution in the above method provided in the method embodiment of the present application.
[0135] In some embodiments, terminal 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. Processor 701, memory 702, and peripheral device interface 703 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 703 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a display screen 704, a camera assembly 705, an audio circuit 706, and a power supply 707.
[0136] The peripheral interface 703 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702 and the peripheral interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702 and the peripheral interface 703 can be implemented on a separate chip or circuit board, and the embodiments of the present application are not limited in this regard.
[0137] The display screen 704 is used to display a UI (User Interface). The UI can include graphics, text, icons, video and any combination thereof. When the display screen 704 is a touch display screen, the display screen 704 also has the ability to collect touch signals on or above the surface of the display screen 704. The touch signals can be input as control signals to the processor 701 for processing. At this time, the display screen 704 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 704 can be one, arranged on the front panel of the terminal 700; in some other embodiments, the display screen 704 can be at least two, arranged on different surfaces of the terminal 700 or in a folding design; in some other embodiments, the display screen 704 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 700. Even, the display screen 704 can also be arranged in an irregular shape other than a rectangle, i.e. a special-shaped screen. The display screen 704 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0138] The camera assembly 705 is used to capture images or videos. In some embodiments, the camera assembly 705 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera and a long-focus camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 705 can also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash is a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.
[0139] The audio circuit 706 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 701 for processing. The microphone can be multiple for the purpose of stereo sound collection or noise reduction, and arranged at different positions of the terminal 700 respectively. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert an electrical signal from the processor 701 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, the electrical signal can be converted into not only sound waves audible to human, but also sound waves inaudible to human for ranging and other purposes. In some embodiments, the audio circuit 706 can further include a headphone jack.
[0140] The power supply 707 is used to supply power to various components in the terminal 700. The power supply 707 can be alternating current, direct current, disposable batteries or rechargeable batteries. When the power supply 707 includes rechargeable batteries, the rechargeable batteries can support wired charging or wireless charging. The rechargeable batteries can also be used to support fast charging technology.
[0141] Those skilled in the art can understand that the structure shown in FIG. 8 is not a limitation on the terminal 700, and the terminal 700 can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements. Figure 7 Those skilled in the art can understand that the structure shown in FIG. 8 is not a limitation on the terminal 700, and the terminal 700 can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements.
[0142] Taking a computer device as an example, Figure 8 FIG. 8 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 800 can be quite different in configuration or performance, and can include one or more processors (Central Processing Units, CPU) 801 and one or more memories 802. The one or more memories 802 store at least one computer program, which is loaded and executed by the one or more processors 801 to implement the image processing method described above. Of course, the server 800 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, which are not described here in detail.
[0143] An embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium is controlled to execute the above method. Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.
[0144] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.
[0145] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. An ISP chip, characterized in that: include: Demosaicing module for: Acquire an initial image, and determine the edge direction of each initial pixel in the initial image; the initial pixel that is missing the first color channel is the first pixel to be interpolated; performing interpolation of the first color channel on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the first intermediate image includes a plurality of first intermediate pixels; the first intermediate pixels that are missing the second color channel are second pixels to be interpolated; performing second color channel interpolation on each of the second pixels to be interpolated based on the edge direction of each of the second pixels to be interpolated to obtain a second intermediate image; the second intermediate image includes a plurality of second intermediate pixels; the second intermediate pixels that are missing the third color channel are third pixels to be interpolated; performing the third color channel interpolation on each of the third pixel points to be interpolated based on the edge direction of each of the third pixel points to be interpolated, to obtain a third intermediate image; Performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value for each of the first intermediate pixel points; The target filter value of each of the first intermediate pixels is superimposed on the third intermediate image to obtain a target image.
2. The chip according to claim 1, characterized in that The first color channel is a green channel, the second color channel is a red channel, and the third color channel is a blue channel; or The first color channel is the green channel, the second color channel is the blue channel, and the third color channel is the red channel.
3. The chip according to claim 1, characterized in that The acquiring of the initial image and determining the edge direction of each initial pixel in the initial image includes: For each initial pixel point, obtaining a first adjacent pixel point and a second adjacent pixel point; the first adjacent pixel point and the second adjacent pixel point are the initial pixel point adjacent to the initial pixel point in a first direction; performing a first filtering on the initial pixel point, the first adjacent pixel point, and the second adjacent pixel point to obtain a first filtered value of the initial pixel point; Obtaining a third adjacent pixel point and a fourth adjacent pixel point, where the third adjacent pixel point and the fourth adjacent pixel point are the initial pixel point adjacent to the initial pixel point in the second direction; performing a second filtering on the initial pixel point, the third adjacent pixel point, and the fourth adjacent pixel point to obtain a second filtered value of the initial pixel point; The edge direction of the initial pixel point is determined based on the first filter value and the second filter value.
4. The chip according to claim 3, characterized in that The first filtering is a high-pass filtering, and the second filtering is a high-pass filtering.
5. The chip according to claim 3, characterized in that The determining the edge direction of the initial pixel point based on the first filtering value and the second filtering value includes: Calculating the first filter value, the pixel value of the first adjacent pixel point, and the pixel value of the second adjacent pixel point to obtain a first value; Calculating the second filtered value, the pixel value of the third adjacent pixel point, and the pixel value of the fourth adjacent pixel point to obtain a second value; The edge direction of the initial pixel is determined based on the first value and the second value.
6. The chip according to claim 1, characterized in that The performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value of each of the first intermediate pixel points includes: For each first intermediate pixel point, when the edge direction matched with the first intermediate pixel point is the first direction, obtaining a first adjacent intermediate pixel point and a second adjacent intermediate pixel point, where the first adjacent intermediate pixel point and the second adjacent intermediate pixel point are the first intermediate pixel points adjacent to the first intermediate pixel point in the first direction; performing a third filtering on the green channel value of the first middle pixel point, the green channel value of the first adjacent middle pixel point, and the green channel value of the second adjacent middle pixel point to obtain an intermediate filtered value; Acquire a third adjacent middle pixel point and a fourth adjacent middle pixel point, where the first adjacent middle pixel point and the second adjacent middle pixel point are the first middle pixel points adjacent to the first middle pixel point in the second direction; A fourth filtering is performed on the intermediate filtered value, the green channel value of the third adjacent intermediate pixel point, and the green channel value of the fourth adjacent intermediate pixel point to obtain the target filtered value of the first intermediate pixel point.
7. The chip according to claim 6, characterized in that The third filtering is a low-pass filtering, and the fourth filtering is a high-pass filtering.
8. An image processing method, characterized in that: include: Acquire an initial image, and determine the edge direction of each initial pixel in the initial image; the initial pixel that is missing the first color channel is the first pixel to be interpolated; performing interpolation of the first color channel on each first pixel to be interpolated based on the edge direction of each first pixel to be interpolated to obtain a first intermediate image; the first intermediate image includes a plurality of first intermediate pixels; the first intermediate pixels that are missing the second color channel are second pixels to be interpolated; performing second color channel interpolation on each of the second pixels to be interpolated based on the edge direction of each of the second pixels to be interpolated to obtain a second intermediate image; the second intermediate image includes a plurality of second intermediate pixels; the second intermediate pixels that are missing the third color channel are third pixels to be interpolated; performing the third color channel interpolation on each of the third pixel points to be interpolated based on the edge direction of each of the third pixel points to be interpolated, to obtain a third intermediate image; Performing directional filtering on each of the first intermediate pixel points to obtain a target filtering value for each of the first intermediate pixel points; The target filter value of each of the first intermediate pixels is superimposed on the third intermediate image to obtain a target image.
9. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one program, and the at least one program is loaded by the processor to execute the image processing method according to claim 8.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the image processing method according to claim 8.