A RAW domain-based image sharpening method, system and device

Abstract

The application provides a RAW domain-based image sharpening method, system and device, the method comprising the following steps: S1: obtaining a RAW image through sensor acquisition; S2: performing multi-frame fusion denoising on the acquired image and extracting a random noise image; S3: performing channel separation according to an actual Bayer format; S4: extracting high-frequency components from each channel of the multi-frame fusion denoised image through Gaussian filtering; S5: processing the high-frequency component image and the random noise image and superimposing them on the multi-frame fusion denoised image to obtain a sharpening result of each channel; and S6: combining the RAW image and outputting the image after filtering each channel. The image sharpening is performed in the RAW domain, weak texture is enhanced, and image precision loss caused by data compression is avoided. The image sharpening is performed in the RAW domain, the intensity of texture details and the fineness of texture details are controlled, noise-related characteristics caused by related operations such as image interpolation in the back end are avoided, noise is structural and some false textures exist.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to an image sharpening method, system and apparatus based on the RAW domain. Background Technology

[0002] In existing technologies, image sharpening, as an image processing technique, aims to enhance the edges and details of an image, making it appear clearer and more vivid. The core idea of ​​sharpening is to highlight the high-frequency components of an image (such as edges and textures) while minimizing the impact on low-frequency components (such as smooth areas); it has important applications in fields such as photography, medical imaging, computer vision, and satellite remote sensing. Multi-frame fusion denoising, on the other hand, reduces random noise and preserves texture details by fusing multiple RAW images.

[0003] Early image sharpening was mainly based on spatial filtering, such as the Laplacian operator and unsharpening masks; later, with the introduction of Fourier transform and wavelet transform, it was possible to enhance details and textures in the frequency domain. In addition, there are methods based on multi-scale analysis and machine learning.

[0004] The Laplacian operator is used to directly extract and enhance the high-frequency components of an image, and then superimpose them back onto the original image. It can directly extract high-frequency components, but it is relatively sensitive to noise.

[0005] The Unsharp Masking (USM) method involves subtracting a Gaussian blur from the original image to obtain high-frequency components, enhancing them, and then superimposing them back onto the original image. This method is simple and easy to implement, but it can easily amplify noise and produce a halo effect.

[0006] However, the existing technology has the following drawbacks:

[0007] 1. Existing sharpening methods are often designed for interpolated brightness domain images. The original data is usually compressed, and relevant details and weak textures are lost. Subsequent image sharpening cannot effectively enhance weak textures.

[0008] 2. Furthermore, after interpolation, the noise in the brightness domain image is often structural and contains some erroneous textures. After a series of linear and nonlinear processing by back-end modules, the noise-related characteristics are destroyed, the noise model becomes more complex, and the difficulty of image sharpening is increased. Summary of the Invention

[0009] To address the aforementioned problems, the purpose of this application is to provide an image sharpening method, system, and apparatus based on the RAW domain, thereby overcoming the shortcomings of the prior art. It mainly includes:

[0010] 1. Sharpening images in the RAW domain can avoid the fact that brightness domain sharpening cannot effectively enhance weak textures. This is because operations in the brightness domain will cause data compression and loss of image precision.

[0011] 2. Operations in the luminance domain, such as interpolation, often result in structured noise that contains some erroneous textures; the noise model is also more complex, and luminance domain sharpening is less effective at distinguishing between texture details and noise.

[0012] Specifically, the present invention provides an image sharpening method based on the RAW domain, the method comprising the following steps:

[0013] S1: The sensor acquires RAW images, which are frames 0, 1, 2, ..., N in sequence, with frame N being the current frame;

[0014] S2: Perform multi-frame fusion denoising on the acquired images and extract random noise images;

[0015] S3: Based on the actual Bayer format, the multi-frame fused and denoised image and the random noise image are divided into multiple channels to achieve channel separation;

[0016] The following operations are performed on each of the channels after the aforementioned channel separation;

[0017] S4: Gaussian filtering is used to extract high-frequency components from each channel of the multi-frame fused and denoised image.

[0018] S5: Process the high-frequency component image and the random noise image and overlay them onto the multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below:

[0019] I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y)

[0020] Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This represents the final sharpening result for each channel;

[0021] S6: After filtering each channel, merge them into a RAW image and output the result;

[0022] The process of merging into a RAW image involves placing each pixel from the filtered channel image back into its original position in the RAW image, while maintaining its original position, thus forming a complete processed RAW image.

[0023] Step S2 includes the following steps:

[0024] S201: Multi-frame fusion denoising of the acquired images improves image quality and reduces image noise by combining information from multiple frames.

[0025]

[0026] Where (x, y) are the coordinates of the pixel; I i (x,y) is the pixel value at position (x,y) in the i-th frame; Fus (x,y) is the result of N frames fused together;

[0027] S202: Based on the result of multi-frame fusion and denoising and the current frame image, extract the random noise image of the current frame. The specific implementation method is as follows:

[0028] I Noise (x,y)=I N (x,y)-I Fus (x,y)

[0029] Among them, I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted image of random noise;

[0030] Based on the principle of noise formation in image imaging, its distribution in the spatiotemporal domain is uniform and random. Therefore, subtracting the fused and denoised image from the current frame image can yield a random noise image.

[0031] In step S3, assuming the Bayer format is RGGB, the four channels are R, Gr, Gb and B; that is, the data of the R, Gr, Gb and B channels are extracted, and the positional relationship of each pixel is kept unchanged, so that four images can be obtained. It is easy to see from the principle that their size is one-quarter of the original RAW image.

[0032] Step S3 further includes:

[0033] Image input is used to receive a RAW image in Bayer format. Assuming the Bayer format is RGGB, the four channels are R, Gr, Gb and B. The RAW image is composed of periodically arranged R, Gr, Gb and B pixels.

[0034] Assume the arrangement is as follows:

[0035]

[0036] Channel separation processing is configured as follows:

[0037] The RAW image is separated into four independent channel images: R, Gr, Gb, and B. Each channel image is one-quarter the size of the original RAW image. Specifically, the R channel extracts all R pixels while preserving their original position coordinates, the Gr channel extracts all Gr pixels while preserving their original position coordinates, the Gb channel extracts all Gb pixels while preserving their original position coordinates, and the B channel extracts all B pixels while preserving their original position coordinates.

[0038] The arrangement of the separated four-channel images is assumed to be as follows:

[0039] R channel:

[0040] R00 R02

[0041] R20 R22;

[0042] Gr channel:

[0043] Gr01 Gr03

[0044] Gr21 Gr23;

[0045] Gb channel:

[0046] Gb10 Gb12

[0047] Gb30 Gb32;

[0048] Channel B:

[0049] B11 B13

[0050] B31 B33;

[0051] The numbers represent the row and column coordinates of the pixel in the original RAW image;

[0052] Output is used to output the separated four-channel image for subsequent image processing.

[0053] Step S4 further includes:

[0054] S401: Perform Gaussian filtering on each channel with controllable parameters. Different parameters result in different Gaussian filtering intensities. The specific implementation method is shown below:

[0055]

[0056] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; IFus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering;

[0057] S402: Then, the high-frequency component image is obtained, that is, the high-frequency component image is obtained by subtracting the images before and after filtering.

[0058] I High (x,y)=I Fus (x,y)-I Gauss (x,y).

[0059] This application also relates to a RAW domain-based image sharpening system, the system being applicable to any of the methods described above, the system comprising:

[0060] The image acquisition module is used to acquire the current frame image in RAW format through a sensor;

[0061] A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include:

[0062] Pixel-level fusion is performed on multiple RAW images using the following formula:

[0063]

[0064] Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted random noise image, calculated using the following formula:

[0065] I Noise (x,y)=I N (x,y)-I Fus (x,y);

[0066] The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format. The channel separation module separates the image into four channels (R, Gr, Gb, and B) according to the RGGB Bayer format, with each channel having a size one-quarter of the original RAW image. The high-frequency extraction module is used to perform Gaussian filtering on the denoised image of each channel and obtain the high-frequency component image by subtracting the images before and after filtering. The high-frequency extraction module performs the following operations: performing Gaussian filtering on the denoised image of each channel, with the filtering formula being:

[0067]

[0068] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering;

[0069] Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details.

[0070] I High (x,y)=I Fus (x,y)-I Gauss (x,y);

[0071] The sharpening enhancement module is used to weightedly overlay the high-frequency component image and the random noise image onto the denoised image to obtain the sharpening results for each channel; the operations performed by the sharpening enhancement module include:

[0072] The high-frequency component image and the random noise image are processed and then superimposed onto a multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below:

[0073] I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y)

[0074] Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This represents the final sharpening result for each channel;

[0075] The intensity of random noise folding can control the degree of fragmentation of image texture details. This is because RAW domain images contain original image information, and the noise characteristics are not destroyed. The noise distribution is often uniform and fragmented. Therefore, when some random noise is folded back onto the texture details, the texture details will appear more fragmented in the sensory sense.

[0076] The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image. Merging into a RAW image involves placing each pixel in the filtered channel image back into its original position in the RAW image while keeping its position unchanged, thus forming a complete processed RAW image.

[0077] This application also relates to an image sharpening apparatus based on the RAW domain, the apparatus being suitable for any of the methods described above, comprising:

[0078] An image sensor is used to acquire the current frame image in RAW format;

[0079] Memory, used to store multiple frames of RAW images;

[0080] The processor is configured to execute the following modules:

[0081] A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include:

[0082] Pixel-level fusion is performed on multiple RAW images using the following formula:

[0083]

[0084] Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted random noise image, calculated using the following formula:

[0085] I Noise (x,y)=I N (x,y)-I Fus (x,y);

[0086] The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format; the channel separation module is configured to include separating the image into four channels, R, Gr, Gb, and B, according to the RGGB Bayer format;

[0087] The high-frequency extraction module performs Gaussian filtering on the denoised image of each channel and obtains the high-frequency component image by subtracting the images before and after filtering; the operations performed by the high-frequency extraction module include:

[0088] Gaussian filtering is applied to the denoised image of each channel. The filtering formula is as follows:

[0089]

[0090] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering;

[0091] Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details.

[0092] I High (x,y)=I Fus (x,y)-I Gauss (x,y);

[0093] The sharpening enhancement module is used to superimpose the high-frequency component image and the random noise image onto the denoised image according to weights to obtain the sharpening results of each channel;

[0094] The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image.

[0095] The device also includes an output interface for transmitting the processed RAW image to a display device or a storage device.

[0096] Therefore, the advantage of this solution is:

[0097] 1. Sharpening images in the RAW domain can effectively enhance weak textures and avoid the loss of image precision caused by data compression;

[0098] 2. Sharpening images in the RAW domain can effectively control the intensity and fineness of texture details, avoiding the destruction of noise-related characteristics caused by backend image interpolation and other related operations, resulting in structured noise and some erroneous textures. Attached Figure Description

[0099] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0100] Figure 1 This is a flowchart illustrating the method used in this application.

[0101] Figure 2 This is a schematic diagram of an actual arrangement in this application.

[0102] Figure 3 It is in this application Figure 2 A schematic diagram of the separated channels.

[0103] Figure 4 This is a block diagram of the system in this application.

[0104] Figure 5 This is a block diagram of the device in this application. Detailed Implementation

[0105] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0106] This application relates to an image sharpening method, system, and apparatus based on the RAW domain. To achieve the above objectives, this application provides the following technical solutions, such as... Figure 1 As shown, the method includes the following steps:

[0107] S1: The sensor acquires RAW images, which are frames 0, 1, 2, ..., N in sequence, with frame N being the current frame;

[0108] S2: Perform multi-frame fusion denoising on the acquired images and extract random noise images;

[0109] Step S2 mainly includes the following steps:

[0110] S201: Perform multi-frame fusion denoising on the acquired images. The purpose is to improve image quality and reduce image noise by combining information from multiple frames.

[0111]

[0112] Where (x, y) are the coordinates of the pixel; I i (x,y) is the pixel value at position (x,y) in the i-th frame; Fus (x,y) is the result of N frames fused together;

[0113] S202: Based on the result of multi-frame fusion and denoising and the current frame image, extract the random noise image of the current frame. The specific implementation method is as follows:

[0114] I Noise (x,y)=I N (x,y)-I Fus (x,y)

[0115] Among them, I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted image of random noise;

[0116] Based on the principle of noise formation in image imaging, its distribution in the spatiotemporal domain is often uniform and random. Therefore, subtracting the fused and denoised image from the current frame image can yield a random noise image.

[0117] S3: Based on the actual Bayer format, the multi-frame fused denoised image and the random noise image are sequentially divided into multiple channels to achieve channel separation; here, "sequentially" means that both the denoised image and the random noise image need to be separated into channels, and there is no requirement for the order of separation; of course, they can also be done in parallel.

[0118] Assuming the Bayer format is RGGB, then the four channels are R, Gr, Gb, and B. A practical arrangement is as follows: Figure 2 As shown, channel separation is as follows Figure 3 As shown, the R, Gr, Gb, and B channel data are extracted sequentially, without any order requirement. Alternatively, they can be extracted in parallel while maintaining the positional relationships of each pixel, resulting in four images. It's easy to see from the principle that their size is one-quarter of the original RAW image. The numbers represent the positions within the images; for example, "B11" means "11," indicating that the point is in the first row and first column of the original image. Note that the row and column numbers start from row 0 and column 0.

[0119] The following operations are performed sequentially on each of the previously separated channels; "sequentially" means that the same operation is performed on each of the separated channels; these operations can be performed sequentially without any order requirement; they can also be performed in parallel. The same operations S4-S5 are performed on all four channels separated from the RAW image, and then, referring to S6, the filtered results are merged into a RAW image; S4: Gaussian filtering is used to extract high-frequency components from each separated channel of the multi-frame fused and denoised image, as shown below:

[0120] S401: Perform Gaussian filtering on each channel with controllable parameters. Different parameters result in different Gaussian filtering intensities. Combined with the subsequent S402 step, different high-frequency component images can be obtained. The specific implementation method is shown below:

[0121]

[0122] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss(x,y) is the result after Gaussian filtering; if the Bayer format is fixed as RGGB, the Gaussian filtering window includes a fixed 5*5 window size, and the standard deviation parameter can be set to 1.2;

[0123] S402: Subtracting the images before and after filtering yields the high-frequency component image. This high-frequency component image contains texture details, and enhancing the high-frequency components can improve the intensity of the image's texture details.

[0124] I High (x,y)=I Fus (x,y)-I Gauss (x,y);

[0125] The high-frequency component image contains texture detail information of the image. Enhancing the high-frequency components can improve the intensity of the texture detail of the image. The enhancement here is only to supplement the purpose of extracting the high-frequency component image. The specific enhancement method is as described in step S5 below, namely the sharpening enhancement module.

[0126] S5: Process the high-frequency component image and the random noise image and overlay them onto the multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below:

[0127] I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y)

[0128] Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This is the final sharpening result for each channel; the parameters are set to k1 = 2 and k2 = 0.5.

[0129] The intensity of random noise folding can control the degree of fragmentation of image texture details. This is because RAW domain images contain original image information, and the noise characteristics are not destroyed. The noise distribution is often uniform and fragmented. Therefore, when some random noise is folded back onto the texture details, the texture details will appear more fragmented in the sensory sense.

[0130] S6: After filtering each channel, merge them into a RAW image and output the result;

[0131] The principle of merging into a RAW image is similar to that of channel separation. Each pixel in the filtered channel image is placed back into its original position in the RAW image while maintaining its original position, thus forming a complete processed RAW image.

[0132] like Figure 4 As shown, this application also relates to an image sharpening system based on the RAW domain, the system being applicable to any of the methods described above, the system comprising:

[0133] The image acquisition module is used to acquire the current frame image in RAW format through a sensor;

[0134] A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include:

[0135] Pixel-level fusion is performed on multiple RAW images using the following formula:

[0136]

[0137] Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted random noise image, calculated using the following formula:

[0138] I Noise (x,y)=I N (x,y)-I Fus (x,y);

[0139] The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format. The channel separation module separates the image into four channels (R, Gr, Gb, and B) according to the RGGB Bayer format, with each channel having a size one-quarter of the original RAW image. The high-frequency extraction module is used to perform Gaussian filtering on the denoised image of each channel and obtain the high-frequency component image by subtracting the images before and after filtering. The high-frequency extraction module performs the following operations: performing Gaussian filtering on the denoised image of each channel, with the filtering formula being:

[0140]

[0141] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering;

[0142] Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details.

[0143] I High (x,y)=I Fus (x,y)-I Gauss (x,y);

[0144] The sharpening enhancement module is used to weightedly overlay the high-frequency component image and the random noise image onto the denoised image to obtain the sharpening results for each channel; the operations performed by the sharpening enhancement module include:

[0145] The high-frequency component image and the random noise image are processed and then superimposed onto a multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below:

[0146] I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y)

[0147] Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This represents the final sharpening result for each channel;

[0148] The intensity of random noise folding can control the degree of fragmentation of image texture details. This is because RAW domain images contain original image information, and the noise characteristics are not destroyed. The noise distribution is often uniform and fragmented. Therefore, when some random noise is folded back onto the texture details, the texture details will appear more fragmented in the sensory sense.

[0149] The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image. Merging into a RAW image involves placing each pixel in the filtered channel image back into its original position in the RAW image while keeping its position unchanged, thus forming a complete processed RAW image.

[0150] In addition, such as Figure 5 As shown, this application also relates to an image sharpening device based on the RAW domain, comprising:

[0151] An image sensor is used to acquire the current frame image in RAW format;

[0152] Memory, used to store multiple frames of RAW images;

[0153] The processor is configured to execute the following modules:

[0154] A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include:

[0155] Pixel-level fusion is performed on multiple RAW images using the following formula:

[0156]

[0157] Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted random noise image, calculated using the following formula:

[0158] I Noise (x,y)=I N (x,y)-I Fus (x,y);

[0159] The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format; the channel separation module is configured to include separating the image into four channels, R, Gr, Gb, and B, according to the RGGB Bayer format;

[0160] The high-frequency extraction module performs Gaussian filtering on the denoised image of each channel and obtains the high-frequency component image by subtracting the images before and after filtering; the operations performed by the high-frequency extraction module include:

[0161] Gaussian filtering is applied to the denoised image of each channel. The filtering formula is as follows:

[0162]

[0163] Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering;

[0164] Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details.

[0165] I High (x,y)=I Fus (x,y)-I Gauss (x,y);

[0166] The sharpening enhancement module is used to superimpose the high-frequency component image and the random noise image onto the denoised image according to weights to obtain the sharpening results of each channel;

[0167] The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image.

[0168] The device also includes an output interface for transmitting the processed RAW image to a display device or a storage device.

[0169] In summary, the present invention can achieve: image sharpening in the RAW domain, combined with multi-frame fusion denoising, which can effectively improve the intensity of weak textures and achieve controllable texture detail intensity and the degree of texture detail fragmentation.

[0170] This technical solution can be applied to all series of chips from Ingenic Semiconductor and similar chips.

[0171] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image sharpening method based on the RAW domain, characterized in that, The method includes the following steps: S1: The sensor acquires RAW images, which are frames 0, 1, 2, ..., N in sequence, with frame N being the current frame; S2: Perform multi-frame fusion denoising on the acquired images and extract random noise images; S3: Based on the actual Bayer format, the multi-frame fused and denoised image and the random noise image are divided into multiple channels to achieve channel separation; The following operations are performed on each of the channels after the aforementioned channel separation. That is, the same operations as steps S4 to S5 are performed on the four channels separated from the RAW image. Then, referring to step S6, the filtered results are merged into a RAW image. S4: Gaussian filtering is used to extract high-frequency components from each channel of the multi-frame fused and denoised image. S5: Process the high-frequency component image and the random noise image and overlay them onto the multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below: I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y) Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This represents the final sharpening result for each channel; S6: After filtering each channel, merge them into a RAW image and output the result; The process of merging into a RAW image involves placing each pixel from the filtered channel image back into its original position in the RAW image, while maintaining its original position, thus forming a complete processed RAW image.

2. The image sharpening method based on the RAW domain according to claim 1, characterized in that, Step S2 includes the following steps: S201: Multi-frame fusion denoising of the acquired images improves image quality and reduces image noise by combining information from multiple frames. Where (x, y) are the coordinates of the pixel; I i (x,y) is the pixel value at position (x,y) in the i-th frame; Fus (x,y) is the result of N frames fused together; S202: Based on the result of multi-frame fusion and denoising and the current frame image, extract the random noise image of the current frame. The specific implementation method is as follows: I Noise (x,y)=I N (x,y)-I Fus (x,y) Among them, I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted image of random noise; Based on the principle of noise formation in image imaging, its distribution in the spatiotemporal domain is uniform and random. Therefore, subtracting the fused and denoised image from the current frame image can yield a random noise image.

3. The image sharpening method based on the RAW domain according to claim 2, characterized in that, In step S3, assuming the Bayer format is RGGB, the four channels are R, Gr, Gb, and B. That is, the R, Gr, Gb, and B channel data are extracted while maintaining the positional relationship of each pixel. This yields four images, whose size is easily determined by the principle to be one-quarter of the original RAW image. The principle refers to the fact that the original image contains four channels, and by extracting each channel sequentially, four images are obtained. The area of ​​each image is one-quarter of the original RAW image, that is, the width is half of the original image, and the height is also half of the original image.

4. The image sharpening method based on the RAW domain according to claim 3, characterized in that, Step S3 further includes: Image input is used to receive a RAW image in Bayer format. Assuming the Bayer format is RGGB, the four channels are R, Gr, Gb and B. The RAW image is composed of periodically arranged R, Gr, Gb and B pixels. Assume the arrangement is as follows: Channel separation processing is configured as follows: The RAW image is separated into four independent channel images: R, Gr, Gb, and B. Each channel image is one-quarter the size of the original RAW image. The R channel extracts all R pixels while preserving their original position coordinates. The Gr channel extracts all Gr pixels while preserving their original position coordinates. Extract all Gb pixels from the Gb channel while preserving their original position coordinates. Extract all B pixels from the B channel while preserving their original position coordinates. The arrangement of the separated four-channel images is assumed to be as follows: R channel: R00 R02 R20 R22; Gr channel: Gr01 Gr03 Gr21 Gr23; Gb channel: Gb10 Gb12 Gb30 Gb32; Channel B: B11 B13 B31 B33; The numbers represent the row and column coordinates of the pixel in the original RAW image; Output is used to output the separated four-channel image for subsequent image processing.

5. The image sharpening method based on the RAW domain according to claim 4, characterized in that, Step S4 further includes: S401: Perform Gaussian filtering on each channel with controllable parameters. Different parameters result in different Gaussian filtering intensities. The specific implementation method is shown below: Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering; S402: This leads to the high-frequency component image, which is obtained by subtracting the images before and after filtering. I High (x,y)=I Fus (x,y)-I Gauss (x,y)。 6. An image sharpening system based on the RAW domain, characterized in that, The system is applicable to the method described in any one of claims 1-5 above, and the system comprises: The image acquisition module is used to acquire the current frame image in RAW format through a sensor; A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include: Pixel-level fusion is performed on multiple RAW images using the following formula: Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the Nth frame of the image, i.e., the current frame, I Noise It is an extracted random noise image, calculated using the following formula: I Noise (x,y)=I N (x,y)-I Fus (x,y); The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format. The channel separation module separates the image into four channels (R, Gr, Gb, and B) according to the RGGB Bayer format, with each channel having a size one-quarter of the original RAW image. The high-frequency extraction module is used to perform Gaussian filtering on the denoised image of each channel and obtain the high-frequency component image by subtracting the images before and after filtering. The high-frequency extraction module performs the following operations: performing Gaussian filtering on the denoised image of each channel, with the filtering formula being: Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering; Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details. I High (x,y)=I Fus (x,y)-I Gauss (x,y); The sharpening enhancement module is used to weightedly overlay the high-frequency component image and the random noise image onto the denoised image to obtain the sharpening results for each channel; the operations performed by the sharpening enhancement module include: The high-frequency component image and the random noise image are processed and then superimposed onto a multi-frame fused denoised image to obtain the sharpening results for each channel, as shown below: I Out (x,y)=I Fus (x,y)+k1·I High (x,y)+k2·I Noise (x,y) Wherein, k1 controls the aliasing intensity of high-frequency components, i.e., controls the intensity of image texture details, and k2 controls the aliasing intensity of random noise, i.e., controls the fineness of image texture details; I Noise (x,y) is the pixel value at position (x,y) of a certain channel of the random noise image. Out This represents the final sharpening result for each channel; The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image. Merging into a RAW image involves placing each pixel in the filtered channel image back into its original position in the RAW image while keeping its position unchanged, thus forming a complete processed RAW image.

7. An image sharpening device based on the RAW domain, characterized in that, The apparatus is applicable to the method described in any one of claims 1-5, comprising: An image sensor is used to acquire the current frame image in RAW format; Memory, used to store multiple frames of RAW images; The processor is configured to execute the following modules: A multi-frame fusion denoising module is used to fuse and denoise multiple RAW images to obtain a denoised image, and to extract random noise images based on the current frame image and the denoised image; the operations performed by the multi-frame fusion denoising module include: Pixel-level fusion is performed on multiple RAW images using the following formula: Among them, I i (x,y) represents the pixel value at position (x,y) in the i-th frame image. Fus (x,y) represents the fused image; based on I N For the first N The frame image is the current frame, I Noise It is an extracted random noise image, calculated using the following formula: I Noise (x,y)=I N (x,y)-I Fus (x,y); The channel separation module is used to separate the denoised image and the random noise image into multiple channels according to the Bayer format; the channel separation module is configured to include separating the image into four channels, R, Gr, Gb, and B, according to the RGGB Bayer format; The high-frequency extraction module performs Gaussian filtering on the denoised image of each channel and obtains the high-frequency component image by subtracting the images before and after filtering; the operations performed by the high-frequency extraction module include: Gaussian filtering is applied to the denoised image of each channel. The filtering formula is as follows: Where (x,y) are the coordinates of the pixel; σ is the standard deviation of the Gaussian function, controlling the smoothness of the filter; G(x,y) is the weight at the location (x,y) of the Gaussian kernel; k is the radius of the Gaussian kernel; I Fus (x+i, y+j) is the pixel value at position (x+i, y+j) of a certain channel of the multi-frame fused and denoised image; I Gauss (x,y) is the result after Gaussian filtering; Subtracting the images before and after filtering yields the high-frequency component image, which contains texture details. Enhancing the high-frequency components can improve the intensity of the image's texture details. I High (x,y)= I Fus (x,y)- I Gauss (x,y); The sharpening enhancement module is used to superimpose the high-frequency component image and the random noise image onto the denoised image according to weights to obtain the sharpening results of each channel; The channel merging module is used to re-merge the sharpening results of each channel according to the Bayer format and output the processed RAW image.

8. The image sharpening device based on the RAW domain according to claim 7, characterized in that, The device also includes an output interface for transmitting the processed RAW image to a display device or a storage device.

Images