Image processing apparatus, method, electronic device, and medium

Abstract

The embodiment of the present disclosure provides an image processing device, method, electronic equipment, storage medium and computer program product, and relates to the technical field of image processing. The image processing device comprises: an extraction unit configured to extract M color channel images corresponding to M color channels from a to-be-processed image, the to-be-processed image comprising a plurality of pixels, and M being an integer greater than or equal to 2; a fitting unit configured to, for each color channel image in the M color channel images, perform linear fitting on pixel values of the color channel image by using pixel values of at least one color channel image in the other M-1 color channel images, to obtain M calculation results for the M color channel images; and a merging unit configured to merge the M calculation results to obtain a processing result of the to-be-processed image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing apparatus, method, electronic device, storage medium, and computer program product. Background Technology

[0002] The function of an image sensor is to convert light signals into digital signals (such as electrical signals) that can be processed by an ISP (Image Signal Processor). The basic unit of light-sensing in a sensor is called a photosite. Each sensor carries millions or even more photosites, which are arranged in an orderly and regular array to form the sensor's photosensitive area. When visible light passes through the lens and enters the photosite, it is absorbed by the photosensitive area, and there is a certain probability that electrons are excited. This process is called photoelectric conversion.

[0003] ISP noise reduction primarily focuses on noise generated during image acquisition. Noise signals are produced during acquisition due to the physical properties of the devices themselves, the principles of photosensitive processing, voltage fluctuations, and other factors. Common noise reduction methods include mean filtering, median filtering, and Gaussian filtering. The main idea is to correct the value of the current pixel by adjusting the values ​​of pixels surrounding it. For example, mean filtering sets the value of the current pixel to the average value of pixels within a rectangular area centered on the current pixel. This approach reduces the intensity of noise signals but introduces blurring at image edges. Summary of the Invention

[0004] This disclosure provides an image processing apparatus, method, electronic device, storage medium, and computer program product.

[0005] According to a first aspect of the present disclosure, an image processing apparatus is provided, comprising: an extraction unit configured to extract M color channel images corresponding to M color channels respectively from an image to be processed, the image to be processed including a plurality of pixels, and M being an integer greater than or equal to 2; a fitting unit configured to linearly fit the pixel values ​​of each color channel image in the M color channel images using the pixel values ​​of at least one color channel image in the other M-1 color channel images to obtain M calculation results for the M color channel images; and a merging unit configured to merge the M calculation results to obtain a processing result for the image to be processed.

[0006] According to embodiments of this disclosure, the calculation result for each color channel image includes at least one fitting result; the fitting unit is further configured to: for a color channel image, determine a target window with a size of m×m centered on a target pixel in the color channel image, where m is an odd number greater than 1; take at least one color channel image from the other M-1 color channel images as a reference channel image, determine a reference pixel at a position corresponding to the target pixel in the reference channel image, determine a reference window with a size of m×m centered on the reference pixel; and use the pixel values ​​in the reference window to perform linear fitting on the pixel values ​​in the target window to obtain at least one fitting result for the color channel image.

[0007] According to embodiments of this disclosure, the fitting unit is further configured to: determine the inter-channel linear relationship based on the pixel values ​​in the reference window and the pixel values ​​in the target window, wherein the inter-channel linear relationship characterizes the linear correlation between the photosensitive intensities of pixels in different color channels; and fit the target pixels in the color channel image based on the inter-channel linear relationship to obtain at least one fitting result of the color channel image.

[0008] According to an embodiment of this disclosure, the fitting unit is further configured to: transform each pixel value in the reference window according to the initial inter-channel linear relationship to obtain multiple predicted pixel values ​​in the target window; determine the difference between the multiple predicted pixel values ​​in the target window and each pixel value in the target window according to a preset loss function; and adjust at least one parameter value included in the initial inter-channel linear relationship according to the difference.

[0009] According to an embodiment of this disclosure, the fitting unit is further configured to: for a color channel image, perform a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain the calculation result of the color channel image.

[0010] According to embodiments of this disclosure, the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel; the fitting unit is further configured to: for any one of the color channels (red, green, and blue), perform linear fitting on the color channel image using the pixel values ​​of the color channel image of the near-infrared channel to obtain the calculation result of the color channel image.

[0011] According to embodiments of this disclosure, the fitting unit is further configured to: for a color channel image, perform a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain a first calculation result of the color channel image; and use the pixel values ​​of the color channel image of the near-infrared channel to perform linear fitting on the first calculation result to obtain a second calculation result of the color channel image, and determine the second calculation result of the color channel image as the calculation result of the color channel image.

[0012] According to a second aspect of the present disclosure, an imaging device is provided, comprising: an image acquisition device configured to acquire an image to be processed; and an image processing device provided in the above embodiments of the present disclosure.

[0013] According to a third aspect of the present disclosure, an image processing method is provided, comprising: extracting M color channel images corresponding to M color channels from an image to be processed, wherein the image to be processed includes a plurality of pixels, and M is an integer greater than or equal to 2; for each color channel image in the M color channel images, linearly fitting the pixel values ​​of the color channel image using the pixel values ​​of at least one color channel image in the other M-1 color channel images to obtain M calculation results for the M color channel images; and merging the M calculation results to obtain a processing result for the image to be processed.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the image processing method described in the embodiments of the present disclosure.

[0015] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the image processing method described in the embodiments of the present disclosure.

[0016] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described in the embodiments of the present disclosure.

[0017] The image processing apparatus, method, electronic device, storage medium, and computer program products provided in this disclosure utilize the frame alignment characteristics between multi-channel data of a single-frame image to linearly fit pixel values ​​between different channels of the image, mapping data from other channels to the current channel in a linear manner to obtain the calculation result for that channel. This image processing apparatus can effectively solve the edge blurring problem easily caused by conventional noise reduction methods, maintaining edge sharpness while reducing noise, and achieving relatively accurate color reconstruction. Attached Figure Description

[0018] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings. It should be noted that throughout the drawings, the same elements are indicated by the same or similar reference numerals. In the figures:

[0019] Figure 1A schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure is shown;

[0020] Figure 2 A schematic diagram illustrating the principle of extracting color channel images from an image to be processed according to an embodiment of the present disclosure is shown;

[0021] Figure 3 A schematic diagram of the fitting unit according to an embodiment of the present disclosure is shown;

[0022] Figure 4 A schematic diagram illustrating the principle of determining the calculation results of a color channel image according to an embodiment of the present disclosure is shown;

[0023] Figure 5 A schematic diagram of the structure of an imaging apparatus suitable for an image processing device according to an embodiment of the present disclosure is shown;

[0024] Figure 6 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;

[0025] Figure 7 A flowchart illustrating the fitting of a target window using a reference window according to an embodiment of this disclosure is shown; and

[0026] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the described embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. In the following description, some specific embodiments are for illustrative purposes only and should not be construed as limiting this disclosure in any way, but are merely examples of embodiments of this disclosure.

[0028] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning as understood by those skilled in the art. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components.

[0029] It is understood that in the specific embodiments of this disclosure, data related to image processing is involved. When the following embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0030] First, a brief introduction to the technical terms and basic principles involved in this disclosure will be given.

[0031] Visible-near-infrared sensor (RGB-NIR sensor): An image sensor that combines a visible light (RGB) and near-infrared (NIR) binocular camera to acquire images in four channels: red (R), green (G), blue (B), and NIR.

[0032] Because image sensors can only capture light intensity, they can only produce grayscale images when acquiring images and cannot recognize colors. Therefore, colored filters (such as red, blue, and green) are usually superimposed on each pixel of the image sensor to capture light of a specific wavelength (i.e., color).

[0033] In low-light scenes, the low light intensity results in weak electrical signals generated by the RGB channels on the sensor. Furthermore, noise signals due to the inherent physical properties of the device still exist, leading to a low signal-to-noise ratio, numerous noisy images, and unclear display, making noise reduction difficult. To address this issue, a simple noise reduction approach is to capture multiple images of the same scene. Since noise appears randomly at a certain point, a weighted average of the multiple images can be used to reduce the impact of random noise at that point, resulting in a clearer image with less noise interference.

[0034] However, the above noise reduction approach requires capturing multiple frames of images, and these frames must be strictly aligned; otherwise, weighted averaging will result in a more blurred image. Such ideal multi-frame image data is unattainable in real-world scenarios.

[0035] In realizing the present invention, the inventors discovered that since the multi-channel data of a single frame image is based on the arrangement of photosensitive points on the sensor and has strict frame alignment characteristics, the weighted average of the multi-channel data can be performed by utilizing the noise randomness between the multi-channels of the single frame data to perform mutual noise reduction between different channels.

[0036] Furthermore, the inventors discovered that, due to the different light sensitivity of pixels in different color channels, even with the randomness of noise between multiple channels in a single frame of data, it is impossible to directly perform a weighted average of the fitting results of multi-channel images.

[0037] Therefore, the inventors established a theoretical assumption that the local light sensitivity of different color channels in a single frame of image data is linearly correlated. That is, for example, in a local region of an image, if the light sensitivity of a pixel in the R channel is y, and the light sensitivity of the corresponding pixel in the NIR channel is x, then there is a linear relationship between them: y = ax + b, where a and b are intermediate parameters to be solved. Therefore, as long as a and b are solved, the problem of different pixel light sensitivity in different color channels can be solved using linear transformations between the different channels.

[0038] In view of this, embodiments of the present disclosure provide an image processing apparatus, method, electronic device, storage medium, and computer program product. According to embodiments of the present disclosure, during image processing, the frame alignment characteristics between multi-channel data of a single frame image are utilized, and the high signal-to-noise ratio of NIR channel images and the linear correlation of local photosensitive intensity between different color channels are leveraged to perform mutual noise reduction between different channels, solving the edge blurring problem easily caused by conventional noise reduction methods, and obtaining a higher quality image.

[0039] The various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 1 A schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0041] As shown in FIG1, the data processing device 100 of this embodiment includes an extraction unit 110, a fitting unit 120 and a merging unit 130.

[0042] The extraction unit 110 is configured to extract M color channel images corresponding to M color channels from the image to be processed. The image to be processed includes multiple pixels, and M is an integer greater than or equal to 2.

[0043] The image to be processed is the image to be denoised. For example, the image to be processed can be acquired by an image acquisition device. The image to be processed can be the original image inside the image acquisition device, and the file extension of the original image can be .raw.

[0044] Extraction unit 110 splits the image to be processed according to color channels to obtain M color channel images. Since the image to be processed is a single frame image data, these M color channel images are multi-channel data of the single frame image data and have strict frame alignment characteristics.

[0045] The fitting unit 120 is configured to perform linear fitting on the pixel values ​​of each color channel image in the M color channel images by using the pixel values ​​of at least one color channel image in the other M-1 color channel images, so as to obtain M calculation results for the M color channel images.

[0046] The merging unit 130 is configured to merge M calculation results to obtain the processing result of the image to be processed.

[0047] It should be noted that the embodiments disclosed herein do not utilize every color channel image among the other M-1 color channel images to process the color channel image to be denoised. For example, in low-light scenes, the electrical signals excited by the R, G, and B channels are weak and the noise intensity is high, while the NIR channel has a high signal-to-noise ratio. In this case, a higher quality channel image can be obtained by using only the NIR channel image to denoise any one of the R, G, and B channels.

[0048] Furthermore, during the merging process of the M calculation results, the merging unit 130 performs the merging according to the pixel arrangement order of the image acquisition device required for the image to be processed. The merging process of the merging unit 130 is the reverse process of the extraction unit 110.

[0049] The image processing apparatus provided in this disclosure utilizes the frame alignment characteristics between multi-channel data of a single-frame image to linearly fit pixel values ​​between different channels of the image, mapping data from other channels to the current channel in a linear manner to obtain the calculation result for that channel. This method effectively solves the edge blurring problem easily caused by conventional noise reduction methods, maintaining edge sharpness while reducing noise, and achieving relatively accurate color reconstruction.

[0050] For ease of understanding, the following example uses a visible light near-infrared sensor in 2×2 mode as an example, along with the accompanying diagram. Figures 2-4 The image processing apparatus according to embodiments of the present disclosure will be described in detail. In 2×2 mode, the color filters superimposed on each pixel of the acquired image to be processed are 2×2 repeating units, which are red filter, green filter, near-infrared filter and blue filter from left to right and from top to bottom.

[0051] Figure 2 A schematic diagram illustrating the principle of extracting color channel images from an image to be processed according to an embodiment of the present disclosure is shown.

[0052] like Figure 2 As shown, the image to be processed 200 can be captured by a visible light near-infrared sensor in 2×2 mode, that is, M color channels including red channel, green channel, blue channel and near-infrared channel. Therefore, the extraction unit 110 can split the image to be processed 200 according to color channels to obtain red channel image 201, green channel image 202, near-infrared channel image 203 and blue channel image 204.

[0053] In this embodiment of the disclosure, the calculation result of each color channel image includes at least one fitting result; the fitting unit 120 is further configured to: for the color channel image, determine a target window with a size of m×m centered on the target pixel in the color channel image, where m is an odd number greater than 1; take at least one color channel image among the other M-1 color channel images as a reference channel image, determine a reference pixel at the position corresponding to the target pixel in the reference channel image, determine a reference window with a size of m×m centered on the reference pixel; and use the pixel values ​​in the reference window to perform linear fitting on the pixel values ​​in the target window to obtain at least one fitting result of the color channel image.

[0054] It should be noted that the reference channel image is relative to the channel image to be denoised; that is, any channel image different from the channel image to be denoised can be a reference channel image. For example, if the R channel image is used as the channel image to be denoised, then the G channel, B channel, and NIR channel images can all be reference channel images. Alternatively, at least one of the G channel, B channel, and NIR channel images can be selected as the reference channel image.

[0055] Figure 3 A schematic diagram of the fitting unit according to an embodiment of the present disclosure is shown.

[0056] like Figure 3 As shown, taking the near-infrared channel image 303 as the reference channel image and denoising the red channel image 301 as an example, the process of obtaining a fitting result of the red channel image 301 is explained.

[0057] First, the target pixel is determined in the red channel image 301 to be denoised. This target pixel can be determined by traversing from left to right as shown in 301A and from top to bottom as shown in 301B. For the determined target pixel 301C, a target window 301D with a size of m×m is taken, centered on the target pixel 301C. If m is an odd number greater than 1, it ensures that the target window 301D is centered on the target pixel 301C. Typically, m is taken as 5 or 7.

[0058] Next, for the target pixel 301C, a reference pixel 303C corresponding to the target pixel 301C is determined in the near-infrared channel image 303, and then a reference window 303D with the reference pixel 303C as the center and a size of m×m is taken.

[0059] As can be seen, the target window 301D and the reference window 303D are located in the same position in the image to be processed, and their window sizes are exactly the same, but they are in different color channels. Therefore, the pixel values ​​in the reference window 303D can be used to perform linear fitting on the pixel values ​​in the target window 301D to obtain a fitting result for the red channel image 301.

[0060] It should be noted that for a color channel image to be denoised, fitting the image to that color channel image using each of the other M-1 color channel images will yield a fitting result for that color channel image. Therefore, the number of fitting results for each color channel image is determined based on the number of reference channel images.

[0061] Understandable, Figure 3 The illustrated embodiment uses the near-infrared channel image 303 as a reference channel image and illustrates noise reduction of the red channel image 301 as an example. The same method can be used to process each of the M color channel images; details will not be elaborated further.

[0062] The image processing apparatus provided in this embodiment performs linear fitting on local windows at corresponding positions between different color channels of the image to be processed, and maps other channel data onto the current channel in a linear manner to obtain at least one fitting result for that channel.

[0063] Based on the aforementioned fundamental principles, the inventors have established a theoretical assumption that the local light sensitivity intensity of different color channels in a single frame of image data is linearly correlated. In this embodiment, the fitting unit 120 is further configured to: determine the inter-channel linear relationship based on the pixel values ​​in the reference window and the pixel values ​​in the target window, wherein the inter-channel linear relationship characterizes the linear correlation between the pixel light sensitivity intensities of different color channels; and, based on the inter-channel linear relationship, fit the target pixels in the color channel image to obtain at least one fitting result for the color channel image.

[0064] For example, targeting pixel y in the color channel image to be denoised. i , where i is the pixel index, and assuming a reference pixel x in a certain reference channel image. i With the target pixel y i There exists a linear relationship between channels as shown in equation (1):

[0065] y i =ax i +b,i∈w k (1)

[0066] In equation (1), w kIt is a reference window with a size of m×m; a and b are two parameter values ​​to be solved.

[0067] Based on this inter-channel linear relationship containing the parameter values ​​to be solved, a linear fit is performed between each target pixel in the target window of the color channel image to be denoised and each reference pixel in the reference window of the corresponding reference channel image. After iterative iteration, the optimal parameter values ​​are obtained, and thus a clear inter-channel linear relationship is obtained.

[0068] Next, based on the explicit linear relationship between channels, the reference pixel x is used. i For target pixel y i By performing fitting, the target pixel y is obtained. i The fitting result is obtained by iterating through any target pixel of the color channel image.

[0069] Since the inter-channel linear relationship initially includes at least one parameter value to be solved, these parameter values ​​need to be adjusted to obtain the optimal parameter values ​​a and b before a clear inter-channel linear relationship is obtained. In this embodiment of the disclosure, the fitting unit 120 is further configured to: transform each pixel value in the reference window according to the initial inter-channel linear relationship to obtain multiple predicted pixel values ​​in the target window; determine the difference between the multiple predicted pixel values ​​in the target window and each pixel value in the target window according to a preset loss function; and adjust at least one parameter value included in the initial inter-channel linear relationship according to the difference.

[0070] For example, targeting pixel y in the color channel image to be denoised. i Assign initial values ​​a and b to a in equation (1) above. k b k This yields the initial inter-channel linear relationship. Based on this initial inter-channel linear relationship, the reference pixel x in a certain reference window is... i After transformation, the predicted pixel values ​​in the target window are obtained as shown in equation (2):

[0071] y i ′=a k x i +b k , i∈w k (2)

[0072] In equation (2), y i ' is the target pixel y i The predicted pixel value.

[0073] Next, the preset loss function is established as shown in equation (3):

[0074]

[0075] In equation (3), ε is the convergence coefficient, 0 < ε < 1; E(a k b k ) represents the preset loss function. Therefore, the preset loss function can be used to determine the predicted pixel values ​​y in the target window. i ′ and the individual pixel values ​​y in the target window i The differences between them.

[0076] In this embodiment of the disclosure, the optimal parameter values ​​a and b are obtained by minimizing a preset loss function.

[0077] Specifically, the preset loss function shown in equation (3) is applied to a k b k Taking the partial derivatives separately, we can obtain:

[0078]

[0079]

[0080] Since the partial derivative at the minimum is 0, let the preset loss function be set to b. k Since the partial derivative of is 0, we can solve for:

[0081]

[0082]

[0083] In equation (7), mean(·) represents the average of the array; I is any pixel in the target window; x is the reference window w k Any pixel in the array.

[0084] Similarly, let the preset loss function apply to a k Since the partial derivative of is 0, we can solve for:

[0085]

[0086] In equation (8), y is any pixel in the target window.

[0087] Since the formulas for variance and covariance can be expressed as:

[0088] Var(x) = E[x] 2 ]-E[x] 2 (9)

[0089] Conv(x,y)=E[xy]-E[x]·E[y] (10)

[0090] In equation (9), Var(x) is the variance of x; E(·) represents the expectation; in equation (10), Cony(x, y) is the covariance of x and y.

[0091] Based on equations (9) and (10) above, equation (8) can be transformed into:

[0092]

[0093] Based on equations (7) and (11) above, the optimal parameter values ​​a and b can be obtained. Then, using these optimal parameter values ​​a and b, the reference pixel x is... i Substituting the inter-channel linear relationship into equation (1), we obtain the target pixel y in the color channel image. i The fitting results.

[0094] In this embodiment of the disclosure, the fitting unit 120 is further configured to: for a color channel image, perform a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain the calculation result of the color channel image.

[0095] Figure 4 A schematic diagram illustrating the principle of determining the calculation results of a color channel image according to an embodiment of the present disclosure is shown.

[0096] like Figure 4 As shown, when there are M color channels including red, green, blue, and near-infrared channels, taking a target pixel R in the red channel image 401 as the pixel to be denoised as an example, the target pixel in the red channel image 401 can be denoised using the green channel image 402, the near-infrared channel image 403, and the blue channel image 404 as reference channel images, respectively, to obtain the first fitting result R of the red channel image fitted using the green channel. G The second fitting result R of the red channel image fitted using the near-infrared channel. N And the third fitting result R of the red channel image fitted using the blue channel. B .

[0097] For example, consider a target pixel R and the fitting result R of a color channel image. G R N R B Assigning the same weight of 0.25, the weighted summation yields the calculation result R′ of the red channel image 401 as shown in equation (12):

[0098]

[0099] The image processing apparatus provided by the embodiments of this disclosure uses a real current channel data and at least one fitting result obtained by linear fitting of other channel data to reduce noise and improve the signal-to-noise ratio by weighted summation, which is applicable to situations where there is a certain intensity of noise even in the NIR channel.

[0100] In one optional implementation, the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel; the fitting unit 120 is further configured to: for any one of the color channels (red, green, and blue), use the pixel values ​​of the color channel image of the near-infrared channel to perform linear fitting on the color channel image to obtain the calculation result of the color channel image.

[0101] For example, if the channel image to be denoised is a red channel image, this implementation method is suitable for situations where the NIR channel is relatively clean and the noise intensity of the red channel image is low. The denoised data of the red channel image can be directly fitted using the pixel distribution of the NIR channel.

[0102] The image processing apparatus provided by the embodiments of this disclosure can take advantage of the high signal-to-noise ratio of NIR channel images to perform mutual noise reduction between different channels, and obtain higher quality images even in low-light scenes.

[0103] In another optional implementation, the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel; the fitting unit 120 is further configured to: for the color channel image, perform a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain a first calculation result of the color channel image; and, using the pixel values ​​of the color channel image of the near-infrared channel, perform a linear fit on the first calculation result to obtain a second calculation result of the color channel image, and determine the second calculation result of the color channel image as the calculation result of the color channel image.

[0104] For example, if the image to be denoised is a red channel image, this implementation method is suitable for cases where the noise intensity of the red channel image is high. If only the near-infrared channel is used to perform linear fitting on the other channels, due to noise, the optimal parameter values ​​a and b will have a large error compared to the ideal linear relationship between the channels. Therefore, this implementation method first uses weighted summation to perform the first denoising on the red channel, and then uses the near-infrared channel to perform linear fitting on the data after the first denoising to perform the second denoising on the red channel, thus obtaining the calculation result.

[0105] Furthermore, the inventors conducted experimental verification of the image processing device shown in the above embodiments. The experimental results show that the image processing device shown in the above embodiments can effectively reduce the edge blurring problem that is easily caused by conventional noise reduction methods, and the actual results can maintain the edge sharpness while reducing noise.

[0106] For low-light scenes, such as at night, the images captured by the visible light and near-infrared sensors are processed. By comparing the linearly fitted channels with the original channels, it is found that the numerical distribution ranges are similar, and the color reconstruction is relatively accurate from the exported images.

[0107] The image processing apparatus provided in this disclosure utilizes the frame alignment characteristics between multi-channel data of a single-frame image, leverages the high signal-to-noise ratio of NIR channel images, and takes advantage of the linear correlation between the photosensitive intensities of different pixels in a local image. It performs mutual noise reduction between different channels of the image to be processed captured by a visible-near-infrared sensor, resulting in a higher-quality image. Furthermore, this image processing apparatus is also applicable to low-light scenes, effectively reducing edge blurring problems easily caused by conventional noise reduction methods. The actual results maintain edge sharpness while reducing noise, and color reconstruction is relatively accurate.

[0108] As an extension of the image processing apparatus shown in the above embodiments, this disclosure also provides an imaging device. Figure 5 A schematic diagram of the structure of an imaging apparatus suitable for an image processing device according to an embodiment of the present disclosure is shown.

[0109] like Figure 5 As shown, the imaging device 500 provided in this embodiment includes the image processing device 100 and the image acquisition device 510 provided in the foregoing embodiments. The image acquisition device 510 is configured to acquire an image to be processed.

[0110] The image acquisition device 510 can be any type of image acquisition device, such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, etc. In one embodiment, when the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel, the image acquisition device 510 can be, for example, a visible light near-infrared sensor. In other embodiments, the type of image acquisition device 510 is not specifically limited.

[0111] In some embodiments, the imaging device 500 may be a smartphone, camera, camcorder, tablet computer, laptop computer, surveillance equipment, etc.

[0112] It should be noted that the division of units in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be physically stored separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0113] For example, any plurality of modules in the extraction unit 110, fitting unit 120, merging unit 130, and image acquisition device 510 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the extraction unit 110, fitting unit 120, merging unit 130, and image acquisition device 510 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the extraction unit 110, fitting unit 120, merging unit 130, and image acquisition device 510 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0114] Embodiments of this disclosure also provide an image processing method performed by an image processing apparatus, which will be described below in conjunction with... Figure 6 and Figure 7 The method is described in detail.

[0115] Figure 6 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0116] like Figure 6 As shown, the image processing method of this embodiment can be applied to, for example, the image processing apparatus 100 according to an embodiment of this disclosure. The image processing method of this embodiment includes operations S610 to S630.

[0117] In operation S610, M color channel images corresponding to M color channels are extracted from the image to be processed. The image to be processed includes multiple pixels, and M is an integer greater than or equal to 2.

[0118] In operation S620, for each color channel image in the M color channel images, the pixel values ​​of that color channel image are linearly fitted using the pixel values ​​of at least one color channel image in the other M-1 color channel images, to obtain M calculation results for the M color channel images.

[0119] In operation S630, the M calculation results are merged to obtain the processing result of the image to be processed.

[0120] The image processing method provided in this disclosure utilizes the frame alignment characteristics between multi-channel data in a single frame image to linearly fit pixel values ​​across different channels of the image. Data from other channels is linearly mapped to the current channel to obtain the calculation result for that channel. This method effectively reduces edge blurring issues that are easily caused by conventional noise reduction methods, maintaining edge sharpness while reducing noise, and achieving relatively accurate color reconstruction.

[0121] Figure 7 A flowchart illustrating the fitting of a target window using a reference window according to an embodiment of the present disclosure is shown.

[0122] like Figure 7 As shown in the embodiments of this disclosure, the calculation result of each color channel image includes at least one fitting result; the above operation S620 may also include operations S721 to S723.

[0123] In operation S721, for a color channel image, a target window with a size of m×m is determined, centered on the target pixel in the color channel image, where m is an odd number greater than 1.

[0124] In operation S722, at least one color channel image from the other M-1 color channel images is used as a reference channel image. In the reference channel image, a reference pixel corresponding to the target pixel is determined, and a reference window with a size of m×m centered on the reference pixel is determined.

[0125] In operation S723, each pixel value in the reference window is used to perform linear fitting on each pixel value in the target window to obtain at least one fitting result of the color channel image.

[0126] In this embodiment of the disclosure, operation S723 may further include: determining the inter-channel linear relationship based on the pixel values ​​in the reference window and the pixel values ​​in the target window, wherein the inter-channel linear relationship characterizes the linear correlation between the photosensitive intensities of pixels in different color channels; and fitting the target pixels in the color channel image based on the inter-channel linear relationship to obtain at least one fitting result of the color channel image.

[0127] In this embodiment of the disclosure, determining the inter-channel linear relationship based on the pixel values ​​in the reference window and the pixel values ​​in the target window may further include: transforming the pixel values ​​in the reference window according to the initial inter-channel linear relationship to obtain multiple predicted pixel values ​​in the target window; determining the difference between the multiple predicted pixel values ​​in the target window and the pixel values ​​in the target window according to a preset loss function; and adjusting at least one parameter value included in the initial inter-channel linear relationship according to the difference.

[0128] In this embodiment of the disclosure, the image processing method may further include, after the above operation S723: for the color channel image, performing a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain the calculation result of the color channel image.

[0129] In one optional implementation, the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel; the above operation S620 may further include: for the color channel image of any one of the red, green, and blue channels, using the pixel values ​​of the color channel image of the near-infrared channel, performing linear fitting on the color channel image to obtain the calculation result of the color channel image.

[0130] In another optional implementation, the above operation S620 may further include: for the color channel image, performing a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain a first calculation result of the color channel image; and using the pixel values ​​of the color channel image of the near-infrared channel, performing a linear fit on the first calculation result to obtain a second calculation result of the color channel image, and determining the second calculation result of the color channel image as the calculation result of the color channel image.

[0131] It should be noted that the steps of the above method embodiments correspond to the actions performed by each unit in the foregoing image processing device embodiments. For the specific implementation flow of each step in the method embodiments of this disclosure, please refer to the detailed functional description of each unit in the foregoing image processing device embodiments, which will not be repeated here.

[0132] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure.

[0133] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0134] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0135] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0136] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image processing method according to the embodiments of this disclosure.

[0137] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0138] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the image processing method of the embodiments of this disclosure.

[0139] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0140] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0141] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0142] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module sub-circuit, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0145] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image processing apparatus, comprising: The extraction unit is configured to extract M color channel images corresponding to M color channels from the image to be processed. The image to be processed includes multiple pixels, where M is an integer greater than or equal to 2, and the M color channels include a red channel, a green channel, a blue channel, and a near-infrared channel. The fitting unit is configured to perform linear fitting on the pixel values ​​of each color channel image in the M color channel images using the pixel values ​​of at least one color channel image in the other M-1 color channel images, to obtain M calculation results for the M color channel images. Each calculation result includes at least one fitting result. The fitting unit is further configured to: for the color channel image, determine a target window of size m×m centered on the target pixel in the color channel image, where m is an odd number greater than 1; use at least one color channel image from the other M-1 color channel images as a reference channel image, determine a reference pixel at the position corresponding to the target pixel in the reference channel image, determine a reference window of size m×m centered on the reference pixel; and perform linear fitting on each pixel value in the target window using each pixel value in the reference window to obtain at least one fitting result for the color channel image. The fitting unit is further configured to: for the color channel image, perform a weighted summation of the pixel values ​​of the color channel image and at least one fitting result of the color channel image to obtain a first calculation result of the color channel image; use the pixel values ​​of the color channel image of the near-infrared channel to perform a linear fit on the first calculation result to obtain a second calculation result of the color channel image, and determine the second calculation result of the color channel image as the calculation result of the color channel image; as well as The merging unit is configured to merge the M calculation results to obtain the processing result of the image to be processed.

2. The image processing apparatus according to claim 1, wherein, The fitting unit is further configured to: Based on the pixel values ​​in the reference window and the pixel values ​​in the target window, a linear relationship between channels is determined, wherein the linear relationship between channels characterizes the linear correlation between the photosensitive intensities of pixels in different color channels; and Based on the linear relationship between the channels, the target pixels in the color channel image are fitted to obtain at least one fitting result for the color channel image.

3. The image processing apparatus according to claim 2, wherein, The fitting unit is further configured to: Based on the initial inter-channel linear relationship, the pixel values ​​in the reference window are transformed to obtain multiple predicted pixel values ​​in the target window; Based on a preset loss function, the difference between multiple predicted pixel values ​​in the target window and each pixel value in the target window is determined; as well as Based on the difference, adjust at least one parameter value included in the initial inter-channel linear relationship.

4. The image processing apparatus according to claim 1, wherein, The fitting unit is further configured to: For the color channel image, the pixel values ​​of the color channel image and at least one fitting result of the color channel image are weighted and summed to obtain the calculation result of the color channel image.

5. The image processing apparatus according to claim 1, wherein, The fitting unit is further configured to: For any one of the red, green, and blue color channels, the pixel values ​​of the near-infrared color channel image are used to perform linear fitting on the color channel image to obtain the calculation result of the color channel image.

6. An imaging device, comprising: An image acquisition device, configured to acquire images to be processed; as well as The image processing apparatus according to any one of claims 1-5.

7. An image processing method, comprising: Extract M color channel images corresponding to M color channels from the image to be processed. The image to be processed includes multiple pixels, where M is an integer greater than or equal to 2, and the M color channels include red channel, green channel, blue channel and near-infrared channel. For each of the M color channel images, a linear fit is performed on the pixel values ​​of that color channel image using the pixel values ​​of at least one color channel image from the other M-1 color channel images, resulting in M ​​calculation results for the M color channel images. Each calculation result includes at least one fitting result, including: for the color channel image, determining a target window with a size of m×m centered on the target pixel in the color channel image, where m is an odd number greater than 1; using at least one color channel image from the other M-1 color channel images as a reference channel image, determining a reference pixel at the position corresponding to the target pixel in the reference channel image, and determining a reference pixel at the position corresponding to the target pixel in the reference channel image. A reference window centered at a reference pixel and measuring m×m; and using the pixel values ​​in the reference window, performing linear fitting on the pixel values ​​in the target window to obtain at least one fitting result for the color channel image; for the color channel image, performing a weighted sum of the pixel values ​​of the color channel image and the at least one fitting result for the color channel image to obtain a first calculation result for the color channel image; using the pixel values ​​of the color channel image of the near-infrared channel, performing linear fitting on the first calculation result to obtain a second calculation result for the color channel image; and determining the second calculation result of the color channel image as the calculation result of the color channel image; and The M calculation results are merged to obtain the processing result of the image to be processed.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to claim 7.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to claim 7.

Images