Image processing method, apparatus and electronic device

Abstract

The application discloses an image processing method and device, electronic equipment and medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring N observation images of a shooting scene under N wave bands, wherein N is an integer greater than 4; one wave band corresponds to one observation image; estimating a first light component of the shooting scene based on the N observation images and N light constraint information; the light constraint information is used to constrain the relevance between the light distribution of the shooting scene under different wave bands; determining a first reflection component of the shooting scene based on the first light component and the N observation images; and generating a first image based on the first reflection component.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, and electronic device. Background Technology

[0002] Currently, Retinex-based image enhancement methods can be used to enhance red-green-blue (RGB) images captured by electronic devices to improve the quality of RGB images.

[0003] Among them, the Retinex-based image enhancement method can simulate the characteristics of the human visual system. By decomposing the pixel values ​​of an RGB image into illumination and reflection components, it estimates and removes the illumination component from the RGB image to restore the true details and colors of the RGB image. Specifically, the Retinex-based image enhancement method can perform illumination estimation on each channel of the RGB image using conventional filters or edge-preserving filters.

[0004] However, illumination estimation using filters has a large error, resulting in halos, artifacts, or color distortion in the enhanced image. In other words, the image enhancement effect of related techniques is relatively poor. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, apparatus, and electronic device that can accurately estimate illumination, thereby avoiding problems such as halo, artifacts, and color distortion caused by inaccurate illumination estimation, and thus improving the image quality of enhanced images.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring N observation images of a shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image; estimating a first illumination component of the shooting scene based on the N observation images and N illumination constraint information; the illumination constraint information being used to constrain the correlation between the illumination distribution of the shooting scene in different bands; determining a first reflection component of the shooting scene based on the first illumination component and the N observation images; and generating a first image based on the first reflection component.

[0007] Thus, since N illumination constraint information is used when estimating the first illumination component of the shooting scene to constrain the correlation between the illumination distribution of the shooting scene in different wavelengths, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located. That is, the illumination component can be accurately estimated. Therefore, the problems of halo, artifact and color distortion caused by inaccurate illumination estimation can be avoided from the source, thereby improving the image quality of the generated first image.

[0008] In some possible implementations, the aforementioned illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between illumination distributions of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of illumination distributions of the shooting scene in one wavelength band.

[0009] Thus, since each observed image corresponds to a first constraint information for ensuring consistency of illumination components across bands and a second constraint information for ensuring spatial smoothness of illumination components, on the one hand, the edges of illumination components in different bands of the shooting scene can be made to be consistent; on the other hand, the second constraint information can prevent the illumination components of the shooting scene from jumping in each band, thus ensuring that the illumination components in each band remain spatially smooth. This ensures that the estimated first illumination component is consistent with the illumination distribution in the real illumination field where the shooting scene is located.

[0010] In some possible implementations, estimating the first illumination component of the shooting scene based on the N observed images and N illumination constraint information includes:

[0011] Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined;

[0012] Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene;

[0013] Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene;

[0014] The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

[0015] Thus, since the initial illumination component can be updated based on N observed images, the aforementioned initial reflection component, and N illumination constraint information, and then the initial reflection component can be updated based on N observed images and the second illumination component, it can be ensured that the N illumination constraint information directly correlates and constrains the illumination components under different wavelengths, and can also indirectly constrain the reflection component of the shooting scene. Therefore, by alternately updating the illumination component and the reflection component, the illumination component and the reflection component of the shooting scene can gradually approach the real illumination field of the shooting scene, thereby improving the accuracy of the estimation of the illumination component of the shooting scene.

[0016] In some possible implementations, the above-mentioned updating of the initial reflection component based on the N observed images and the second illumination component to obtain the second reflection component of the shooting scene includes:

[0017] Based on the N observed images, the second illumination component, and the N reflection constraint information, the initial reflection component is updated to obtain the second reflection component of the shooting scene;

[0018] The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

[0019] Thus, since the initial reflection component can be updated based on N observed images, the second illumination component, and N reflection constraint information, the updated reflection component can be controlled to remain spatially smooth through the reflection constraint information. This ensures that the updated reflection component better matches the true reflected illumination of the physical illumination field of the shooting scene. When further updating the illumination component of the shooting scene based on the updated reflection component, the illumination component can be guided to move closer to the true illumination in the physical illumination field, thereby further improving the estimation accuracy of the illumination component.

[0020] In some possible implementations, the acquisition of N observation images of the shooting scene in N bands includes:

[0021] The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band.

[0022] The RGB image of the scene being photographed is acquired using an RGB sensor.

[0023] Based on the RGB image, the N mosaic images are de-mosaiced to obtain N observation images.

[0024] Thus, since RGB images can be used to guide the de-mosaicing of N mosaic images acquired by a multispectral sensor, that is, the high spatial resolution information of RGB images can be used to restore the missing texture details of the mosaic images, the N observation images obtained after de-mosaicing can provide richer spatial details for subsequent illumination component estimation, thereby ensuring the accuracy of illumination estimation.

[0025] In some possible implementations, the above-mentioned de-mosaic processing is performed on the N mosaic images based on the RGB image to obtain N observation images, including:

[0026] Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images;

[0027] The N upsampled images are registered with the RGB image to obtain N registered images;

[0028] Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image;

[0029] The guide image corresponding to each registration image is a channel image of the RGB image.

[0030] Thus, since RGB images can be used to sequentially upsample, register, and guide filtering N mosaic images, the high-frequency details of the RGB images can be injected into the N mosaic images under the N bands. This can significantly improve the spatial resolution of the N observation images obtained after de-mosaicing, thereby ensuring that the N observation images provide richer and finer edges of illumination changes for illumination estimation, and thus improving the accuracy of illumination estimation.

[0031] In some possible implementations, before generating the first image based on the first reflection component, the method further includes:

[0032] The first reflection component undergoes a first processing step;

[0033] The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

[0034] Thus, on the one hand, denoising based on low-rank spectral priors can remove high-frequency, sparse noise from the first reflection component while preserving the object's true, low-frequency reflection signal, thereby improving the reliability of the first reflection component. On the other hand, detail enhancement based on spatially guided filtering can suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the object's true texture and edge details, preventing the image from becoming overly smooth due to denoising, thus ensuring that the final enhancement result has better visual clarity and spatial detail representation. Furthermore, spectral interpolation of the first reflection component can reconstruct a more continuous spectral curve from discretely sampled reflectance, making subsequent color space conversion calculations more accurate, helping to reduce color calculation errors caused by insufficient spectral sampling, and improving the color fidelity of the enhanced image. Therefore, by performing targeted first processing on the first reflection component, noise can be suppressed in the spectral dimension to improve data quality, or details can be enhanced in the spatial dimension to improve visual clarity. This results in an enhanced image that is not only more realistic in color, but also cleaner and sharper visually, thus ensuring that the image quality of the enhanced image is further optimized.

[0035] Secondly, embodiments of this application provide an image processing apparatus, which includes:

[0036] The acquisition module is used to acquire N observation images of the shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image.

[0037] Processing module, used for:

[0038] Based on the N observed images and N illumination constraint information acquired by the acquisition module, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands.

[0039] Based on the first illumination component and the N observed images, the first reflection component of the shooting scene is determined;

[0040] A first image is generated based on the first reflection component.

[0041] Thus, since N illumination constraint information is used when estimating the first illumination component of the shooting scene to constrain the correlation between the illumination distribution of the shooting scene in different wavelengths, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located. That is, the illumination component can be accurately estimated. Therefore, the problems of halo, artifact and color distortion caused by inaccurate illumination estimation can be avoided from the source, thereby improving the image quality of the generated first image.

[0042] In some possible implementations, the aforementioned illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between illumination distributions of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of illumination distributions of the shooting scene in one wavelength band.

[0043] Thus, since each observed image corresponds to a first constraint information for ensuring consistency of illumination components across bands and a second constraint information for ensuring spatial smoothness of illumination components, on the one hand, the edges of illumination components in different bands of the shooting scene can be made to be consistent; on the other hand, the second constraint information can prevent the illumination components of the shooting scene from jumping in each band, thus ensuring that the illumination components in each band remain spatially smooth. This ensures that the estimated first illumination component is consistent with the illumination distribution in the real illumination field where the shooting scene is located.

[0044] In some possible implementations, the above processing module is specifically used for:

[0045] Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined;

[0046] Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene;

[0047] Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene;

[0048] The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

[0049] Thus, since the initial illumination component can be updated based on N observed images, the aforementioned initial reflection component, and N illumination constraint information, and then the initial reflection component can be updated based on N observed images and the second illumination component, it can be ensured that the N illumination constraint information directly correlates and constrains the illumination components under different wavelengths, and can also indirectly constrain the reflection component of the shooting scene. Therefore, by alternately updating the illumination component and the reflection component, the illumination component and the reflection component of the shooting scene can gradually approach the real illumination field of the shooting scene, thereby improving the accuracy of the estimation of the illumination component of the shooting scene.

[0050] In some possible implementations, the above processing module is specifically used to update the initial reflection component based on the N observed images, the second illumination component, and the N reflection constraint information to obtain the second reflection component of the shooting scene;

[0051] The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

[0052] Thus, since the initial reflection component can be updated based on N observed images, the second illumination component, and N reflection constraint information, the updated reflection component can be controlled to remain spatially smooth through the reflection constraint information. This ensures that the updated reflection component better matches the true reflected illumination of the physical illumination field of the shooting scene. When further updating the illumination component of the shooting scene based on the updated reflection component, the illumination component can be guided to move closer to the true illumination in the physical illumination field, thereby further improving the estimation accuracy of the illumination component.

[0053] In some possible implementations, the aforementioned acquisition module is specifically used for:

[0054] The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band.

[0055] The RGB image of the scene being photographed is acquired using an RGB sensor.

[0056] The processing module is further configured to perform de-mosaic processing on the N mosaic images acquired by the acquisition module based on the RGB image acquired by the acquisition module, to obtain N observation images.

[0057] Thus, since RGB images can be used to guide the de-mosaicing of N mosaic images acquired by a multispectral sensor, that is, the high spatial resolution information of RGB images can be used to restore the missing texture details of the mosaic images, the N observation images obtained after de-mosaicing can provide richer spatial details for subsequent illumination component estimation, thereby ensuring the accuracy of illumination estimation.

[0058] In some possible implementations, the above processing module is specifically used for:

[0059] Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images;

[0060] The N upsampled images are registered with the RGB image to obtain N registered images;

[0061] Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image;

[0062] The guide image corresponding to each registration image is a channel image of the RGB image.

[0063] Thus, since RGB images can be used to sequentially upsample, register, and guide filtering N mosaic images, the high-frequency details of the RGB images can be injected into the N mosaic images under the N bands. This can significantly improve the spatial resolution of the N observation images obtained after de-mosaicing, thereby ensuring that the N observation images provide richer and finer edges of illumination changes for illumination estimation, and thus improving the accuracy of illumination estimation.

[0064] In some possible implementations, the above-mentioned processing module is further configured to perform a first processing on the first reflection component before generating the first image based on the first reflection component;

[0065] The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

[0066] Thus, on the one hand, denoising based on low-rank spectral priors can remove high-frequency, sparse noise from the first reflection component while preserving the object's true, low-frequency reflection signal, thereby improving the reliability of the first reflection component. On the other hand, detail enhancement based on spatially guided filtering can suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the object's true texture and edge details, preventing the image from becoming overly smooth due to denoising, thus ensuring that the final enhancement result has better visual clarity and spatial detail representation. Furthermore, spectral interpolation of the first reflection component can reconstruct a more continuous spectral curve from discretely sampled reflectance, making subsequent color space conversion calculations more accurate, helping to reduce color calculation errors caused by insufficient spectral sampling, and improving the color fidelity of the enhanced image. Therefore, by performing targeted first processing on the first reflection component, noise can be suppressed in the spectral dimension to improve data quality, or details can be enhanced in the spatial dimension to improve visual clarity. This results in an enhanced image that is not only more realistic in color, but also cleaner and sharper visually, thus ensuring that the image quality of the enhanced image is further optimized.

[0067] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0068] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0069] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0070] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0071] In this embodiment, N observation images of the shooting scene in N bands can be acquired, where N is an integer greater than 4; one band corresponds to one observation image; based on the N observation images and N illumination constraint information, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands; based on the first illumination component and the N observation images, the first reflection component of the shooting scene is determined; based on the first reflection component, a first image is generated. Through this scheme, since N illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands when estimating the first illumination component, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located, that is, accurate estimation of the illumination component can be achieved. Therefore, problems such as halo, artifacts, and color distortion caused by inaccurate illumination estimation can be avoided at the source, thus improving the image quality of the generated first image. Attached Figure Description

[0072] Figure 1 This is a schematic flowchart of an image processing method provided in some embodiments of this application;

[0073] Figure 2 This is a schematic diagram of the structure of a multispectral sensor provided in some embodiments of this application;

[0074] Figure 3 This is a schematic diagram of the structure of another multispectral sensor provided in some embodiments of this application;

[0075] Figure 4 This is a schematic diagram of the demosaic process in the image processing method provided in some embodiments of this application;

[0076] Figure 5 This is another schematic flowchart of an image processing method provided in some embodiments of this application;

[0077] Figure 6 This is yet another schematic flowchart of an image processing method provided in some embodiments of this application;

[0078] Figure 7 These are schematic diagrams of the structure of an image processing apparatus provided in some embodiments of this application;

[0079] Figure 8 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application;

[0080] Figure 9 This is another structural schematic diagram of an electronic device provided in some embodiments of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0082] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0083] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0084] The following section provides explanations for the terms and concepts used in this application specification.

[0085] A Fabry-Pérot cavity (FP) is an optical cavity consisting of two parallel mirrors separated by a transparent medium. The working principle of a miniature FP cavity is based on the phenomenon of light interference; light reflects back and forth between the two mirrors, creating multi-beam interference.

[0086] By changing the cavity length of the FP cavity, the center wavelength of its transmission peak can be changed continuously or in steps. For example, the cavity length of the FP cavity can be changed using MEMS voltage tuning.

[0087] A multispectral sensor equipped with an FP cavity can operate in a time-series manner. When capturing the same scene, it rapidly scans a series of preset wavelengths. At each wavelength scan, all pixels in the sensor simultaneously respond to light in that band. After completing the scan of multiple bands, a complete, full-resolution two-dimensional observation image is generated for each band. Stacking these two-dimensional observation images sequentially forms a spatially complete "spectral cube" (x, y, λ); where (x, y) represents a pixel in the multispectral sensor, and λ represents a band.

[0088] The three channels of a Red, Green, and Blue (RGB) image are the "color language" of the RGB image. The three channels of an RGB image include:

[0089] R channel: Represents the intensity of the red component in an RGB image.

[0090] G channel: Represents the intensity of the green component in an RGB image.

[0091] B channel: Represents the intensity of the blue component in an RGB image.

[0092] It should be noted that the RGB images mentioned above are the processed images obtained from the original RGB sensor using an image signal processor (ISP). The ISP's processing of the original image includes, but is not limited to, depixelation, white balance, color correction, and gamma encoding. Simply put, the three channels of an RGB image equal the result after ISP processing.

[0093] The three-channel response of an RGB sensor refers to the signal actually measured at each pixel after the Bayer filter array covering the RGB image sensor. For example, an RGB image sensor can be a CMOS sensor.

[0094] Photoelectric sensors are monochromatic, sensitive to light intensity but not to color. By covering the surface of the photoelectric sensor with a tiny array of red, green, and blue filters, typically arranged in an RGGB pattern, each pixel receives light only within a specific wavelength range. Specifically, the three-channel response of an RGB sensor includes:

[0095] R-pixel response: An R-pixel represents a pixel covered by a red filter. An R-pixel allows red light with wavelengths within the range of 600nm to pass through, while blocking most green and blue light. The R-pixel response value reflects the voltage or charge number converted from the number of photons in the red light band by the G-pixel, thus reflecting the intensity of the red light band.

[0096] G-pixel response: A G-pixel represents a pixel covered by a red filter. G-pixels allow green light with wavelengths within the wavelength range [500nm, 600nm] to pass through. The response value of a G-pixel reflects the voltage or charge number converted from the number of photons in the green light band by the G-pixel, thus reflecting the light intensity in the green light band.

[0097] B-pixel response: A B-pixel represents a pixel covered by a blue filter. B-pixels allow blue light with wavelengths within the range [400nm, 500nm] to pass through. The response value of a B-pixel reflects the voltage or charge number converted from the number of photons in the blue light band by the B-pixel, thus reflecting the light intensity in the blue light band.

[0098] The three-channel response of an RGB sensor is the "spectral energy reading" of the three channels of the RGB sensor. Simply put, the three-channel response of an RGB sensor equals the raw image data acquired by the RGB sensor.

[0099] The image processing methods, apparatus, electronic devices, and media provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0100] The image processing method provided in this application embodiment can be applied in image capture or video capture scenarios.

[0101] For example, the image processing method provided in this application embodiment is applied to the following shooting scenarios:

[0102] Low-light environments: nighttime monitoring and astronomical observation;

[0103] High dynamic range scenarios: backlighting, mixed indoor and outdoor lighting;

[0104] Color-sensitive applications: digitization of artworks, medical imaging;

[0105] Remote sensing image processing scenarios: satellite imagery, aerial photography.

[0106] It should be noted that in the embodiments of this application, "multispectral sensor", "area array multispectral sensor" and "multispectral imaging sensor" have the same meaning, all referring to a sensor capable of acquiring N observation images in N bands.

[0107] The image processing method provided in this application is executed by an image processing device, which can be an electronic device, or a functional module or entity within an electronic device. This application does not limit the specific implementation of this method. The following will use an image processing device as an example to illustrate the image processing method provided in this application.

[0108] This application provides an image processing method, such as... Figure 1As shown, the image processing method provided in this application embodiment may include the following steps 101 to 104.

[0109] Step 101: The image processing device acquires N observation images of the shooting scene under N multi-wavelength conditions.

[0110] One band corresponds to one observation image, and N is an integer greater than 4.

[0111] In some embodiments of this application, the value of N can be 8, 12, 16, 25, 32, 50, or 64, etc.

[0112] In some embodiments of this application, the image processing device can acquire N observation images of a shooting scene using a multispectral sensor. The process by which the image processing device acquires these N observation images varies depending on the type of multispectral sensor.

[0113] In some embodiments of this application, the aforementioned N observation images can also be referred to as the spectral response values ​​of the shooting scene at the full image pixel size for N bands. In other words, the observation image of each band is a complete, full-resolution two-dimensional observation image. Full resolution means that the resolution of the observation image is the same as the resolution of the multispectral sensor.

[0114] For example, the size of a multispectral sensor is If the pixels are N, then the size of each N observed image is... m and k are positive integers.

[0115] In some embodiments of this application, the multispectral sensor includes at least four bands that can cover the visible light and possibly the near-infrared region.

[0116] It is understood that in the embodiments of this application, N can represent the number of bands corresponding to the multispectral sensor, and N can be simply referred to as the number of bands below.

[0117] In some embodiments of this application, each of the N bands is a narrow spectral band. That is, the N bands can also be referred to as N spectral bands or N band ranges.

[0118] In some embodiments of this application, the multispectral sensor has N color channels, each color channel corresponding to a wavelength band. A color channel can be simply referred to as a channel.

[0119] For example, the multispectral sensor can be an N-channel multispectral sensor, where N is an integer greater than 4.

[0120] In some embodiments of this application, the above-mentioned N bands are continuous bands, or they can be discontinuous bands. Of course, some bands can be continuous and some can be discontinuous. This application does not limit the embodiments.

[0121] For example, taking a 16-channel multispectral sensor as an example, the above N bands can be continuous narrow bands, as shown in Table 1.

[0122] Table 1

[0123]

[0124] For example, taking a 16-channel multispectral sensor as an example, as shown in Table 2, the above N bands can be discontinuous narrow bands.

[0125] Table 2

[0126]

[0127] In some embodiments of this application, the multispectral sensor may be any of the following:

[0128] Method 1: A planar multispectral sensor with a narrowband filter array, hereinafter referred to as a narrowband multispectral sensor. A narrowband multispectral sensor may include multiple pixel units, each pixel unit having a narrowband filter array integrated on its surface. The number of filters in the narrowband filter array is the same as the number of pixels in the pixel unit, and each filter in the narrowband filter array corresponds to a narrow band, such as one of the narrow bands in Table 1 or Table 2 above.

[0129] For example, such as Figure 2 As shown, the narrowband multispectral sensor 20 includes multiple pixel units 21, each pixel unit 21 includes N pixels 22 distributed in an array, and a narrowband filter array is disposed on the surface of each pixel unit 21. The narrowband filter array includes N narrowband filters, and each narrowband filter corresponds to a pixel and a wavelength band.

[0130] It is understandable that for each of the N bands mentioned above, the raw image acquired by the narrowband multispectral sensor is discontinuous spectral data. Specifically, since each color channel occupies only one pixel in a pixel unit, for each band, one pixel in each pixel unit of the narrowband multispectral sensor can acquire the spectral response value of that band to the scene being captured. Therefore, the raw image acquired by the narrowband multispectral sensor is a set of mosaic images, which includes N two-dimensional mosaic images corresponding one-to-one with the N color channels. Each two-dimensional mosaic image is a discrete spectral response value of one color channel to the scene being captured.

[0131] For example, taking a 16-channel narrowband multispectral sensor as an example, if the size of the narrowband multispectral sensor is... If there are 1 pixel, then the narrowband multispectral sensor acquires a mosaic array, the shape of which is: The mosaic array includes 16 two-dimensional mosaic images that correspond one-to-one with the 16 color channels. Each two-dimensional mosaic image is a color channel’s discrete spectral response value to the shooting scene.

[0132] In this embodiment, it is assumed that the pixel unit in the N-channel narrowband multispectral sensor is... 1 pixel. If the size of the narrowband multispectral sensor is 1 In pixels, the narrowband multispectral sensor captures images of a scene, resulting in 16 mosaic images, also known as a mosaic array. However, the spatial resolution of each mosaic image is reduced to a fraction of the original resolution of the narrowband multispectral sensor. ,in, p and q are both positive integers.

[0133] For example, a set of mosaic images acquired by a 16-channel narrowband multispectral sensor has its spatial resolution reduced to the original resolution of the narrowband multispectral sensor. .

[0134] Method 2 involves a multispectral array sensor with a miniature FP cavity, hereinafter referred to as an FP cavity type multispectral sensor. For example... Figure 3 As shown, each pixel 31 in the FP-cavity multispectral sensor 30 is provided with a miniature FP cavity. For further descriptions of FP cavities or area array multispectral sensors with miniature FP cavities, please refer to the relevant descriptions in the glossary section above.

[0135] In some embodiments of this application, taking an N-channel FP cavity multispectral sensor as an example, if the size of the FP cavity multispectral sensor is... If there are 100 pixels, then the image data acquired by the FP cavity multispectral sensor is a full-pixel spectral cube, and the shape of this full-pixel spectral cube is: For each of the N bands, the full-pixel spectral cube corresponds to a complete, full-resolution two-dimensional observation image. Here, the "full-pixel spectral cube" can also be called a "continuous array".

[0136] For example, taking a 16-channel FP cavity sensor as an example, the size of the FP cavity sensor is... If there are 100 pixels, the image data acquired by the FP cavity sensor is a continuous array, and the shape of this continuous data is... The light field response values ​​of each color channel to the shooting scene.

[0137] It is understandable that since the N observation images acquired by the FP cavity sensor include a two-dimensional observation image of full pixels corresponding to each band, there is no need for demosaic operation, which can save processing resources and processing time and reduce power consumption.

[0138] In some embodiments of this application, in mode 1, step 101 may include steps 101A to 101C.

[0139] Step 101A: The image processing device acquires N mosaic images of the scene in N bands using a multispectral sensor with a narrowband filter array.

[0140] Step 101B: The image processing device acquires an RGB image of the above-mentioned shooting scene using an RGB sensor.

[0141] In some embodiments of this application, each band corresponds to a mosaic image.

[0142] It is understandable that after a multispectral sensor with a narrowband filter array captures an image of a scene, it can obtain a raw mosaic image at the sensor level. This raw mosaic image is an image composed of data from all channels output by the multispectral sensor, interwoven together, such as... Figure 4 The original mosaic image 40 is shown in the figure. The original mosaic image 40 is split according to the bands to obtain N mosaic images.

[0143] In some embodiments of this application, the above N mosaic images can also be referred to as sparse images corresponding to each of the N bands.

[0144] In some embodiments of this application, the above-mentioned N mosaic images can constitute a mosaic image group or a set of mosaic images.

[0145] In some embodiments of this application, the time interval between the acquisition of N mosaic images and RGB images is less than or equal to a first time threshold.

[0146] For example, the initial time threshold can be 0, 0.05ms, 1ms, 5ms, etc., which are not limited in this application. The acquisition time interval between the mosaic image and the RGB image should be minimized as much as possible to ensure that N mosaic images and RGB images correspond to the same physical light field conditions.

[0147] In some embodiments of this application, the RGB sensor and the area array multispectral sensor are disposed on the same electronic device.

[0148] Step 101C: The image processing device performs de-mosaic processing on N mosaic images based on the RGB image to obtain N observation images.

[0149] In some embodiments of this application, such as Figure 4 As shown, since the mosaic image includes discrete spectral response values ​​of the scene at each band, it is necessary to perform de-mosaic processing on the multi-band spectral mosaic image to obtain full-image multispectral information 41 of the scene at each band, i.e., "spectral cube" data. That is, the above N observed images, where N is an integer greater than 4. In pixels.

[0150] It is understandable that if N bands are N consecutive bands, then as... Figure 4 As shown, the continuous spectral response curve 42 of the shooting scene can be extracted from N observed images 41.

[0151] In some embodiments of this application, the image processing apparatus can utilize the spatial details of a high-resolution RGB image to guide the restoration of spatial details in a mosaic image.

[0152] It can be understood that performing de-mosaic processing on the above mosaic image is equivalent to repairing the discontinuous spectral response values ​​of each band originally acquired by the narrowband multispectral sensor through pixel interpolation, so as to obtain the spectral response value of each channel to the shooting scene at the full pixel size.

[0153] The “spectral response value of each color channel to the shooting scene at the full pixel size” can be: each color channel covers only one pixel in each pixel unit of the area array multispectral sensor, and through interpolation, each pixel position has the spectral response values ​​of all N channels.

[0154] For example, taking a 16-channel narrowband multispectral sensor as an example, the mosaic array acquired by the narrowband multispectral sensor is... Then, the 16 observation images obtained after removing the mosaic can form a shape of The image set. This image set can be represented as a spectral cube. ,in, The location is based on pixels. Indicates the spectral band. The position of each pixel in this spectral cube. Each has 16 channels of spectral response values, meaning each pixel location... This corresponds to a 16-dimensional spectral response vector.

[0155] It is understandable that after de-mosaicing N mosaic images, the size of the spectral response value corresponding to each band is restored to... Pixel.

[0156] In some embodiments of this application, the size of the RGB sensor is the same as the size of the multispectral sensor, such as both being... 1 pixel.

[0157] It is understandable that the purpose of de-mosaicing a mosaic image is to recover the full-image pixel-level spectral response values ​​for each band.

[0158] Thus, since RGB images can be used to guide the de-mosaicing of N mosaic images acquired by a multispectral sensor, that is, the high spatial resolution information of RGB images can be used to restore the missing texture details of the mosaic images, the N observation images obtained after de-mosaicing can provide richer spatial details for subsequent illumination component estimation, thereby ensuring the accuracy of illumination estimation.

[0159] In some embodiments of this application, step 101C may include steps 101C1 to 101C3.

[0160] Step 101C1: The image processing device upsamples N mosaic images according to the size of the RGB images to obtain N upsampled images.

[0161] In some embodiments of this application, the image processing device upsamples N mosaic images according to the size of the RGB image using a first interpolation algorithm to obtain N upsampled images that correspond one-to-one with the N mosaic images.

[0162] The first interpolation algorithm may include any of the following: bilinear interpolation, adaptive interpolation, or deep learning-based interpolation algorithm.

[0163] Step 101C2: The image processing device registers the N upsampled images with the RGB image to obtain N registered images.

[0164] It is understandable that, due to the different physical positions of the RGB sensor and the multispectral sensor in the electronic device, such as a certain offset, there is a slight spatial offset in the viewing angle of the images acquired by the two sensors. Therefore, it is necessary to register the upsampled images with the RGB images to align the above N upsampled images with the RGB images at the pixel level.

[0165] In some embodiments of this application, the image processing device registers N upsampled images with an RGB image. This can be achieved by: the image processing device calculating the affine transformation matrix corresponding to the N upsampled images based on the RGB channel spectral response of the RGB image; and then registering the N upsampled images based on the affine transformation matrix to obtain N registered images.

[0166] For example, an image processing device can select three upsampled images from N upsampled images whose bands most closely match the spectral responses of the RGB channels. These three upsampled images are then matched with feature points of the R, G, and B channels of the RGB image, respectively, to obtain multiple pairs of matched feature points. Based on these matched feature points, an affine transformation matrix is ​​calculated. This affine transformation matrix is ​​then applied to each of the N upsampled images to obtain N registered images. These N registered images are spatially aligned with the RGB image.

[0167] Step 101C3: The image processing device performs guided filtering on each registration image based on the guide image corresponding to each registration image to obtain the observation image corresponding to each registration image.

[0168] Each registration image corresponds to a guide image, which is a channel image of an RGB image.

[0169] In some embodiments of this application, the image processing device can select a guide image corresponding to each registration image based on the similarity between the band corresponding to each registration image and the bands of the RGB three channels of the RGB image.

[0170] For example, if the wavelength difference between the center wavelength of a registered image and the center wavelength of the R channel of an RGB image is the smallest, then the R channel image of the RGB image is used as the guide image.

[0171] If the wavelength difference between the center wavelength of a registered image and the center wavelength of the G channel of the RGB image is the smallest, then the G channel image of the RGB image is used as the guide image.

[0172] If the wavelength difference between the center wavelength of a band-registered image and the center wavelength of the B channel of the RGB image is the smallest, then the B channel image of the RGB image is used as the guide image.

[0173] Thus, since RGB images can be used to sequentially upsample, register, and guide filtering N mosaic images, the high-frequency details of the RGB images can be injected into the N mosaic images under the N bands. This can significantly improve the spatial resolution of the N observation images obtained after de-mosaicing, thereby ensuring that the N observation images provide richer and finer edges of illumination changes for illumination estimation, and thus improving the accuracy of illumination estimation.

[0174] Method 2: The multispectral sensor is an area array multispectral sensor with an FP cavity.

[0175] In some embodiments of this application, step 101B above may include step 101B4 below.

[0176] Step 101B4: The image processing device acquires N observation images of the scene being photographed using a multispectral sensor with an FP cavity.

[0177] In some embodiments of this application, the multispectral sensor can be pre-calibrated with dark current, flat field, and spectral response to obtain dark current data, flat field data, and spectral response errors. Thus, after acquiring multi-band raw spectral data of the scene using the multispectral sensor, the raw multi-band data can be pre-processed, such as with dark current, flat field, and spectral response calibration, to ensure data accuracy and consistency.

[0178] In Method 1, the original multi-band spectral data can be a multi-band mosaic image, while in Method 2, it can be N observation images acquired by an FP-cavity array multispectral sensor.

[0179] Step 102: The image processing device estimates the first illumination component of the shooting scene based on N observed images and N illumination constraint information.

[0180] The aforementioned lighting constraint information can be used to constrain the correlation between lighting distributions in different wavelength bands of the shooting scene.

[0181] In some embodiments of this application, N illumination constraint information items are associated with N observation images in a one-to-one manner, that is, each observation image corresponds to one illumination constraint information item.

[0182] In some embodiments of this application, N bands correspond one-to-one with N illumination constraint information, that is, each observation image corresponds to one illumination constraint information.

[0183] In some embodiments of this application, the aforementioned N illumination constraint information can be used to constrain the illumination components corresponding to different bands to tend towards consistency in spatial gradient. In other words, the aforementioned N illumination constraint information can utilize the inter-band structural correlation to constrain the illumination edges corresponding to different bands to tend towards consistency.

[0184] For example, illumination constraint information can force similar spatial gradient structures between the estimated illumination components λ and λ' in different bands. The physical basis for this is that the illumination conditions, such as shadow and highlight distributions, are the same for the same scene under different narrow bands. Therefore, this illumination constraint information provides additional, cross-band illumination correction signals for N observed images.

[0185] Specifically, when the illumination estimation of an observed image in a certain band λ is erroneous at a certain edge due to noise or texture interference, the correct edge information in other bands λ' can be obtained through the illumination constraint information corresponding to that band λ, which can reduce or eliminate the error, thus "pulling" the edge of the illumination estimation corresponding to band λ back to the correct position. This is equivalent to using multi-band spectral information, i.e., N observed images, for multi-view cross-validation and joint correction, which greatly improves the robustness and accuracy of illumination estimation, especially at edges.

[0186] In some embodiments of this application, each illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in each band and the illumination distribution in the remaining N-1 bands. The remaining N-1 bands refer to all bands other than each of the N bands.

[0187] For example, if N=16, representing bands 1 to 16, then: the illumination constraint information corresponding to band 1 is used to constrain: the correlation between the illumination distribution of the shooting scene in band 1 and the illumination distribution in band 2, the correlation between the illumination distribution of the shooting scene in band 1 and the illumination distribution in band 3, ..., and the correlation between the illumination distribution of the shooting scene in band 1 and the illumination distribution in band 16.

[0188] In some embodiments of this application, the aforementioned illumination constraint information may include first constraint information and second constraint information. The first constraint information is used to constrain the correlation between illumination distributions of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of illumination distributions of the shooting scene in one wavelength band.

[0189] For example, each illumination constraint information includes a first constraint information and a second constraint information. The first constraint information can be used to constrain the correlation between the illumination distribution of the shooting scene in each band and the illumination distribution in the remaining N-1 bands. The remaining N-1 bands refer to all bands other than each of the N bands. The second constraint information can be used to constrain the spatial smoothness of the illumination distribution of the shooting scene in a band.

[0190] In some embodiments of this application, the aforementioned second constraint information can be used to constrain the illumination components of the shooting scene in a certain band to remain spatially smooth, so as to suppress the generation of edge abrupt changes in the illumination components.

[0191] In some embodiments of this application, the first constraint information may also be referred to as cross-band consistency constraint information, and the second constraint information may also be referred to as illumination spatial smoothing constraint information.

[0192] In some embodiments of this application, the aforementioned second constraint information can be based on the physical prior that the illumination component in a natural scene typically changes slowly in space, and can be used to constrain non-physical, abrupt, or discontinuous brightness jumps in the illumination component, thereby avoiding halo artifacts in the final generated image, such as the first image below.

[0193] It can be understood that the essence of the second constraint information is the introduction of strong prior knowledge regarding the piecewise smoothness of the illumination components in space. It mathematically forces the estimated illumination field to tend towards flatness by penalizing the gradients of the illumination components, such as the total variation of the illumination components, allowing only necessary jumps at the actual geometric boundaries of the object. This directly avoids and suppresses the illumination components corresponding to each band from crossing the object's edge, thus suppressing the generation of "halos" from the source.

[0194] Thus, since each observed image corresponds to a first constraint information for ensuring consistency of illumination components across bands and a second constraint information for ensuring spatial smoothness of illumination components, on the one hand, the edges of illumination components in different bands of the shooting scene can be made to be consistent; on the other hand, the second constraint information can prevent the illumination components of the shooting scene from jumping in each band, thus ensuring that the illumination components in each band remain spatially smooth. This ensures that the estimated first illumination component is consistent with the illumination distribution in the real illumination field where the shooting scene is located.

[0195] In some embodiments of this application, the first illumination component mentioned above includes N illumination components of the shooting scene in N bands, that is, each band corresponds to one illumination component.

[0196] In some embodiments of this application, the image processing device can iteratively estimate the first illumination component of the above-mentioned shooting scene based on N observed images and N illumination constraint information through a multi-band joint estimation method.

[0197] In some embodiments of this application, the aforementioned illumination constraint information includes preset weight coefficients and constraint forms, which are determined before iterative solution and remain unchanged throughout the solution process, used to guide the update direction of the illumination components. Specifically, the first constraint information corresponds to the similarity constraint between illumination gradients of different wavelength bands, and the second constraint information corresponds to the spatial smoothness constraint of each wavelength band's illumination itself.

[0198] In some embodiments of this application, the image processing device can, within the Retinex framework, use the illumination constraint information corresponding to N observed images as prior information or constraint conditions to directly participate in the estimation of illumination components, thereby improving the accuracy of illumination estimation.

[0199] It is understandable that imaging results of the same scene at different spectral bands share the same physical illumination field. Therefore, when estimating illumination components, by jointly inverting N observation images at N spectral bands, the spectral correlation between spectral bands can be fully utilized to correct the illumination edges in the observation images corresponding to different spectral bands, thereby improving the accuracy of illumination component estimation for the scene.

[0200] In some embodiments of this application, the first illumination component of the shooting scene can be configured as a three-dimensional array, hereinafter referred to as the illumination component array, and the shape of the illumination component array is as follows: The first illumination component contains the illumination components of the shooting scene in the N-band, which can provide accurate illumination priors for subsequent Retinex decomposition. The first illumination component can characterize the illumination distribution features in the shooting scene, where N is the number of bands.

[0201] In some embodiments of this application, step 102 may include steps 102A to 102D.

[0202] Step 102A: The image processing device determines the initial illumination component and initial reflection component of the shooting scene based on N observed images.

[0203] In some embodiments of this application, in one approach, the image processing device can determine the initial illumination component of the shooting scene through the following steps:

[0204] Step 11: The image processing device processes each of the N observed images. Each pixel value is uniformly added with a minimum value greater than 0, and then for each observed image... Taking the natural logarithm yields the observed image. Corresponding logarithmic field image This minimum value can prevent pixel values ​​from appearing after conversion to the logarithmic domain. In some embodiments of this application, the selection of this minimum value needs to consider the pixel value quantization bits of the multispectral sensor and the dynamic range corresponding to N observed images. For example, for a 16-channel multispectral sensor, the minimum value can be 0.001, 0.0015, etc.

[0205] Step 12: The image processing device determines a maximum illumination envelope image based on N logarithmic domain images.

[0206] Specifically, in the logarithmic domain, for each pixel location, iterate through N logarithmic domain images to find the maximum logarithmic value at each pixel location. The pixel values ​​corresponding to the maximum logarithmic values ​​at all pixel locations can then be used to generate a single-channel image. This single-channel image The spatial resolution is the same as that of the logarithmic domain image, i.e., the size is the same. This single-channel image... This can be expressed as follows: At each spatial point, take the pixel value with the largest logarithmic value among N logarithmic domain images, and use it as the initial estimate of the "illuminance envelope" of that spatial point.

[0207] For example, taking N=16, since the 16 logarithmic domain images are of the same size, there are 16 logarithmic values ​​for each pixel position. These 16 logarithmic values ​​come from the 16 logarithmic domain images respectively. In this way, the image processing device can extract the maximum logarithmic value at all pixel positions and generate a maximum illumination envelope image from the pixel values ​​corresponding to all the maximum logarithmic values.

[0208] Step 13: The image processor uses the reference image to perform guided filtering on the maximum illumination envelope image to obtain a basic illumination image.

[0209] It is understandable that the base lighting image retains large edge information such as shadows from the maximum lighting envelope image, while erasing texture details from the maximum lighting envelope image.

[0210] The reference image mentioned above can be an RGB image captured by an RGB sensor in the same shooting scene, or it can be one of N logarithmic domain images; this application does not impose any limitations on it.

[0211] Step 14: The image processor copies the basic lighting image N times to obtain the initial lighting components of the shooting scene.

[0212] Step 15: The image processing device calculates the initial reflection component of the scene to be captured based on the initial illumination component and N observed images.

[0213] In some embodiments of this application, the image processing device can use the initial illumination component to guide Retinex decomposition to obtain the initial reflection component of the shooting scene. The image processing device can calculate the initial reflection component of the shooting scene according to formula (1) based on the initial illumination component of the shooting scene and N observed images:

[0214] (1)

[0215] Where λ represents the waveband. This is the observed image corresponding to band λ. This represents the initial reflection component corresponding to band λ. Let λ be the initial illumination component corresponding to band λ. Indicates the first The pixel position is a simplified representation of the pixel's two-dimensional coordinates (x, y). N is the number of bands, and λ is a positive integer.

[0216] Specifically, the image processing device can transform formula (1) into the logarithmic domain to obtain formula (2):

[0217] (2)

[0218] The logarithmic domain reflection component corresponding to band λ is calculated according to formula (2). Then to By performing an antilogarithmic transformation, the initial reflection components of the scene in band λ can be obtained.

[0219] For example, taking N=16 as an example, the initial illumination components of the shooting scene include 16 initial illumination components under 16 bands. The image processing device can calculate the 16 initial reflection components of the shooting scene under 16 bands according to the 16 initial illumination components and the above formula (2). Each band corresponds to one initial reflection component and one initial reflection component.

[0220] It can be understood that the initial reflection components of the shooting scene can be constructed into a three-dimensional array, and the shape of this three-dimensional array is... , Where N is the number of pixels in the multispectral sensor, and N is the number of bands.

[0221] In some embodiments of this application, in another approach, the image processing device can construct the initial illumination component of the shooting scene based on the average pixel values ​​of N observed images.

[0222] For example, assuming N observation images include observation images in 16 bands, the pixel values ​​of the 16 observation images can be added together to obtain a sum; then, based on the average of this sum, the initial illumination components of the scene in the 16 bands can be constructed. It can be seen that the illumination component values ​​of the initial illumination components in each of the 16 bands are the same.

[0223] Step 102B: The image processing device updates the initial illumination component based on N observed images, the aforementioned initial reflection component, and N illumination constraint information to obtain the second illumination component of the shooting scene.

[0224] Step 102C: The image processing device updates the initial reflection component based on N observed images and the second illumination component to obtain the second reflection component of the shooting scene.

[0225] Step 102D: The image processing device iteratively updates the illumination component and reflection component of the shooting scene to obtain the first illumination component of the shooting scene.

[0226] In some embodiments of this application, the image processing device can iteratively update the illumination component and reflection component of the shooting scene to obtain the first illumination component of the shooting scene.

[0227] In some embodiments of this application, the image processing device can obtain the first illumination component of the shooting scene by repeatedly alternating and iterating the illumination component and reflection component of the shooting scene until a preset condition is met.

[0228] The preset conditions may include at least one of the following:

[0229] Condition 1) The number of iterations reaches the preset number;

[0230] Condition 2) The change in the reflection component of the shooting scene is less than or equal to the first change threshold;

[0231] Condition 3) The change in the illumination component of the shooting scene is less than or equal to the second change threshold;

[0232] Condition 4) The value of the first function is less than or equal to the preset value. The first function is a binary function of the reflection component and the illumination component of the shooting scene; for example, the first illumination component is the illumination component that minimizes the value of the first function.

[0233] In some embodiments of this application, the image processing device can determine the illumination component update formula and the reflection component update formula according to the following formula (3), and then update the illumination component and reflection component of the shooting scene using the illumination component update formula and the reflection component update formula respectively:

[0234] (3)

[0235] In formula (3), This refers to the illumination component corresponding to band λ in the first illumination component. For the first function, Used in When the function value is at its minimum, the illumination component of the shooting scene is output, and the range of λ is [missing value]. N is an integer greater than 4.

[0236] For example, taking the lighting constraint information as including the first constraint information and the second constraint information, the above formula (3) can be expressed as the following formula (4):

[0237] (4)

[0238] In the above formula (4), This refers to the illumination component corresponding to band λ in the first illumination component. Let N be the first function, where N is the number of bands and is an integer greater than 4. The observed image is for band λ. and These represent the reflection and illumination components of the scene being photographed in the λ band, respectively. This is the second constraint information. This is the first constraint information. It is the square of the gradient or the square of the L2 norm. For the total variational regular operator, in order to use the Fast Fourier Transform, we can... It is approximately equal to the square of the illumination gradient, i.e., the L2 norm of the illumination gradient. Let λ be the illumination gradient of the wavelength band. For the prior gradient of illumination, For spatial gradient operators, and For different frequency bands, and The range of values ​​is α is the spatial smoothness weight of illumination, and β is the cross-band illumination correlation weight. Both α and β are greater than 0.

[0239] For example, the range of values ​​for α can be... .

[0240] For example, the range of values ​​for β can be... .

[0241] For example, α and β are 0.1 and 1, respectively.

[0242] In some embodiments of this application, The range of values ​​is .

[0243] In some embodiments of this application, the current illumination gradient under N-1 bands that are different from each of the N bands can be used as the prior illumination gradient, such as... .

[0244] For example, when updating the illumination components under band λ for the i-th time, N-1 illumination gradients can be calculated based on the N-1 illumination components of the shooting scene obtained in the (i-1)-th update, excluding the illumination component of band λ. These N-1 illumination gradients correspond one-to-one with the N illumination components, and are used as the prior illumination gradients of band λ for the i-th illumination update. For instance, the sum of the divergences of these N-1 illumination gradients can be used as the prior illumination gradient of band λ for the i-th illumination update, where i is a positive integer greater than 1, N is an integer greater than 4, and the value range of λ is... .

[0245] Alternatively, the current illumination gradient of a reference band can be used as the prior illumination gradient to require the updated illumination gradients of other bands to converge towards the illumination gradient of the reference band. The reference band can be any one of N bands, such as the infrared band, the green band, etc., and this application does not limit the scope of the embodiments.

[0246] Alternatively, the value of the above-mentioned prior illumination gradient can be set to 0, indicating that the illumination gradients of different wavelengths should be as similar as possible.

[0247] In some embodiments of this application, in actual implementation, the above α, β, The value of can be determined through cross-validation; see relevant techniques for details.

[0248] In some embodiments of this application, the image processing apparatus may determine the illumination component update formula and the reflection component update formula according to the above formula (4) through the following steps:

[0249] Step 21: The image processing device can convert the first function in formula (4) to the logarithmic domain to obtain the logarithmic domain formula (5) of the first function.

[0250] Specifically, assuming the logarithmic domain observation image under band λ is The logarithmic domain reflection component under band λ is The logarithmic domain illumination component in band λ is ,and Let be the square of the gradient of the illumination component under band λ, that is, the L2 norm of the illumination component under band λ, thus obtaining the following logarithmic domain formula (5):

[0251] (5)

[0252] In formula (5), For the first function, , and Let N be the logarithmic domain representation of the observed image for band λ, the logarithmic domain representation of the reflection component, and the logarithmic domain representation of the illumination component, respectively. N is the number of bands, and N is an integer greater than 4. This is the second constraint information. This is the first constraint information. Let λ be the illumination gradient of the wavelength band. For the prior gradient of illumination, The reflection gradient under band λ, The illumination gradient is at band λ. For spatial gradient operators, and For different frequency bands, and The range of values ​​is α is the spatial smoothness weight of illumination, and β is the cross-band illumination correlation weight. Both α and β are greater than 0.

[0253] It can be understood that Δ in formula (5) is the Laplace operator, which acts on the illumination component of each band when updating illumination, and on the reflection component of each band when updating reflectivity.

[0254] Step 22: The image processing device uses a fixed variable approach to alternately solve the above formula (5) to obtain the illumination component update formula and the reflection component update formula.

[0255] In some embodiments of this application, one variable in formula (5) is fixed, and the derivative with respect to another variable is taken, assuming the derivative is 0; then, the fast Fourier transform (FFT) algorithm is used to transform formula (5) to the frequency domain, and the system of linear equations with respect to the other variable is solved in the frequency domain to obtain the illumination component update formula and the reflection component update formula. Here, the one variable and the other variable are the illumination component and the reflection component of the shooting scene, respectively. The frequency domain can be understood as the logarithmic domain. Fixing one variable in the embodiments of this application means treating that variable as a constant. The system of linear equations includes linear equations with the illumination component as the variable and linear equations with the reflection component as the variable.

[0256] For example, the reflection component in formula (5) is fixed, that is, the reflection component in formula (5) As a constant, for each band λ, formula (5) is used to calculate... The variational derivative is then transformed into the frequency domain using FFT, resulting in the illumination component update formula as shown in formula (6):

[0257] (6)

[0258] In formula (6), Let be the logarithmic domain representation of the illumination component updated for band λ. This is the inverse Fourier transform. For Fourier transform, Let be the logarithmic domain representation of the observed image for band λ. Let λ be the logarithmic domain representation of the reflection component obtained from the most recent update at band λ, where α is the illumination spatial smoothness weight, β is the cross-band illumination correlation weight, α and β are both greater than 0, N is an integer greater than 4, and λ is a positive integer, with values ​​ranging from 0 to 1. , For spatial gradient operators, For the Laplace operator, For the Fourier transform of the Laplace operator, For the prior gradient of illumination, For divergence operators, For the prior gradient of illumination Find the divergence. The corresponding result is also a fixed numerical field.

[0259] In some embodiments of this application, in formula (6) , α, β, N, as well as These are known values, and together they constitute the lighting constraint information. For example, , α, β, N, as well as The values ​​are the same for all bands.

[0260] For example, the illumination component in formula (5) is fixed, that is, the illumination component in formula (5) is fixed. As a constant, formula (5) can be used to calculate the value of λ for each band. The variational derivative is then transformed into the frequency domain using FFT, resulting in the illumination component update formula shown in formula (7):

[0261] (7)

[0262] In formula (7), Let be the logarithmic domain representation of the updated illumination components at band λ. Let be the logarithmic domain representation of the observed image at band λ. Let be the logarithmic domain representation of the reflection component obtained from the most recent update under band λ. It can be seen that formula (7) follows the Retinex imaging model.

[0263] In some embodiments of this application, during the iterative updating of the illumination component and reflection component of the shooting scene, formulas (6) and (7) can be directly used to calculate the updated illumination component and reflection component of the shooting scene.

[0264] In some embodiments of this application, after updating the illumination component and reflection component of the shooting scene once, the updated illumination component and reflection component can be substituted into the above formula (4) or (5) to calculate a value of a first function. The value of the first function is used for convergence judgment and does not participate in the update of the reflection component and illumination component itself.

[0265] In some embodiments of this application, the image processing device may perform a convergence judgment after obtaining multiple values ​​of the first function to save power consumption.

[0266] The following example uses the above N observation images, including 16 observation images under 16 multi-band conditions, to illustrate steps 102B to 102D.

[0267] For example, the N observation images include 16 observation images in 16 multi-band formats, and these 16 observation images can form a dataset of size [size missing]. The three-dimensional array 1, with 16 observation images represented as follows: The initial illumination components of the shooting scene include 16 illumination components corresponding one-to-one with 16 wavelengths. These 16 initial illumination components are represented as follows: The initial reflection components of the shooting scene include 16 illumination components corresponding one-to-one with 16 wavelengths, denoted as follows: Therefore: based on three-dimensional array 1, and Using the formulas (6) and (7) above, the illumination and reflection components of the shooting scene are updated multiple times. Considering the spectral correlation between all bands and the constraint of smooth change in illumination space, the optimal unified illumination field of the shooting scene is calculated, namely the first illumination component. The first illumination component is a three-dimensional array, and the shape of the array of the first illumination component is as follows: , m and k represent the number of pixels in the multispectral sensor, where m and k are both positive integers.

[0268] Specifically, based on the observed image and reflection component of each band, as well as the illumination constraint information, the illumination component of each band can be updated using formula (6) to obtain 16 updated illumination components for 16 bands, expressed as follows: It is understandable that the second illumination component of the shooting scene includes .

[0269] Then, based on the observed images and updated illumination components for each band, the reflection components for each band can be updated using formula (7) to obtain the updated reflection components for the 16 bands, expressed as follows: It can be understood that the second reflection component of the shooting scene includes .

[0270] Taking the aforementioned preset condition 4) as an example, the image processing device can process 16 observed images. Second reflection component Second illumination component Substitute into the above formula (5) to calculate the first value of the first function.

[0271] Then, the image processing device can continue to substitute the reflection component corresponding to each observation image in the second reflection component into formula (6) to update the illumination component in each band for the second time, obtaining 16 updated illumination components in 16 bands, denoted as: Substituting the 16 illumination components and 16 observation images obtained from the most recent update into formula (7), we obtain the 16 updated reflection components for the 16 bands, which can be expressed as follows: Therefore, based on 16 observed images, as well as The second value of the first function is calculated.

[0272] This process continues until the Tth value of the first function is less than or equal to a preset value, and the illumination component of the shooting scene corresponding to the Tth value is determined as the first illumination component, where T is a positive integer.

[0273] Thus, since the initial illumination component can be updated based on N observed images, the aforementioned initial reflection component, and N illumination constraint information, and then the initial reflection component can be updated based on N observed images and the second illumination component, it can be ensured that the N illumination constraint information directly correlates and constrains the illumination components under different wavelengths, and can also indirectly constrain the reflection component of the shooting scene. Therefore, by alternately updating the illumination component and the reflection component, the illumination component and the reflection component of the shooting scene can gradually approach the real illumination field of the shooting scene, thereby improving the accuracy of the estimation of the illumination component of the shooting scene.

[0274] In some embodiments of this application, step 102C above can be implemented by step 102C1 below.

[0275] Step 102C1: The image processing device updates the initial reflection component based on N observed images, the second illumination component, and N reflection constraint information to obtain the second reflection component of the shooting scene.

[0276] Among them, reflection constraint information can be used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

[0277] In some embodiments of this application, N reflection constraint information corresponds one-to-one with N observation images, that is, each reflection constraint information corresponds to an observation image in one band.

[0278] In some embodiments of this application, the image processing device updates the initial reflection component based on N observed images, the second illumination component, and N reflection constraint information to obtain the second reflection component of the shooting scene. This may include: the image processing device updates the initial reflection component based on N observed images, the second illumination component, and N reflection constraint information using formula (8) to obtain the second reflection component of the shooting scene. Formula (8) is as follows:

[0279] (8)

[0280] In formula (8), Let be the logarithmic domain representation of the updated reflection component at band λ. This is the inverse Fourier transform. For Fourier transform, For the Laplace operator, For the Fourier transform of the Laplace operator, Let be the logarithmic domain representation of the observed image at band λ. Let γ be the logarithmic domain representation of the illumination component corresponding to band λ in the second illumination component, where γ is the reflection spatial smoothness weight, γ is greater than 0, and λ is a positive integer, with values ​​ranging from 0 to 1. N is an integer greater than 4.

[0281] In some embodiments of this application, γ in formula (8) These are known values, and together they constitute the reflection constraint information.

[0282] In some embodiments of this application, the reflection constraint information corresponding to different wavebands is the same.

[0283] In some embodiments of this application, the above formula (8) fixes the reflection component in formula (9), that is, the illumination component in formula (9). As a constant, and for each band λ, formula (9) is used to calculate λ. The variational derivative is then transformed into the frequency domain using FFT, resulting in the illumination component update formula as shown in formula (8), where formula (9) is as follows:

[0284] (9)

[0285] In formula (9), Let the logarithmic field expression of the first function be given. , and Here, are the logarithmic domain representations of the observed image for band λ, representing the reflection and illumination components, respectively. N is the number of bands, and N is an integer greater than 4. This is the second constraint information. This is the first constraint information. Let λ be the illumination gradient of the wavelength band. For the prior gradient of illumination, The reflection gradient under band λ, For spatial gradient operators, and For different frequency bands, and The range of values ​​is α is the spatial smoothness weight of illumination, β is the cross-band illumination correlation weight, and γ is the spatial smoothness weight of reflection. α, β, and γ are all greater than 0.

[0286] For example, the range of values ​​for γ can be: .

[0287] For example, α, β, and γ are 0.1, 0.2, and 0.2, respectively. .

[0288] The following example, with N=16, illustrates step 102C1 above.

[0289] For example, taking N=16 as an example, the 16 observed images are as follows: The second illumination component of the shooting scene includes Therefore, based on the observed images and reflection components of each band, as well as the illumination constraint information, the reflection components of each band can be updated using formula (6) to obtain 16 updated reflection components for 16 bands, expressed as follows: It can be understood that the second reflection component of the shooting scene includes .

[0290] It should be noted that since the illumination constraint information in formula (9) is the same as that in formula (5), the reflection component is fixed, and formula (9) is calculated for each band λ. The variational derivative is then transformed into the frequency domain using FFT, resulting in the illumination component update formula shown in formula (6).

[0291] Thus, since the initial reflection component can be updated based on N observed images, the second illumination component, and N reflection constraint information, the updated reflection component can be controlled to remain spatially smooth through the reflection constraint information. This ensures that the updated reflection component better matches the true reflected illumination of the physical illumination field of the shooting scene. When further updating the illumination component of the shooting scene based on the updated reflection component, the illumination component can be guided to move closer to the true illumination in the physical illumination field, thereby further improving the estimation accuracy of the illumination component.

[0292] In some embodiments of this application, the first function includes a data fidelity term and a lighting constraint term.

[0293] In some embodiments of this application, a data fidelity term is used to constrain the product of the illumination component and the reflection component of the shooting scene in each band to approximate the pixel value of the observed image in each band. This ensures that the illumination estimation process for the shooting scene does not deviate from the true physical imaging principle, thus avoiding distortion of the first illumination component.

[0294] In some embodiments of this application, the data fidelity item can be constructed based on the Retinex imaging model, which is constructed based on the Retinex model and the physical imaging model.

[0295] The Retinex model can be represented as: I represents the observed image, R represents the reflection component, and L represents the illumination component. The physical imaging model means that the response value of each band to the light field includes both the illumination component and the reflection component. Therefore, the Retinex imaging model can be expressed as the following formula (10):

[0296] (10)

[0297] Where λ represents the band, which can be a band number. This represents the observed image under band λ. This represents the reflection component at band λ. Let λ represent the illumination component at wavelength λ, and x represent the pixel position, which is the pixel's two-dimensional coordinates. A simplified expression.

[0298] In some embodiments of this application, formula (10) can be simplified as: ,in, , and These represent the observed image, reflection component, and illumination component at band λ, respectively.

[0299] In some embodiments of this application, referring to formula (4), the data guarantee term can be expressed as: Where λ is the waveband. , and Let N represent the observed image, reflection component, and illumination component at band λ, respectively. N is the number of bands, an integer greater than or equal to 4, and λ is a positive integer with a range of values. .

[0300] In some embodiments of this application, the above-mentioned lighting constraint terms are used to indicate the above-mentioned N lighting constraint information.

[0301] For example, taking the illumination constraint information as including the first constraint information and the second constraint information mentioned above, the illumination constraint terms may include: cross-band consistency constraint terms and illumination spatial smoothness constraint terms.

[0302] As shown in formula (4), the cross-band consistency constraint term can be expressed as the third term in formula (4), that is... The illumination spatial smoothing constraint term can be expressed as the second term in formula (4), i.e. .

[0303] Where N is the number of bands, and N is an integer greater than 4. The observed image is for band λ. and These represent the reflection and illumination components of the scene being photographed in the λ band, respectively. This is the second constraint information. This is the first constraint information. It is the square of the gradient or the square of the L2 norm. For the total variational regular operator, in order to use the Fast Fourier Transform, we can... It is approximately equal to the square of the illumination gradient, i.e., the L2 norm of the illumination gradient. Let λ be the illumination gradient of the wavelength band. For the prior gradient of illumination, For spatial gradient operators, and For different frequency bands, and The range of values ​​is α is the spatial smoothness weight of illumination, and β is the cross-band illumination correlation weight. Both α and β are greater than 0.

[0304] It is understandable that the above lighting constraints directly affect the lighting components of the shooting scene, but indirectly affect the update quality of the reflection components through the coupling relationship between the lighting components and the reflection components defined by the data fidelity terms.

[0305] The following example, using illumination constraints including cross-band consistency constraints and illumination spatial smoothness constraints, briefly illustrates how the coupling relationship defined by data guarantee terms works.

[0306] During the iterative solution of formula (4) or (9) in multiple bands, the illumination component and reflection component of the shooting scene are not independent of each other, but are tightly coupled through the data fidelity term.

[0307] Specifically, when updating the illumination components, the illumination constraint term penalizes unevenness and inconsistencies between bands, while the data fidelity term forces the product of the reflection component and the illumination component in each band to approximate the corresponding observed image. Therefore:

[0308] When lighting components are constrained by adding lighting constraints to make them "better," such as smoother or more consistent across bands:

[0309] 1. For the illumination spatial smoothness constraint, if the illumination components in a certain wavelength band... The lighting is constrained too much to the point that it loses the subtle variations that real lighting should have. Therefore, in order to satisfy... The reflection component in this band The changes that originally belonged to the illumination component are mixed in, thus affecting the reflection component. This can lead to pseudo-textures or noise. Conversely, if the lighting space smoothness constraint affects the lighting components... The constraints are just right, limiting it to a "reasonable" range of illumination patterns, such as low frequency and inter-band alignment. Then, in order to fit a more accurate image to the observed image... Fitted observation image Reflection component They will be "forced" to express the high-frequency details and correct colors that truly belong to the reflective properties of objects.

[0310] 2. The cross-band consistency constraint has a deeper effect: It can force edge alignment of illumination components across different bands. If the illumination component in a certain band is mis-estimated at the object's edge, the cross-band consistency constraint will use correct edge information from other bands to "pull" it back to the correct position, as mentioned above regarding illumination priors. This corrects the edges of the illumination components in that band.

[0311] After edge correction of the illumination component, the updated reflection component of the band can be calculated using the above formula (7) or (8) based on the updated illumination component of the band. It can be understood that the updated reflection component will have reduced halo or color distortion at the edge compared to the initial reflection component. Therefore, the cross-band consistency constraint term indirectly achieves cross-band correction of the reflection component by directly constraining the illumination components under different bands to maintain correlation.

[0312] In some embodiments of this application, the first function described above may include a data fidelity term, a lighting constraint term, and a reflection constraint term. For a description of the data fidelity term and the lighting constraint term, please refer to the relevant descriptions of the data fidelity term and the lighting constraint term in the above embodiments.

[0313] In some embodiments of this application, the reflection constraint term is used to indicate the above N reflection constraint information.

[0314] It is understandable that the aforementioned reflection constraint term directly affects the reflection component of the shooting scene, but indirectly affects the update quality of the illumination component through the coupling relationship between the illumination component and the reflection component defined by the data fidelity term.

[0315] The following is a brief explanation of how the coupling relationship defined by the data guarantee term works when considering reflection constraints.

[0316] In the process of multi-band joint iterative solution of formula (9), the illumination component and reflection component of the shooting scene are not independent of each other, but are tightly coupled through the data fidelity term.

[0317] Specifically, when updating the reflection components, the reflection constraint term penalizes spatial non-smoothness of the reflection components, while the data fidelity term forces the product of the reflection component and the illumination component in each band to approximate the corresponding observed image. Therefore:

[0318] When reflection components are constrained by adding reflection constraint terms to make them "better," such as smoother and more consistent across bands:

[0319] If the reflection component in a certain band If the reflection constraint makes the surface too smooth, resulting in pseudo-textures or noise, then: in order to satisfy... In this wavelength range, the illumination component loses the subtle variations that true illumination should have. Conversely, if the reflection constraint term affects the reflection component... The spatial smoothing constraints are just right, such as the reflection component expressing the high-frequency details and correct colors that truly belong to the object's reflection properties. Therefore: to fit a more accurate image to the observed image... Fitted observation image Light component They will be "forced" to express the subtle changes that real lighting should have, such as low-frequency information.

[0320] It should be noted that the above-mentioned illumination constraint terms directly impose constraints on the illumination components of the shooting scene, and the reflection constraint terms directly impose constraints on the reflection components of the shooting scene; and both the illumination constraint terms and the reflection constraint terms can optimize and constrain the "responsibility division" between the illumination components and the reflection components of the shooting scene through the "bridge" of the data fidelity term, thereby ultimately affecting and improving the estimation quality of the illumination components of the shooting scene.

[0321] It is understandable that since formula (4) includes data fidelity terms, illumination smoothing constraints and cross-band consistency constraints, it can be ensured that the first illumination component estimated by formula (4) is the optimal illumination component that conforms to the actual observed image and satisfies the spatial smoothing constraints and cross-band consistency physical laws. Thus, it can be ensured that the true reflection component of the shooting scene can be accurately decomposed from N observed images based on the first illumination component.

[0322] In some embodiments of this application, the image processing device can estimate the first illumination component of the shooting scene based on N observed images through a multi-band joint optimization model.

[0323] The aforementioned multi-band joint optimization model can be an Alternating Direction Method of Multipliers (ADMM) model, a Half-Quadratic Splitting (HQS) model, or a lightweight U-shaped network model, referred to as the UNet model. Among them, the ADMM model and the HQS model can set the above formula (3).

[0324] In some embodiments of this application, a lightweight UNet can be pre-trained to learn from N observed images of the shooting scene, solve the above formula (3), and output the first illumination component of the shooting scene.

[0325] For example, a large number of N observation images from different shooting scenarios can be used as training data. The real illumination components and real reflection components of these observation images can be used as labels to train a UNet model until the model parameters of the UNet model converge, so as to obtain a multi-band joint optimization model.

[0326] Step 103: The image processing device determines the first reflection component of the above-mentioned shooting scene based on the first illumination component and N observation images.

[0327] In some embodiments of this application, the first reflection component is a set of N optimal reflection components of the shooting scene in N bands.

[0328] In some embodiments of this application, the image processing device fuses N observed images with Retinex deep fusion. Specifically, within the framework of Retinex theory, the N observed images are used as prior information or constraints of the whole and directly participate in the estimation process of illumination components, thereby fundamentally improving the accuracy of illumination estimation and image enhancement performance.

[0329] In some embodiments of this application, after estimating the first illumination component, the image processing device can reconstruct the spectral reflectance characteristics of the object based on N observed images and the first illumination component, using formula (10) to obtain the first reflection component.

[0330] It should be noted that, since the first reflection component of the shooting scene can be decomposed from N observation images based on the first illumination component using the Retinex imaging model, the reflection component decomposed from N observation images is closer to the real texture and details of the object in the physical illumination field of the shooting scene than the reflection component obtained in the last update. It can also eliminate the noise introduced by iteratively updating the reflection component, thus allowing for a more accurate determination of the reflection component of the shooting scene.

[0331] In some embodiments of this application, the image processing device can determine the first reflection component of the shooting scene based on the first illumination component and N observed images, using the aforementioned multi-band joint optimization model.

[0332] It can be understood that the first reflection component is a three-dimensional array, and the shape of this three-dimensional array is... ,in, Where is the size of the multispectral sensor, N is the number of bands, N is an integer greater than 4, and m and k are both positive integers.

[0333] In some embodiments of this application, step 103 may include steps 103A and 103B as described below.

[0334] Step 103A: For each of the N bands, the image processing device calculates the reflection component of the scene in the i-th band based on the observed image corresponding to the i-th band and the illumination component in the first illumination component corresponding to the i-th band.

[0335] Step 103B: The image processing device determines the reflection components of the scene in the above N bands as the first reflection component.

[0336] Where the range of values ​​for i is... , where i is a positive integer.

[0337] For example, a 16-channel multispectral sensor, and the size of the multispectral sensor is... For example, the image processing device can use the first illumination component to guide Retinex decomposition, combine the traditional Retinex model with the physical imaging model, and establish an imaging model as shown in formula (11):

[0338] (11)

[0339] Where λ represents the waveband, The observed image is for band λ. Let be the reflection component of band λ. For light component, Indicates the first The pixel position is a simplified representation of the pixel's two-dimensional coordinates (x, y). In specific calculations, [the following is a more detailed explanation:] and Transforming to the logarithmic domain, we obtain formula (12):

[0340] (12)

[0341] Therefore, based on the observed image and illumination component corresponding to each band, the corresponding value for each band can be calculated using formula (12). Then, the obtained By performing an antilogarithmic transform, the reflection components for each band can be obtained. In this way, the reflection components of the scene in 16 bands can be obtained, and these 16 reflection components can form a three-dimensional array with the following shape: ,in, Let m be the size of the multispectral sensor, and k be positive integers.

[0342] It is understandable that after obtaining the first illumination component, the first illumination component of the shooting scene is recalculated using the first illumination component and N observation images according to the Retinex decomposition method. This can eliminate the small errors that may accumulate during the iteration process, making the calculated first reflection component more consistent with the real reflection component of the shooting scene.

[0343] Thus, since the first reflection component of the shooting scene can be determined by calculating and combining it band by band based on the first illumination component and N observation images, the spectral dimension processing is also maintained in the reflectivity reconstruction stage. This allows the complete spectral reflectivity information in the N observation images to be retained, rather than just the RGB values, so that the first reflectivity can be closer to the real reflectivity characteristics of the object in the shooting scene.

[0344] Step 104: The image processing device generates a first image based on the first reflection component.

[0345] In some embodiments of this application, the image processing apparatus can convert the first reflection component to a first color space to obtain a first image.

[0346] In some embodiments of this application, the first color space includes, but is not limited to, any one of the following:

[0347] The sRGB color space is the most common color space.

[0348] Adobe RGB color space: It can provide a wider color gamut than sRGB and is often used in professional photography and printing;

[0349] DCI-P3 color space: widely used in digital cinema and high-end consumer electronics displays;

[0350] Rec. 2020 Color Space: Wide color gamut standard for Ultra HD TVs.

[0351] In some embodiments of this application, the image processing apparatus can multiply the first reflection component with a preset transformation matrix to obtain a second image, and then convert the second image to a first color space using the first reflection component. Gamma correction is then performed on the second image to obtain the first image.

[0352] The aforementioned transformation matrix is ​​determined based on the standard of the first color space and the spectral response characteristics of the RGB sensor.

[0353] It can be understood that the spectral response characteristics of an RGB sensor can be represented by the spectral sensitivity function of the RGB sensor, and the spectral response characteristics of an RGB sensor can also be referred to as the three-channel response information of RGB.

[0354] In some embodiments of this application, when the image processing device acquires the aforementioned N observation images via a multispectral sensor, it can also acquire images via an RGB sensor to obtain an RGB image of the shooting scene. From this RGB image, the three-channel response information of the RGB sensor under the same shooting scene can be obtained, i.e., the original image data acquired by the RGB sensor. Therefore, a transformation matrix can be determined based on the three-channel response information of the RGB sensor and the first color standard.

[0355] In some embodiments of this application, the above transformation matrix is The transformation matrix is ​​given by N, where N is the number of color channels in the multispectral sensor, i.e., the number of bands mentioned above. The image processing device can convert images into shapes of... The first reflection component and The transformation matrix of dimension is multiplied to obtain the second image.

[0356] In the image processing method provided in this application embodiment, N observation images of a shooting scene in N bands can be acquired, where N is an integer greater than 4; one band corresponds to one observation image; based on the N observation images and N illumination constraint information, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands; based on the first illumination component and the N observation images, the first reflection component of the shooting scene is determined; based on the first reflection component, a first image is generated. Through this scheme, since N illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands when estimating the first illumination component, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located, that is, accurate estimation of the illumination component can be achieved. Therefore, problems such as halo, artifacts, and color distortion caused by inaccurate illumination estimation can be avoided at the source, thus improving the image quality of the generated first image.

[0357] In some embodiments of this application, before step 104 above, the image processing method provided in the embodiments of this application may further include the following step 105.

[0358] Step 105: The image processing device performs a first processing on the first reflection component.

[0359] The first processing described above may include at least one of the following: denoising processing based on low-rank spectral priors, detail enhancement processing based on spatially guided filtering, and interspectral interpolation processing.

[0360] In some embodiments of this application, denoising processing based on a low-rank spectral prior is performed on the first reflection component to suppress high-frequency, sparse noise in the first reflection component. It is understood that the types of matter in a natural scene are usually limited. Therefore, the reflectance matrix, composed of spectral reflectance vectors from a large number of pixels, such as the aforementioned first reflectance, should have low-rank characteristics. That is, the main information of a substance or object, such as texture and color, can be linearly represented by several basis vectors representing the main material of the object. Noise, anomalous pixels, and local artifacts, on the other hand, are sparse components that deviate from this low-rank structure. Therefore, by performing denoising processing based on a low-rank spectral prior on the first reflection component, high-dimensional, sparse noise in the first reflection component, such as random noise from a multispectral sensor, can be separated and suppressed from the real signal, which is mainly composed of a few basic spectra, thereby reducing or removing noise in the first reflection component.

[0361] The following section explains the denoising process based on low-rank spectral priors.

[0362] In some embodiments of this application, the denoising process based on spectral low-rank priors can also be referred to as spectral low-rank prior processing.

[0363] The image processing device can first stack the first reflection components column-wise to convert them into a two-dimensional matrix. For example, with N=16, the image processing device takes the reflectance values ​​of the 16 bands of each pixel as a column vector, and then arranges the column vectors of all pixels side by side to obtain a two-dimensional matrix for subsequent calculations.

[0364] It is understandable that, assuming the number of material types in the scene is much smaller than the number of bands, that is, the number of material types of objects in the shooting scene is much smaller than the number of bands N, then the signal and noise in the first reflection component converted into a two-dimensional matrix can be separated by the Singular Value Decomposition (SVD) method or the Robust Principal Component Analysis (RPCA) method.

[0365] For example, taking the RPCA method for separating signal and noise as an example, the image processing apparatus performing a first processing on the first reflection component may include the following steps:

[0366] Step 31: Stacking reflectivity matrices , shape The first reflection components R are stacked column-wise to reconstruct the first reflection components R into a single element of size R. A two-dimensional matrix X, where X can be expressed as the following formula (13):

[0367] (13)

[0368] In this matrix, each row of the two-dimensional matrix X represents the total reflectance value for a single band, such as each row including... Each column of the two-dimensional matrix X represents the reflectance value of a pixel across N spectral bands; that is, each column is an N-dimensional reflectance vector of a pixel. R represents the shape of the matrix. The first reflection component, where N is the band number. Let m and k be the number of pixels, where m and k are both positive integers, and N is an integer greater than 4.

[0369] Step 32: Perform RPCA decomposition on matrix X:

[0370] (14)

[0371] Here, Z and S are both matrices, and Z is a low-rank matrix. This matrix Z is the low-rank part of matrix X, corresponding to the clean intrinsic signal composed of a few main material spectra; matrix S is the sparse part of matrix X, corresponding to noise, anomalous pixels and artifacts; η is a trade-off parameter that controls the denoising intensity. As a low-rank nuclear norm, it can minimize the sum of all singular values ​​and is used to force the rank of matrix Z to be as low as possible, which corresponds to the physical fact that the scene is composed of a finite number of linear combinations of "basic spectra". To minimize the sparsity L1 norm, it is used to force noise to be sparse, such that most positions in matrix S are zero. This is suitable for handling noise, bad pixels, or local artifacts.

[0372] It can be understood that performing RPCA decomposition on matrix X can also be called performing robust low-rank decomposition on matrix X.

[0373] Step 33: Extract the denoised reflectance spectrum: Reconstruct the low-rank matrix Z back to its original position. The shape is obtained by performing global spectral denoising on the first reflection component.

[0374] For example, taking the SVD method to separate the signal and noise in the first reflection component as an example, the image processing device may perform the first processing on the first reflection component by the following steps:

[0375] Step 41: By stacking reflectivity matrices , shape The first reflection components R are stacked column-wise to reconstruct the first reflection components R into a single element of size R. The two-dimensional matrix X, where X can be expressed as formula (13).

[0376] Step 42: Perform SVD decomposition on the two-dimensional matrix X, truncate and retain the first G singular values. Typically, the range of G is [value missing]. G is a positive integer.

[0377] Step 43: Reconstruct the retained singular values ​​to obtain the shape as reconstructed. The shape is obtained by performing global spectral denoising on the first reflection component.

[0378] It is understandable that the first reflection component after global spectral denoising is approximated with the low-rank component in matrix X, preserving the main correlation signals between bands and suppressing high-rank noise, such as sensor inherent noise.

[0379] In some embodiments of this application, in the other approach described above, the image processing device can perform detail enhancement processing based on spatially guided filtering on the first reflection component to suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the true texture and edge details of the object, avoiding the image from becoming too smooth due to denoising, thereby ensuring that the final enhancement result has better visual clarity and spatial detail representation, and improving high-frequency and sparse noise in the first reflection component.

[0380] In some embodiments of this application, the detail enhancement processing based on spatially guided filtering can utilize spatial information with a high signal-to-noise ratio as a guide to smooth and enhance the reflection components of all bands while preserving their edges.

[0381] For example, high signal-to-noise ratio spatial information can be the edge information of an object.

[0382] For example, edge details extracted from the near-infrared band.

[0383] For example, edge details extracted from a visible light band that has already undergone preliminary denoising.

[0384] The following is an illustrative example of detail enhancement processing based on spatial guided filtering.

[0385] In some embodiments of this application, the detail enhancement processing based on spatially guided filtering typically uses the edge intensity map of the near-infrared band with a high signal-to-noise ratio as the guiding map to perform guided filtering on the reflection components of each band, in order to further suppress residual noise and artifacts in the reflection components. Specifically, the method may include the following steps:

[0386] Step 51, Guided Map Selection: Select the near-infrared band reflection component from the first reflection component. and will Gradient magnitude map O= As a guide diagram.

[0387] Step 52, Band-by-band guided filtering: For each band λ, using the gradient magnitude map G as the guide map, perform classic guided filtering on the reflection component of band λ. After performing guided filtering on the reflection components of all bands, the optimized first reflection component is obtained.

[0388] In some embodiments of this application, the image processing apparatus may first perform denoising processing based on a low-rank spectral prior on the first reflection component, and then perform detail enhancement processing based on spatial guided filtering on the denoised first reflection component. In this way, denoising processing based on a low-rank spectral prior can globally eliminate cross-band random noise and outliers, resulting in a denoised but potentially spatially smooth first reflection component; then, spatial guided filtering, guided by high signal-to-noise ratio bands, restores and enhances the local spatial details of the denoised first reflection component. It is understood that the image processing apparatus can consider the dual constraints of the spatial and spectral domains, performing joint guided filtering on the first reflection component to remove potential noise in the first reflection component without changing the array shape and size of the first reflection component, and fine-tuning the reflectivity value in the first reflection component to ensure that the final enhanced image retains both spatial details and spectral characteristics.

[0389] Of course, in actual implementation, denoising based on low-rank spectral prior can be performed on the first reflection component to obtain a denoised first reflection component; and detail enhancement based on spatial guided filtering can be performed on the first reflection component to obtain a detail-enhanced first reflection component. Then, the denoised first reflection component and the detail-enhanced first reflection component can be fused to obtain the optimized first reflection component.

[0390] The following explains the inter-spectral interpolation process.

[0391] In some embodiments of this application, by performing spectral interpolation on the first reflection component, the discrete N reflection components in the first reflection component can be converted into a continuous reflectance curve within a first band range, thereby obtaining the continuous spectral reflectance of each pixel location within the first band range. The first band range includes the continuous bands of the aforementioned N bands. Of course, the first reflection component after spectral interpolation can also be discontinuous in the spectral dimension, such as performing spectral interpolation only on the reflection components of a few main bands.

[0392] Thus, on the one hand, denoising based on low-rank spectral priors can remove high-frequency, sparse noise from the first reflection component while preserving the object's true, low-frequency reflection signal, thereby improving the reliability of the first reflection component. On the other hand, detail enhancement based on spatially guided filtering can suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the object's true texture and edge details, preventing the image from becoming overly smooth due to denoising, thus ensuring that the final enhancement result has better visual clarity and spatial detail representation. Furthermore, spectral interpolation of the first reflection component can reconstruct a more continuous spectral curve from discretely sampled reflectance, making subsequent color space conversion calculations more accurate, helping to reduce color calculation errors caused by insufficient spectral sampling, and improving the color fidelity of the enhanced image. Therefore, by performing targeted first processing on the first reflection component, noise can be suppressed in the spectral dimension to improve data quality, or details can be enhanced in the spatial dimension to improve visual clarity. This results in an enhanced image that is not only more realistic in color, but also cleaner and sharper visually, thus ensuring that the image quality of the enhanced image is further optimized.

[0393] The following describes the flow of the image processing method of this application embodiment, using narrowband multispectral sensors and FP cavity multispectral sensors as examples respectively.

[0394] Example 1, such as Figure 5 As shown, the multispectral sensor is a narrowband multispectral sensor. The image processing method provided in this application embodiment may include the following steps:

[0395] Step 501: The image processing device acquires N mosaic images of the scene in N bands using a multispectral sensor with a narrowband filter array, where N is an integer greater than 4.

[0396] Step 502: The image processing device acquires an RGB image of the above-mentioned shooting scene through an RGB sensor.

[0397] Step 503: The image processing device performs de-mosaic processing on the N mosaic images based on the RGB image to obtain N observation images.

[0398] Step 504: The image processing device determines the initial illumination component and initial reflection component of the shooting scene based on N observed images.

[0399] Step 505: The image processing device updates the initial illumination component based on N observed images, the aforementioned initial reflection component, and N illumination constraint information to obtain the second illumination component of the shooting scene.

[0400] Step 506: The image processing device updates the initial reflection component based on N observed images and the second illumination component to obtain the second reflection component of the shooting scene.

[0401] Step 507: The image processing device iteratively updates the illumination component and reflection component of the shooting scene to obtain the first illumination component of the shooting scene.

[0402] For a detailed description of the iterative update process, please refer to the relevant descriptions in the above embodiments. To avoid repetition, they will not be repeated here.

[0403] Step 508: The image processing device performs a first processing on the first reflection component.

[0404] Step 509: The image processing device generates a first image based on the first reflection component after the first processing.

[0405] Thus, firstly, since the first illumination component of the shooting scene can be estimated from N observation images and N illumination constraint information, the accuracy of illumination estimation can be improved: the traditional Retinex method estimates illumination based on local filtering of a single RGB image, which is prone to halo effects at shadow or highlight edges due to assumption failure. In contrast, the high-dimensional spectral data obtained by the multispectral sensor in this application can more directly invert the spectral power distribution of scene illumination. Through joint optimization and constraints across multiple bands, the brightness variations caused by light source characteristics and the texture details of the object itself can be effectively distinguished, thereby achieving more accurate illumination estimation with better edge preservation from the source and fundamentally suppressing the halo effect.

[0406] Secondly, it ensures the color fidelity and consistency of the first image: accurate illumination estimation is a prerequisite for correct color correction. Based on accurately estimated illumination components, the intrinsic reflectance, i.e., intrinsic color, of an object can be more thoroughly separated from the observed image. This effectively solves the color cast problem caused by overall grayscale or color channel imbalance due to illumination estimation errors in the traditional Retinex algorithm. Simultaneously, rich spectral information enables the image processing device to distinguish metamerism, meaning that materials with different spectral compositions appear to have the same color to the human eye, thus maintaining the perceptual consistency of object color under different lighting conditions.

[0407] Furthermore, it possesses adaptability to complex scenes: N observed images provide a physical dimension beyond the RGB three-channel configuration. This enables the image processing device to analyze and distinguish different light sources in the shooting scene, such as the spectral characteristics of sunlight, sodium lamps, and LED lights. Therefore, even in scenes with mixed "interference colors" from multiple light sources, the stability of the illumination estimation can be maintained. Thus, the image processing method provided in this example can maintain reliable enhancement effects in application scenarios with complex and variable indoor and outdoor lighting, or where special artificial light sources exist.

[0408] Example 2, taking an FP cavity type multispectral sensor as an example, such as Figure 6 As shown, the image processing method provided in this application embodiment may include the following steps:

[0409] Step 601: The image processing device acquires N observation images of the shooting scene under N multi-wavelengths through the FP cavity multispectral sensor.

[0410] Step 602: The image processing device determines the initial illumination component and initial reflection component of the shooting scene based on N observed images.

[0411] Step 603: The image processing device updates the initial illumination component based on N observed images, the aforementioned initial reflection component, and N illumination constraint information to obtain the second illumination component of the shooting scene.

[0412] Step 604: The image processing device updates the initial reflection component based on N observed images and the second illumination component to obtain the second reflection component of the shooting scene.

[0413] Step 605: The image processing device iteratively updates the illumination component and reflection component of the shooting scene to obtain the first illumination component of the shooting scene.

[0414] For a detailed description of the iterative update process, please refer to the relevant descriptions in the above embodiments. To avoid repetition, they will not be repeated here.

[0415] Step 606: The image processing device generates a first image based on the first reflection component after the first processing.

[0416] In some embodiments of this application, the image sensor can also acquire an RGB image of the shooting scene via an RGB sensor. This RGB image is used to determine the three-channel response information of the RGB sensor under the shooting scene, i.e., the raw image data acquired by the three channels of the RGB sensor, in order to determine the transformation matrix for color space conversion of the first reflection component. For example, during the acquisition of the above N observation images, an RGB image of the shooting scene can be acquired via an RGB image.

[0417] Thus, in Example 2, by acquiring N observation images of the scene using an FP cavity multispectral sensor, the de-mosaic step can be reduced. This not only achieves all the effects of Example 1, but also saves processing resources and processing time, and reduces power consumption.

[0418] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0419] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0420] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 7 As shown, the image processing apparatus 700 includes:

[0421] The acquisition module 701 is used to acquire N observation images of the shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image.

[0422] Processing module 702 is used for:

[0423] Based on the N observed images and N illumination constraint information acquired by the acquisition module, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands.

[0424] Based on the first illumination component and the N observed images, the first reflection component of the shooting scene is determined;

[0425] A first image is generated based on the first reflection component.

[0426] Thus, since N illumination constraint information is used when estimating the first illumination component of the shooting scene to constrain the correlation between the illumination distribution of the shooting scene in different wavelengths, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located. That is, the illumination component can be accurately estimated. Therefore, the problems of halo, artifact and color distortion caused by inaccurate illumination estimation can be avoided from the source, thereby improving the image quality of the generated first image.

[0427] In some possible implementations, the aforementioned illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between illumination distributions of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of illumination distributions of the shooting scene in one wavelength band.

[0428] Thus, since each observed image corresponds to a first constraint information for ensuring consistency of illumination components across bands and a second constraint information for ensuring spatial smoothness of illumination components, on the one hand, the edges of illumination components in different bands of the shooting scene can be made to be consistent; on the other hand, the second constraint information can prevent the illumination components of the shooting scene from jumping in each band, thus ensuring that the illumination components in each band remain spatially smooth. This ensures that the estimated first illumination component is consistent with the illumination distribution in the real illumination field where the shooting scene is located.

[0429] In some possible implementations, the above processing module is specifically used for:

[0430] Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined;

[0431] Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene;

[0432] Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene;

[0433] The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

[0434] Thus, since the initial illumination component can be updated based on N observed images, the aforementioned initial reflection component, and N illumination constraint information, and then the initial reflection component can be updated based on N observed images and the second illumination component, it can be ensured that the N illumination constraint information directly correlates and constrains the illumination components under different wavelengths, and can also indirectly constrain the reflection component of the shooting scene. Therefore, by alternately updating the illumination component and the reflection component, the illumination component and the reflection component of the shooting scene can gradually approach the real illumination field of the shooting scene, thereby improving the accuracy of the estimation of the illumination component of the shooting scene.

[0435] In some possible implementations, the above processing module is specifically used to update the initial reflection component based on the N observed images, the second illumination component, and the N reflection constraint information to obtain the second reflection component of the shooting scene;

[0436] The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

[0437] Thus, since the initial reflection component can be updated based on N observed images, the second illumination component, and N reflection constraint information, the updated reflection component can be controlled to remain spatially smooth through the reflection constraint information. This ensures that the updated reflection component better matches the true reflected illumination of the physical illumination field of the shooting scene. When further updating the illumination component of the shooting scene based on the updated reflection component, the illumination component can be guided to move closer to the true illumination in the physical illumination field, thereby further improving the estimation accuracy of the illumination component.

[0438] In some possible implementations, the aforementioned acquisition module is specifically used for:

[0439] The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band.

[0440] The RGB image of the scene being photographed is acquired using an RGB sensor.

[0441] The processing module is further configured to perform de-mosaic processing on the N mosaic images acquired by the acquisition module based on the RGB image acquired by the acquisition module, to obtain N observation images.

[0442] Thus, since RGB images can be used to guide the de-mosaicing of N mosaic images acquired by a multispectral sensor, that is, the high spatial resolution information of RGB images can be used to restore the missing texture details of the mosaic images, the N observation images obtained after de-mosaicing can provide richer spatial details for subsequent illumination component estimation, thereby ensuring the accuracy of illumination estimation.

[0443] In some possible implementations, the above processing module is specifically used for:

[0444] Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images;

[0445] The N upsampled images are registered with the RGB image to obtain N registered images;

[0446] Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image;

[0447] The guide image corresponding to each registration image is a channel image of the RGB image.

[0448] Thus, since RGB images can be used to sequentially upsample, register, and guide filtering N mosaic images, the high-frequency details of the RGB images can be injected into the N mosaic images under the N bands. This can significantly improve the spatial resolution of the N observation images obtained after de-mosaicing, thereby ensuring that the N observation images provide richer and finer edges of illumination changes for illumination estimation, and thus improving the accuracy of illumination estimation.

[0449] In some possible implementations, the above-mentioned processing module is further configured to perform a first processing on the first reflection component before generating the first image based on the first reflection component;

[0450] The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

[0451] Thus, on the one hand, denoising based on low-rank spectral priors can remove high-frequency, sparse noise from the first reflection component while preserving the object's true, low-frequency reflection signal, thereby improving the reliability of the first reflection component. On the other hand, detail enhancement based on spatially guided filtering can suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the object's true texture and edge details, preventing the image from becoming overly smooth due to denoising, thus ensuring that the final enhancement result has better visual clarity and spatial detail representation. Furthermore, spectral interpolation of the first reflection component can reconstruct a more continuous spectral curve from discretely sampled reflectance, making subsequent color space conversion calculations more accurate, helping to reduce color calculation errors caused by insufficient spectral sampling, and improving the color fidelity of the enhanced image. Therefore, by performing targeted first processing on the first reflection component, noise can be suppressed in the spectral dimension to improve data quality, or details can be enhanced in the spatial dimension to improve visual clarity. This results in an enhanced image that is not only more realistic in color, but also cleaner and sharper visually, thus ensuring that the image quality of the enhanced image is further optimized.

[0452] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0453] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0454] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 6 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0455] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0456] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0457] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0458] The electronic device 1500 includes, but is not limited to, the following components: radio frequency unit 1501, network module 1502, audio output unit 1503, input unit 1504, sensor 1505, display unit 1506, user input unit 1507, interface unit 1508, memory 1509, processor 1510, and multispectral sensor.

[0459] Those skilled in the art will understand that the electronic device 1500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0460] The processor 1510 is used to acquire N observation images of the shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image.

[0461] Processor 1510, used for:

[0462] Based on the N observed images and N illumination constraint information acquired by the processor 1510, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands.

[0463] Based on the first illumination component and the N observed images, the first reflection component of the shooting scene is determined;

[0464] A first image is generated based on the first reflection component.

[0465] Thus, since N illumination constraint information is used when estimating the first illumination component of the shooting scene to constrain the correlation between the illumination distribution of the shooting scene in different wavelengths, it can be ensured that the first illumination component conforms to the illumination distribution in the real illumination field where the shooting scene is located. That is, the illumination component can be accurately estimated. Therefore, the problems of halo, artifact and color distortion caused by inaccurate illumination estimation can be avoided from the source, thereby improving the image quality of the generated first image.

[0466] In some possible implementations, the aforementioned illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between illumination distributions of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of illumination distributions of the shooting scene in one wavelength band.

[0467] Thus, since each observed image corresponds to a first constraint information for ensuring consistency of illumination components across bands and a second constraint information for ensuring spatial smoothness of illumination components, on the one hand, the edges of illumination components in different bands of the shooting scene can be made to be consistent; on the other hand, the second constraint information can prevent the illumination components of the shooting scene from jumping in each band, thus ensuring that the illumination components in each band remain spatially smooth. This ensures that the estimated first illumination component is consistent with the illumination distribution in the real illumination field where the shooting scene is located.

[0468] In some possible implementations, the aforementioned processor 1510 is specifically used for:

[0469] Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined;

[0470] Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene;

[0471] Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene;

[0472] The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

[0473] Thus, since the initial illumination component can be updated based on N observed images, the aforementioned initial reflection component, and N illumination constraint information, and then the initial reflection component can be updated based on N observed images and the second illumination component, it can be ensured that the N illumination constraint information directly correlates and constrains the illumination components under different wavelengths, and can also indirectly constrain the reflection component of the shooting scene. Therefore, by alternately updating the illumination component and the reflection component, the illumination component and the reflection component of the shooting scene can gradually approach the real illumination field of the shooting scene, thereby improving the accuracy of the estimation of the illumination component of the shooting scene.

[0474] In some possible implementations, the processor 1510 is specifically used to update the initial reflection component based on the N observed images, the second illumination component, and the N reflection constraint information to obtain the second reflection component of the shooting scene;

[0475] The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

[0476] Thus, since the initial reflection component can be updated based on N observed images, the second illumination component, and N reflection constraint information, the updated reflection component can be controlled to remain spatially smooth through the reflection constraint information. This ensures that the updated reflection component better matches the true reflected illumination of the physical illumination field of the shooting scene. When further updating the illumination component of the shooting scene based on the updated reflection component, the illumination component can be guided to move closer to the true illumination in the physical illumination field, thereby further improving the estimation accuracy of the illumination component.

[0477] In some possible implementations, the aforementioned processor 1510 is specifically used for:

[0478] The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band.

[0479] The RGB image of the scene being photographed is acquired using an RGB sensor.

[0480] The processor 1510 is further configured to perform de-mosaic processing on the N mosaic images acquired by the processor 1510 based on the RGB image acquired by the processor 1510, to obtain N observation images.

[0481] Thus, since RGB images can be used to guide the de-mosaicing of N mosaic images acquired by a multispectral sensor, that is, the high spatial resolution information of RGB images can be used to restore the missing texture details of the mosaic images, the N observation images obtained after de-mosaicing can provide richer spatial details for subsequent illumination component estimation, thereby ensuring the accuracy of illumination estimation.

[0482] In some possible implementations, the aforementioned processor 1510 is specifically used for:

[0483] Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images;

[0484] The N upsampled images are registered with the RGB image to obtain N registered images;

[0485] Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image;

[0486] The guide image corresponding to each registration image is a channel image of the RGB image.

[0487] Thus, since RGB images can be used to sequentially upsample, register, and guide filtering N mosaic images, the high-frequency details of the RGB images can be injected into the N mosaic images under the N bands. This can significantly improve the spatial resolution of the N observation images obtained after de-mosaicing, thereby ensuring that the N observation images provide richer and finer edges of illumination changes for illumination estimation, and thus improving the accuracy of illumination estimation.

[0488] In some possible implementations, the processor 1510 is further configured to perform a first processing on the first reflection component before generating the first image based on the first reflection component;

[0489] The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

[0490] Thus, on the one hand, denoising based on low-rank spectral priors can remove high-frequency, sparse noise from the first reflection component while preserving the object's true, low-frequency reflection signal, thereby improving the reliability of the first reflection component. On the other hand, detail enhancement based on spatially guided filtering can suppress local block effects or residual noise in smooth areas of the first reflection component, and sharpen and protect the object's true texture and edge details, preventing the image from becoming overly smooth due to denoising, thus ensuring that the final enhancement result has better visual clarity and spatial detail representation. Furthermore, spectral interpolation of the first reflection component can reconstruct a more continuous spectral curve from discretely sampled reflectance, making subsequent color space conversion calculations more accurate, helping to reduce color calculation errors caused by insufficient spectral sampling, and improving the color fidelity of the enhanced image. Therefore, by performing targeted first processing on the first reflection component, noise can be suppressed in the spectral dimension to improve data quality, or details can be enhanced in the spatial dimension to improve visual clarity. This results in an enhanced image that is not only more realistic in color, but also cleaner and sharper visually, thus ensuring that the image quality of the enhanced image is further optimized.

[0491] It should be understood that, in this embodiment, the input unit 1504 may include a graphics processing unit (GPU) 15041 and a microphone 15042. The GPU 15041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode; wherein, the capture device may include an RGB sensor and a multispectral sensor. The display unit 1506 may include a display panel 15061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0492] The memory 1509 can be used to store software programs and various data. The memory 1509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0493] Processor 1510 may include one or more processing units; optionally, processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1510.

[0494] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0495] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0496] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0497] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0498] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0499] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0500] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0501] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain N observation images of the shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image; Based on the N observed images and N illumination constraint information, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different wavelength bands; Based on the first illumination component and the N observed images, the first reflection component of the shooting scene is determined; A first image is generated based on the first reflection component.

2. The method according to claim 1, characterized in that, The illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of the illumination distribution of the shooting scene in one wavelength band.

3. The method according to claim 1 or 2, characterized in that, The step of estimating the first illumination component of the shooting scene based on the N observed images and N illumination constraint information includes: Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined; Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene; Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene; The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

4. The method according to claim 3, characterized in that, The step of updating the initial reflection component based on the N observed images and the second illumination component to obtain the second reflection component of the shooting scene includes: Based on the N observed images, the second illumination component, and the N reflection constraint information, the initial reflection component is updated to obtain the second reflection component of the shooting scene; The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

5. The method according to claim 1, characterized in that, The acquisition of N observation images of the shooting scene in N bands includes: The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band. The RGB image of the scene being photographed is acquired using an RGB sensor. Based on the RGB image, the N mosaic images are de-mosaiced to obtain N observation images.

6. The method according to claim 5, characterized in that, The process involves de-mosaicing the N mosaic images based on the RGB image to obtain N observation images, including: Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images; The N upsampled images are registered with the RGB image to obtain N registered images; Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image; The guide image corresponding to each registration image is a channel image of the RGB image.

7. The method according to claim 1, characterized in that, Before generating the first image based on the first reflection component, the method further includes: The first reflection component undergoes a first processing step; The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire N observation images of the shooting scene in N bands, where N is an integer greater than 4; one band corresponds to one observation image. Processing module, used for: Based on the N observed images and N illumination constraint information acquired by the acquisition module, the first illumination component of the shooting scene is estimated; the illumination constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different bands. Based on the first illumination component and the N observed images, the first reflection component of the shooting scene is determined; A first image is generated based on the first reflection component.

9. The apparatus according to claim 8, characterized in that, The illumination constraint information includes first constraint information and second constraint information; the first constraint information is used to constrain the correlation between the illumination distribution of the shooting scene in different wavelength bands; the second constraint information is used to constrain the spatial smoothness of the illumination distribution of the shooting scene in one wavelength band.

10. The apparatus according to claim 8 or 9, characterized in that, The processing module is specifically used for: Based on the N observed images, the initial illumination component and initial reflection component of the shooting scene are determined; Based on N observed images, the initial reflection component, and the N illumination constraint information, the initial illumination component is updated to obtain the second illumination component of the shooting scene; Based on the N observed images and the second illumination component, the initial reflection component is updated to obtain the second reflection component of the shooting scene; The illumination component and reflection component of the shooting scene are iteratively updated to obtain the first illumination component of the shooting scene.

11. The apparatus according to claim 10, characterized in that, The processing module is specifically used to update the initial reflection component based on the N observed images, the second illumination component, and the N reflection constraint information, to obtain the second reflection component of the shooting scene; The reflection constraint information is used to constrain the spatial smoothness of the reflection distribution of the shooting scene in a certain band.

12. The apparatus according to claim 8, characterized in that, The acquisition module is specifically used for: The scene is captured using a multispectral sensor with a narrowband filter array, which acquires N mosaic images of the scene in N bands, with one mosaic image corresponding to one band. The RGB image of the scene being photographed is acquired using an RGB sensor. The processing module is further configured to perform de-mosaic processing on the N mosaic images acquired by the acquisition module based on the RGB image acquired by the acquisition module, to obtain N observation images.

13. The apparatus according to claim 12, characterized in that, The processing module is specifically used for: Based on the dimensions of the RGB image, the N mosaic images are upsampled to obtain N upsampled images; The N upsampled images are registered with the RGB image to obtain N registered images; Based on the guide image corresponding to each registration image, guide filtering is performed on each registration image to obtain the observation image corresponding to each registration image; The guide image corresponding to each registration image is a channel image of the RGB image.

14. The apparatus according to claim 8, characterized in that, The processing module is further configured to perform a first processing on the first reflection component before generating the first image based on the first reflection component; The first processing includes at least one of the following: denoising based on low-rank spectral priors; detail enhancement based on spatially guided filtering; and interspectral interpolation.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1-7.

Images