Image processing method, apparatus and electronic device

Abstract

An image processing method, comprising: acquiring a color image collected by a main imaging camera and a first multispectral image collected by a multispectral camera, photographed objects contained in the color image and in the first multispectral image being the same; and, on the basis of the first multispectral image, performing shading correction on the color image to obtain a corrected target image. Performing shading correction on color images by means of high-dimensional multispectral images can avoid errors of removing scene textures as uneven vignetting in shading correction processes of the color images.

Description

Image processing method, device and electronic equipment

[0001] This application claims priority to Chinese patent application No. 202410372576.4 filed on March 28, 2024, entitled “A method, device and electronic device for image processing,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method, device and electronic device. Background Art

[0003] The lens is a key component of a camera module, typically composed of a series of lenses. Its primary task is to control the incoming light and ensure it is properly transmitted to the image sensor. However, due to the varying angles of incident light and the varying refractive indices of lenses for different wavelengths, the resulting image may contain shadows. For example, when capturing an image through the lens of an electronic device, the image may appear bright in the center and gradually darken around the edges, severely impacting image quality.

[0004] Generally speaking, the shadows caused by the lens that appear on the image can be divided into brightness shadows (luma shading) and color shadows (color shading). Among them, brightness shadows are mainly affected by the optical path of the lens, vary little with the spectrum, and can be characterized through a single offline calibration. Color shadows vary significantly with different spectra, and it is necessary to calibrate the differences in module shadows under different spectral lights and dynamically calculate them accordingly. Because the shadow characteristics of different camera modules vary, in actual projects, each camera module needs to be independently calibrated and stored separately, making the process of correcting color images time-consuming and labor-intensive, the correction quality is difficult to unify, and it requires a large amount of storage resources, resulting in poor correction results. Therefore, how to improve the effect of lens shading correction is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The present application provides an image processing method, device, electronic device, computer storage medium and computer product, which can improve the accuracy of lens shading correction.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a main imaging camera and a multispectral camera, and includes: obtaining a color image captured by the main imaging camera and a first multispectral image captured by the multispectral camera, wherein the color image and the first multispectral image contain the same photographed object; based on the first multispectral image, performing shadow correction on the color image to obtain a corrected target image.

[0007] In this solution, the primary camera captures a color image. Compared to the primary camera, a multispectral camera has stronger spectral discrimination capabilities, allowing for more accurate identification of incident light with different spectral distributions. Furthermore, its pixels are larger, resulting in less shading caused by oblique crosstalk. Therefore, shading correction can be performed on the color image using the first multispectral image. This prevents scene texture in the color image from being misinterpreted as uneven vignetting and thus requiring removal.

[0008] In one possible implementation, performing shading correction on a color image includes: performing feature extraction on the color image and the first multispectral image respectively to obtain first image features and second image features; obtaining a first correction table based on the first image features and the second image features; and performing shading correction on the color image based on the first correction table.

[0009] That is to say, when performing shadow correction on a color image, a correction table of the color image can be obtained by extracting image features from the color image and the first multispectral image to correct the color image, thereby improving the accuracy of shadow correction on the color image.

[0010] In one possible implementation, performing shading correction on a color image includes: mapping a first multispectral image to the same space as the color image to obtain a second multispectral image; selecting a second correction table from a set of correction tables based on a ratio of pixels in the second multispectral image and the color image; and performing shading correction on the color image based on the second correction table.

[0011] That is to say, before performing shadow correction on a color image, a shadow correction table calibrated under multiple standard light sources can be obtained in advance. By using the ratio of pixels in the multispectral image and the color image, a second correction table can be screened out from the shadow correction table calibrated under multiple standard light sources. This eliminates the ambiguity of the table selection index for the gradient caused by texture and shadow, making shadow correction of the color image based on the second correction table more accurate and stable.

[0012] In one possible implementation, after mapping the first multispectral image to the same space as the color image to obtain a second multispectral image, the method further includes: performing pixel-level alignment on the color image and the second multispectral image so that the color image and the second multispectral image have the same pixels.

[0013] In other words, the multispectral camera array used to acquire multispectral images has an 8-channel wide-spectrum response. The color space formed by this response is the parent space of traditional RGGB and RYYB cameras, while the color image to be corrected has three RGB channels. To ensure that the acquired multispectral image can be used for shading correction of color images, it is necessary to map the multispectral image into the imaging space of the color image.

[0014] In one possible implementation, performing shadow correction on the color image based on the first multispectral image includes: performing shadow correction on the first multispectral image to obtain a corrected first multispectral image; and performing shadow correction on the color image based on the corrected first multispectral image.

[0015] That is to say, when performing shading correction on the color image using the first multispectral image, high-dimensional lens shading correction can be performed on the multispectral image with lighter shading, so that the corrected multispectral image has more uniform scene content, thereby enabling more accurate shading correction of the color image.

[0016] In one possible implementation, shading correction is performed on the first multispectral image to obtain a corrected first multispectral image, including: determining a lens shading correction compensation table for the first multispectral image based on a spectral distribution in the first multispectral image and a pre-calibrated calibration basis; and performing shading correction on the first multispectral image using the lens shading correction compensation table to obtain the corrected first multispectral image.

[0017] That is, when performing shading correction on the first multispectral image, the shading correction can be performed on the first multispectral image by obtaining a lens shading compensation table for the first multispectral image. Performing high-dimensional lens shading correction on the first multispectral image with lighter shading ensures that the corrected multispectral image has more uniform scene content, thereby enabling more accurate shading correction of the color image.

[0018] In a possible implementation, the color image and the first multispectral image are taken at the same time.

[0019] That is, by simultaneously capturing the color image and the first multispectral image, it is ensured that the captured color image and the first multispectral image contain the same subject. This ensures that when shading correction is performed on the color image based on the first multispectral image, the accuracy of the shading correction can be guaranteed.

[0020] In a second aspect, the present application provides an image processing device, comprising: a main imaging camera for capturing a color image; a multispectral camera for capturing a first multispectral image, wherein the first multispectral image and the color image contain the same shooting object; and a processing module for performing shadow correction on the color image based on the first multispectral image to obtain a corrected target image.

[0021] In one possible implementation, the processing module is used to: perform feature extraction on the color image and the first multispectral image respectively to obtain first image features and second image features; obtain a first correction table based on the first image features and the second image features; and perform shadow correction on the color image based on the first correction table.

[0022] In one possible implementation, the processing module is used to: map the first multispectral image to the same space as the color image to obtain a second multispectral image; filter a second correction table from a set of correction tables based on a ratio of pixels in the second multispectral image and the color image; and perform shading correction on the color image based on the second correction table.

[0023] In one possible implementation, after mapping the first multispectral image to the same space as the color image to obtain the second multispectral image, the processing module is further used to: perform pixel-level alignment on the color image and the second multispectral image so that the color image and the second multispectral image have the same pixels.

[0024] In one possible implementation, the processing module is configured to: perform shadow correction on the first multispectral image to obtain a corrected first multispectral image; and perform shadow correction on the color image based on the corrected first multispectral image.

[0025] In one possible implementation, the processing module is used to: determine a lens shading correction compensation table for the first multispectral image based on the spectral distribution in the first multispectral image and a pre-calibrated calibration base; and perform shading correction on the first multispectral image using the lens shading correction compensation table to obtain a corrected first multispectral image.

[0026] In a possible implementation, the color image and the first multispectral image are taken at the same time.

[0027] In a third aspect, the present application provides an electronic device comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by an electronic device, enables the electronic device to execute the method described in the first aspect or any possible implementation of the first aspect.

[0030] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0032] FIG2 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

[0033] FIG3 is a schematic diagram of a process for performing shadow correction on a color image based on a first multispectral image according to an embodiment of the present application;

[0034] FIG4 is a schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0035] FIG5 is a schematic diagram of a process for performing shadow correction on a color image based on a corrected first multispectral image according to an embodiment of the present application;

[0036] FIG6 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0037] FIG7 is a schematic structural diagram of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0039] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0040] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0041] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0042] Generally, when performing lens shading correction, the original image and shading correction tables corresponding to multiple standard light sources are obtained, the first eigenvalue and the second eigenvalue of each standard light source are calculated according to the three color values ​​in the multiple shading correction tables, and the first eigenvalue set and the second eigenvalue set are obtained. The chromatic aberration correction table is obtained and the first eigenvalue and the second eigenvalue in the chromatic aberration correction table are counted. According to the comparison result of the first eigenvalue and the first eigenvalue set and the comparison result of the second eigenvalue and the second eigenvalue set in the chromatic aberration correction table, the shading correction tables corresponding to the two target standard light sources are determined. The target shading correction table is calculated according to the shading correction tables corresponding to the two target standard light sources, and the shading compensation is performed on the original image.

[0043] Although this correction method can correct lens shading during the imaging process, it requires pre-calculating shading correction tables corresponding to multiple standard light sources as candidate tables. The process of selecting the optimal compensation table from a large number of candidate tables cannot eliminate the ambiguity between texture and shading.

[0044] In view of this, an embodiment of the present application provides an image processing method, which can simultaneously capture a color image and a multispectral image during the process of an electronic device shooting an image, so that the shooting objects contained in the color image and the multispectral image are the same. Then, high-dimensional shading is removed from the multispectral image to obtain a corrected multispectral image. The corrected multispectral image does not have the influence of shading. Shadow correction is performed on the color image based on the corrected multispectral to obtain a target image. By performing shadow correction on the color image using the multispectral image with high-dimensional shading removed, the error of removing the scene texture as uneven shading can be avoided during the shadow correction of the color image.

[0045] It can be understood that the electronic device in the embodiments of the present application can be a mobile phone, a smart car, a sweeping robot, an augmented reality (AR) device, a virtual reality (VR) device, or other device with an image capture function or an image recognition function.

[0046] For example, Figure 1 shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. As shown in Figure 1, the electronic device 100 may include: a processor 110, a main imaging camera 120, a multispectral camera 130, a memory 140, a display 150, a sensor module 160, and a battery 170.

[0047] The main imaging camera 120 is used to capture the target image required by the electronic device 100. The image captured by the main imaging camera 120 can be a color image. In the embodiment of the present application, a color image refers to an image in which each pixel is composed of R, G, and B components, where R, G, and B are described by different grayscale levels.

[0048] The multispectral camera 120 is used to capture multispectral images. The multispectral camera 120 can be an array-type multispectral camera, which trades spatial resolution for frequency domain resolution. By increasing the number of dyes in the filter cycle unit, a multispectral array can achieve color response in more channels. Compared to three-color imaging modules, array-type multispectral modules have stronger spectral discrimination capabilities and can more accurately identify incident light with different spectral distributions. Furthermore, each pixel size is larger, and the degree of shading caused by oblique light crosstalk is less. Therefore, high-dimensional shading correction based on array multispectral images has the advantage of being easier and more accurate. In some embodiments, when using the electronic device 100 to capture images, the main imaging camera 120 and the multispectral camera 130 can capture images simultaneously or with a short time difference. However, during the same capture process, the color image captured by the main imaging camera 120 and the multispectral image captured by the multispectral camera 130 contain the same subject.

[0049] The processor 110 is the computing and control core of the electronic device 100. The processor 110 may include one or more processing units. For example, the processor 110 may include one or more of an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. In this embodiment, the NPU may be a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can quickly process input information and can also continuously self-learn. The NPU can enable intelligent cognitive applications of the electronic device 100, such as image recognition, face recognition, speech recognition, text comprehension, and text recognition. In some embodiments, after acquiring a multispectral image captured by the multispectral camera 130, the processor 110 may remove shading from the multispectral image to obtain a corrected multispectral image. The processor 110 can then perform shading correction on the color image captured by the main imaging camera 120 based on the corrected spectral image. For example, after acquiring the color image and the corrected multispectral image, the processor 110 can input the color image and the corrected multispectral image into two sub-neural networks, respectively, to obtain feature images corresponding to the color image and the corrected multispectral image. A target correction table is generated based on the feature images corresponding to the color image and the corrected multispectral image. Finally, the processor 110 performs shading correction on the color image captured by the main imaging camera 120 based on the target correction table. In other embodiments, after acquiring the multispectral image captured by the multispectral camera 130, the processor 110 can remove shading from the multispectral image to obtain a corrected multispectral image, and then map the corrected multispectral image into the same space as the color image. Then, based on the pixel ratio between the spatially mapped multispectral image and the color image, the processor 110 selects a target correction table from a plurality of pre-stored correction tables corresponding to standard light sources. Finally, the processor 110 performs shading correction on the color image captured by the main imaging camera 120 based on the target correction table.

[0050] The memory 140 may store a program that can be executed by the processor 110, so that the processor 110 executes part or all of the methods required to be executed by the electronic device 100 provided in the embodiments of the present application. The memory 140 may also store data. The processor 110 may read the data stored in the memory 140. The memory 140 and the processor 110 may be provided separately. Alternatively, the memory 140 may also be integrated into the processor 110.

[0051] The display screen 150 is used to display images, videos, and the like. For example, it displays a color image captured by the primary imaging camera 120 and shading-corrected by the processor 110. The display screen 150 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED).

[0052] The electronic device 100 may further include a sensor module 160. The sensor module 160 may include a touch sensor. The touch sensor may be used to detect touch operations acting on or near the electronic device 100. Exemplarily, the touch sensor may be implemented in a variety of ways, such as resistive, capacitive, infrared, and surface acoustic wave. The touch sensor may be disposed on the display screen 150, and the touch sensor and the display screen 150 may form a touch screen, also known as a "touch screen." Of course, the touch sensor may also be disposed on the surface of the electronic device 100, at a different location from the display screen 150. In some embodiments, the sensor module 160 may also include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, an ambient light sensor, or a bone conduction sensor.

[0053] A battery 170 may also be provided on the electronic device 100. The battery 170 may be used to power the electronic device 100.

[0054] It is understandable that the above-mentioned electronic device can be a mobile phone, a smart car, a sweeping robot, AR / VR glasses, or other devices with image capture or image recognition functions. The structure illustrated in FIG1 of the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0055] The above is an introduction to the electronic device provided by the embodiment of the present application. Based on the above content, the image processing method provided by the embodiment of the present application is introduced below.

[0056] For example, FIG2 shows a flowchart of an image processing method provided by an embodiment of the present application. The method can be executed by the electronic device 100 shown in FIG1 , and as shown in FIG2 , the method includes: Step 201 - Step 202 .

[0057] Step 201 : Acquire a color image captured by a main imaging camera and a first multispectral image captured by a multispectral camera, wherein the color image and the first multispectral image contain the same photographic object.

[0058] In this embodiment, the electronic device includes a main imaging camera and a multispectral camera. The main imaging camera can be used to capture color images, and the multispectral camera can be used to capture multispectral images. A color image is an image in which each pixel is composed of R, G, and B components, where R, G, and B are described by different grayscale levels. When the electronic device captures an image, the main imaging camera and the multispectral camera can capture images simultaneously, so that the captured color image and the first multispectral image contain the same subject.

[0059] Step 202 : Based on the first multispectral image, perform shadow correction on the color image to obtain a corrected target image.

[0060] In this embodiment, after acquiring the color image and the first multispectral image, the electronic device can use the first multispectral image to perform shading correction on the color image, wherein the first multispectral image can be an image with shading removed. For example, both can be input into a neural network model, and the first multispectral image can guide the neural network model to perform shading correction on the color image. Then, the neural network model outputs the target image after lens shading correction. Compared with the main imaging camera, the multispectral camera has a stronger spectrum discrimination capability and can more accurately identify incident light with different spectral distributions; at the same time, the size of each pixel is also larger, and the degree of shading caused by oblique light crosstalk is also lighter. Therefore, the first multispectral image can be used to guide the neural network model to perform shading correction on the color image.

[0061] In an embodiment of the present application, by performing shading correction on a color image using a multispectral image containing the same photographed object as the color image, the error of removing the scene texture as uneven shading during the shading correction of the color image can be avoided, thereby improving the accuracy of lens shading correction.

[0062] In some embodiments, as shown in FIG3 , step 202 may include the following steps:

[0063] Step 301 : Perform shadow correction on a first multispectral image to obtain a corrected first multispectral image.

[0064] In this embodiment, when performing shading correction on the first multispectral image, a lens shading correction compensation table for the first multispectral image can be determined based on the spectral distribution of the first multispectral image and a pre-calibrated calibration base. Shading correction is then performed on the first multispectral image using the obtained lens shading correction compensation table to obtain a corrected first multispectral image.

[0065] As a possible implementation, a lens shading correction (LSC) compensation table for the first multispectral image can be determined based on a first multispectral image captured by a multispectral camera and a pre-calibrated calibration basis. Lens shading correction can then be performed on the first multispectral image using the LSC compensation table to obtain a corrected first multispectral image. Specifically, the electronic device can pre-capture multiple multispectral images of a uniform frosted glass scene offline using the multispectral camera as a calibration basis for the high-dimensional LSC. After acquiring the first multispectral image, the electronic device can input the first multispectral image into a neural network. The neural network estimates the spectral distribution from the multispectral signal contained in the input first multispectral image, decomposes the proportions of each monochromatic primary light source component in the first multispectral image, and reconstructs a uniform frosted glass scene using weighted weights based on the proportions of each monochromatic primary light source component. Finally, the reconstructed uniform frosted glass scene is compared with the pre-obtained high-dimensional LSC calibration basis to calculate a high-dimensional LSC table, which can be used to correct the first multispectral image.

[0066] Compared to primary imaging cameras, multispectral cameras have stronger spectral discrimination capabilities and can more accurately identify incident light with different spectral distributions. Furthermore, their pixels are larger, reducing the degree of shading caused by oblique light crosstalk. Therefore, high-dimensional shading correction based on multispectral images is easier and more accurate.

[0067] Step 302: Perform shadow correction on the color image based on the corrected first multispectral image.

[0068] In this embodiment, after obtaining the corrected first multispectral image, image features of the corrected first multispectral image and the color image can be extracted respectively. Based on the image features of the corrected first multispectral image and the color image, a first correction table corresponding to the color image can be obtained, wherein the first correction table can be used to correct the color image. Alternatively, the corrected first multispectral image can be mapped to the same imaging color space as the color image to obtain a second multispectral image. Based on the ratio of pixels in the second multispectral image and the color image, a second correction table can be selected from the correction table set, wherein the second correction table can be used to correct the color image.

[0069] As a possible implementation, when performing shading correction on a color image based on the acquired corrected first multispectral image, the electronic device can implement shading correction on the color image using a neural network or a collection of multiple neural networks. As shown in Figure 4, after acquiring the color image and the corrected first multispectral image, the electronic device can input the color image into a neural network 410 to obtain first image features corresponding to the color image. Furthermore, the corrected first multispectral image can be input into a neural network 420 to obtain second image features corresponding to the corrected first multispectral image. The acquired first and second image features can then be input into a neural network 430. Neural network 430 can perform feature fusion on the received first and second image features, thereby learning features unrelated to shading texture and generating a first correction table based on the obtained features. Shading correction (i.e., shading removal) can be performed on the color image based on the obtained first correction table. The first correction table is dynamically generated, meaning that after each color image is captured by the main imaging camera on the electronic device, a shading correction table corresponding to the color image is regenerated for use in performing shading correction on the image captured by the main imaging camera.

[0070] It is understandable that the shading texture is removed from the corrected first multispectral image. Therefore, the second feature image corresponding to the corrected first multispectral image extracted by the neural network 420 can include image content and image background, that is, the second feature image does not contain shading texture. However, the first feature image corresponding to the color image extracted by the neural network 410 contains features such as shading texture, image content, and image background. Therefore, the first feature image and the second feature image can be fused to learn features unrelated to shading texture and obtain a correction table corresponding to the color image.

[0071] In one possible example, to ensure the accuracy of the correction table for the resulting color image, the first feature image and the second feature image are fused. When learning features unrelated to shading texture, an unprocessed color image captured by the primary imaging camera can also be input. The correction table corresponding to the color image is regressed based on the first feature image, the second feature image, and the unprocessed color image.

[0072] In one possible example, before extracting the first image features corresponding to the color image through the neural network 410 and extracting the image features corresponding to the corrected first multispectral image through the neural network 420, the color image and the corrected first multispectral image may also be downsampled so that the color image and the corrected first multispectral image have the same resolution.

[0073] In this embodiment, shading is removed from the first multispectral image captured by the multispectral camera, alleviating errors caused by metamerism. Simultaneously, high-dimensional LSC is performed on the multispectral image with less shading, helping to obtain a more uniform record of scene content, thereby guiding the primary imaging path (the color image captured by the primary imaging camera) to perform more accurate shading removal. Secondly, during the processing of the color image captured by the primary imaging camera, the uniform scene content provided by the first multispectral image after LSC is used as a reference, eliminating ambiguity between texture and shadow. Furthermore, the dynamic calculation of the shading compensation table based on common scene features using a neural network eliminates the need for prior feature calibration and parameter storage for any imaging components, saving workload and memory.

[0074] As another possible implementation, when the electronic device performs shading correction on the color image based on the acquired corrected first multispectral image, the process can be implemented in a modular manner. Specifically, as shown in FIG5 , after acquiring the corrected first multispectral image and the color image, the electronic device can include the following steps: Steps 501 to 505.

[0075] Step 501: Sampling the corrected first multispectral image and the color image so that the corrected first multispectral image and the color image have the same resolution.

[0076] In this embodiment, after obtaining the corrected first multispectral image and color image, the corrected first multispectral image and color image can be downsampled respectively so that the obtained corrected first multispectral image and color image have the same resolution. It is understood that step 501 is an optional step.

[0077] Step 502 : Map the corrected first multispectral image to the same imaging color space as the color image to obtain a second multispectral image.

[0078] In this embodiment, the high-dimensional color space of the multispectral image can be mapped to a three-color space using a conversion matrix. Specifically, a fixed 8*3 matrix T can be optimized offline to map the corrected first multispectral image to the imaging color space of the color image, thereby obtaining a second multispectral image. This facilitates subsequent processing of the color image and the second multispectral image in the same color space.

[0079] Step 503 : align the color image and the second multispectral image at the pixel level so that the color image and the second multispectral image have the same pixels.

[0080] In this embodiment, due to issues such as field of view and misalignment between the primary imaging camera and the multispectral camera, pixel-level alignment of the color image and the secondary multispectral image is required. Specifically, the color image and the secondary multispectral image can be aligned based on their internal and external parameters (e.g., focal length) to determine their field of view (FOV).

[0081] Step 504 : Filter out a second correction table from the correction table set based on the ratio of pixels in the second multispectral image and the color image.

[0082] In this embodiment, the electronic device pre-stores shading correction tables calibrated under multiple standard light sources. After acquiring the second multispectral image and the color image, the electronic device can apply the ratio of pixels in the second multispectral image and the color image to the pre-generated shading correction tables calibrated under multiple standard light sources to the current scene statistical graph, calculate the corresponding histogram distribution index, and select the optimal shading correction table from these tables. This shading correction table serves as the second correction table for the color image.

[0083] Step 505: Perform shading correction on the color image based on the second correction table.

[0084] In this embodiment, after obtaining the second correction table, shading correction can be performed on the color image according to the second correction table. It will be appreciated that the second correction table obtained in this embodiment of the present application is dynamically generated. That is, after each color image is captured by the main imaging camera of the electronic device, a shading correction table corresponding to the color image is regenerated for use in performing shading correction on the image captured by the main imaging camera.

[0085] In this embodiment, when determining the corresponding second correction table of the color image from the pre-generated shading correction tables calibrated under multiple standard light sources, multispectral statistics after LSC are introduced to eliminate the ambiguity of the table selection index for the gradient caused by texture and shadow, making shading removal more accurate and stable.

[0086] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual circumstances, and the combined solutions are still within the scope of protection of this application.

[0087] For example, FIG6 shows a schematic structural diagram of an image processing device provided in an embodiment of the present application. The image processing device can be deployed on an electronic device with a shooting function. For example, it can be deployed on a computing node of a related electronic device, and the quality of image color restoration can be improved by adapting the software to the hardware (image sensor). Among them, the computing node can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU) or an application specific integrated circuit (ASIC), etc. As shown in FIG6 , the image processing device may include: a first shooting module 610, a second shooting module 620, and a processing module 630.

[0088] The first camera module 610 is used to acquire a color image. In the embodiment of the present application, a color image refers to an image in which each pixel is composed of R, G, and B components, where R, G, and B are described by different grayscale levels.

[0089] The second shooting module 620 is used to obtain a first multispectral image, wherein the first multispectral image and the color image contain the same shooting object.

[0090] The processing module 630 is used to perform shadow correction on the color image according to the first multispectral image to obtain a corrected target image.

[0091] In one possible example, the processing module 630 performs shading correction on the first multispectral image to obtain a corrected first multispectral image, wherein the corrected first multispectral image can be used to perform shading correction on the color image. Specifically, the processing module 630 can determine a lens shading correction compensation table for the first multispectral image based on the spectral distribution in the first multispectral image and a pre-calibrated calibration basis, and then perform shading correction on the first multispectral image using the lens shading correction compensation table to obtain the corrected first multispectral image.

[0092] In one possible example, after obtaining the corrected first multispectral image, processing module 630 may be configured to perform feature extraction on the color image and the corrected first multispectral image to obtain first and second image features. Processing module 630 then generates a first correction table based on the first and second image features. Finally, processing module 630 performs shading correction on the color image based on the obtained first correction table.

[0093] In another possible example, after obtaining the corrected first multispectral image, processing module 630 may be specifically configured to map the corrected first multispectral image to the same space as the color image to obtain a second multispectral image. Processing module 630 then selects a second correction table from the set of correction tables based on the ratio of pixels in the second multispectral image to the color image. Finally, processing module 630 performs shading correction on the color image based on the second correction table.

[0094] In one possible example, after mapping the corrected first multispectral image to the same space as the color image to obtain the second multispectral image, the processing module 630 is further configured to perform pixel-level alignment on the color image and the second multispectral image so that the color image and the second multispectral image have the same pixels.

[0095] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0096] Based on the methods in the above embodiments, embodiments of the present application provide an electronic device comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors execute the methods in the above embodiments. For example, the electronic device may be, but is not limited to, an electronic device with a camera function, such as a mobile phone, a camera, or a computer.

[0097] Based on the methods in the above embodiments, the present application also provides an image processing device. Please refer to Figure 7, which is a schematic diagram of the structure of an image processing device provided in the present application. As shown in Figure 7, the image processing device 700 includes one or more processors 701 and an interface circuit 702. Optionally, the image processing device 700 may also include a bus 703. Herein:

[0098] The processor 701 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in the form of software. The above-mentioned processor 701 can be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods and steps disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0099] The interface circuit 702 can be used to send or receive data, instructions or information. The processor 701 can use the data, instructions or other information received by the interface circuit 702 to process it, and can send the processing completion information through the interface circuit 702.

[0100] Optionally, the image processing apparatus 700 further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. Part of the memory may also include a non-volatile random access memory (NVRAM).

[0101] Optionally, the memory stores an executable software module or a data structure, and the processor can perform corresponding operations by calling an operation instruction stored in the memory (the operation instruction may be stored in an operating system).

[0102] Optionally, the interface circuit 702 may be configured to output the execution result of the processor 701 .

[0103] It should be noted that the corresponding functions of the processor 701 and the interface circuit 702 can be implemented through hardware design, software design, or a combination of hardware and software, which is not limited here.

[0104] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in a processor.

[0105] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, which, when executed by an electronic device (such as the aforementioned Bluetooth device, etc.), causes the electronic device to execute the method described in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc.

[0106] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions. When the instructions are executed by an electronic device (such as the aforementioned Bluetooth device, etc.), the electronic device executes the method described in the above embodiment.

[0107] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0108] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0110] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the electronic equipment including a main imaging camera and a multispectral camera, including: Acquire a color image captured by the main imaging camera and a first multispectral image captured by the multispectral camera, wherein the color image and the first multispectral image contain the same photographic object; Based on the first multispectral image, shadow correction is performed on the color image to obtain a corrected target image.

2. The method according to claim 1, characterized in that The step of performing shadow correction on the color image includes: Performing feature extraction on the color image and the first multispectral image respectively to obtain first image features and second image features; obtaining a first correction table based on the first image feature and the second image feature; Perform shading correction on the color image based on the first correction table.

3. The method according to claim 1, characterized in that The step of performing shadow correction on the color image includes: Mapping the first multispectral image to the same space as the color image to obtain a second multispectral image; Filtering a second correction table from the correction table set based on a ratio of pixels in the second multispectral image and the color image; Based on the second correction table, shading correction is performed on the color image.

4. The method according to claim 3, characterized in that After mapping the first multispectral image to the same space as the color image to obtain a second multispectral image, the method further includes: Pixel-level alignment is performed on the color image and the second multispectral image so that the color image and the second multispectral image have the same pixels.

5. The method according to any one of claims 1 to 4, characterized in that Performing shadow correction on the color image based on the first multispectral image includes: performing shadow correction on the first multispectral image to obtain a corrected first multispectral image; Based on the corrected first multispectral image, shadow correction is performed on the color image.

6. The method according to claim 5, characterized in that The step of performing shadow correction on the first multispectral image to obtain a corrected first multispectral image includes: determining a lens shading correction compensation table for the first multispectral image based on a spectral distribution in the first multispectral image and a pre-calibrated calibration basis; The lens shading correction compensation table is used to perform shading correction on the first multispectral image to obtain a corrected first multispectral image.

7. The method according to any one of claims 1 to 6, characterized in that The color image and the first multispectral image are taken at the same time.

8. An image processing device, characterized in that: include: A main imaging camera for capturing color images; a multispectral camera, configured to capture a first multispectral image, wherein the first multispectral image and the color image contain the same object; A processing module is used to perform shadow correction on the color image according to the first multispectral image to obtain a corrected target image.

9. The device according to claim 8, characterized in that The processing module is used for: Performing feature extraction on the color image and the first multispectral image respectively to obtain first image features and second image features; obtaining a first correction table based on the first image feature and the second image feature; Perform shading correction on the color image based on the first correction table.

10. The device according to claim 8, characterized in that The processing module is used for: Mapping the first multispectral image to the same space as the color image to obtain a second multispectral image; Filtering a second correction table from the correction table set based on a ratio of pixels in the second multispectral image and the color image; Based on the second correction table, shading correction is performed on the color image.

11. The device according to claim 10, characterized in that After mapping the first multispectral image to the same space as the color image to obtain a second multispectral image, the processing module is further configured to: Pixel-level alignment is performed on the color image and the second multispectral image so that the color image and the second multispectral image have the same pixels.

12. The device according to any one of claims 8 to 11, characterized in that The processing module is used for: performing shadow correction on the first multispectral image to obtain a corrected first multispectral image; Based on the corrected first multispectral image, shadow correction is performed on the color image.

13. The device according to claim 12, characterized in that The processing module is used for: determining a lens shading correction compensation table for the first multispectral image based on a spectral distribution in the first multispectral image and a pre-calibrated calibration basis; The lens shading correction compensation table is used to perform shading correction on the first multispectral image to obtain a corrected first multispectral image.

14. The device according to any one of claims 8 to 13, characterized in that The color image and the first multispectral image are taken at the same time.

15. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by an electronic device, enable the electronic device to perform the method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that When the instruction is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images