Image correction method, apparatus and electronic device

Abstract

The application provides an image correction method, device and electronic equipment. The image correction device can include a first input interface, a second input interface, a light source estimation module and a color correction module. The first input interface can be used to receive a multispectral image signal from a multispectral sensor, and the second input interface can be used to receive a three-channel image signal from a three-channel sensor. The light source estimation module can be used to obtain a correction parameter corresponding to each light source in a plurality of light sources according to the multispectral image signal. The color correction module can be used to correct the three-channel image signal according to the correction parameter. The technical scheme provided in the embodiments of the application can identify a plurality of light sources in a scene by using a multispectral image signal, achieve more accurate multi-light source estimation and image correction, and effectively improve the accuracy of color correction in a multi-light source scene, so as to more accurately restore the colors in the photographed scene.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image correction method, apparatus and electronic device. Background Technology

[0002] In recent years, with the continuous development of electronic technology, mobile phones, tablets, digital cameras, and other electronic devices with shooting functions have become an indispensable part of people's lives. Because images captured by camera modules may exhibit color casts, it is usually necessary to correct the image signals captured by the camera module.

[0003] However, existing solutions can only correct light sources of a single color in a scene. In scenes with multiple light sources, correcting the color of a single light source may lead to inaccurate local color correction in the image, failing to accurately reproduce the colors of the captured scene. For example, correcting the primary light source may not prevent color casts caused by secondary light sources from remaining visible. Summary of the Invention

[0004] This application provides an image correction method, apparatus, and electronic device to solve the problem of inaccurate color correction of images in multi-light source scenarios.

[0005] In a first aspect, embodiments of this application provide an image correction device, which may include: a first input interface, a second input interface, a light source estimation module, and a color correction module. The first input interface can be used to receive multispectral image signals from a multispectral sensor, and the second input interface can be used to receive three-channel image signals from a three-channel sensor. The light source estimation module can be used to obtain correction parameters corresponding to each of the multiple light sources based on the multispectral image signals. The color correction module can be used to correct the three-channel image signals according to the correction parameters.

[0006] In the image correction device provided in this application embodiment, the light source estimation module can estimate multiple light sources in a scene based on multispectral image signals to obtain correction parameters corresponding to each light source. The color correction matrix can correct the three-channel image signal based on the correction parameters to obtain an image. Compared to a three-channel sensor, a multispectral sensor has more spectral channels and can collect spectral information from more light sources. Therefore, the technical solution provided in this application embodiment can identify multiple light sources in a scene using multispectral image signals, achieving more accurate multi-light source estimation and image correction. This can effectively improve the accuracy of color correction in multi-light source scenes, so as to more accurately restore the colors in the photographed scene.

[0007] The image correction device provided in this application embodiment may further include a processor. The functions of the aforementioned light source estimation module and color correction module can be implemented through integrated logic circuits in the processor hardware and / or instructions in software form. In specific implementations, the processor can be a chip, such as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), system-on-chip (SoC), central processing unit (CPU), network processor (NP), digital signal processor (DSP), microcontroller unit (MCU), programmable logic device (PLD), or any other chip with processing capabilities. Of course, the processor can also be any other device with data processing capabilities, which will not be listed here.

[0008] In embodiments of this application, the light source estimation module can perform multi-light source estimation on a scene based on multispectral image signals, determine the light source information of each light source in the scene, and determine the correction parameters corresponding to each light source based on the light source information of each light source in the scene. For example, the light source information of each light source in the scene may include the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. In practical applications, the light sources in a shooting scene can be divided into indoor light sources, outdoor light sources, and display light sources (such as light sources formed by computer or television screens). Since the number of light sources in most shooting scenes will not exceed three, the number of light sources obtained by the light source estimation module through multi-light source estimation can be less than or equal to three. This reduces the complexity of hardware and algorithms and improves the computing speed of the image correction device while meeting the image correction requirements of most scenes. In one possible implementation, the correction parameters may include at least one of a white balance gain value or a color correction matrix. For example, when there are two light sources in the scene, indoor and outdoor, by performing multi-light source estimation on the multispectral image signal, the white balance gain value and color correction matrix under the indoor light source, as well as the white balance gain value and color correction matrix under the outdoor light source, can be calculated.

[0009] In some embodiments of this application, the light source estimation module can also be used to obtain grid coefficients for calculating partitions based on the multispectral image signal. Since the multispectral image signal has many spectral channels, it can distinguish image regions corresponding to different light sources. Therefore, the grid coefficients obtained from the multispectral image signal can represent light source partitioning information. For example, the grid coefficients can be bilateral grid coefficients. The color correction module can specifically be used to sample the three-channel image signal based on the grid coefficients to obtain a light source partitioning map. For example, slicing can be used to process the three-channel image signal. The light source partitioning map is essentially a weighted map, used to represent the weight of each light source at each pixel. Compared to the three-channel image signal, the multispectral image signal has a stronger ability to identify multi-light source scenes. Therefore, by sampling the light source partitioning information from the multispectral image signal onto the three-channel image signal, the image can be accurately partitioned according to different light sources.

[0010] Furthermore, the color correction module can also be used to correct three-channel image signals using correction parameters and a light source zoning map. In practice, the weight of each light source at each pixel can be determined based on the light source zoning map. Based on the weight of each light source at each pixel and the corresponding correction parameters, color correction can be performed on each pixel. For example, assuming a scene contains both indoor and outdoor light sources, when correcting the color of a particular pixel, the color can be corrected based on the weight of the indoor light source at that pixel and its corresponding correction parameters, as well as the weight of the outdoor light source at that pixel and its corresponding correction parameters.

[0011] In some embodiments of this application, the image correction device may further include a downsampling module, which can be used to downsample the multispectral image signal, thereby downsampling the original resolution multispectral image signal into a low resolution image signal to reduce the computational load of subsequent processing. The downsampling module can also provide the downsampled multispectral image signal to a light source estimation module, which can specifically be used to obtain correction parameters and grid coefficients based on the downsampled multispectral image signal. Generating correction parameters and grid coefficients using the downsampled multispectral image signal can effectively reduce the computational load of the light source estimation module, enabling real-time video-level image processing. In practical applications, due to the different spatial positions of the multispectral sensor and the three-channel sensor, there may be deviations between the multispectral image signal and the three-channel image signal. In some embodiments, the color correction module may also be used to align the grid coefficients and the three-channel image signal before sampling the three-channel image signal according to the grid coefficients to obtain a light source partition map. In specific implementations, the grid coefficients and the three-channel image signal can be aligned according to the spatial relationship between the multispectral sensor and the three-channel sensor. Specifically, the spatial relationship between the multispectral sensor and the three-channel sensor can be aligned through calibration or optical flow. Utilizing their relative spatial positions, operations such as cropping, warping, and resizing are performed on the grid coefficients to obtain grid coefficients aligned with the three-channel image signal. In this embodiment, the computational requirements for aligning the grid coefficients are relatively low, enabling real-time multi-source image processing on video.

[0012] In some other embodiments of this application, the image correction device may further include a downsampling module, which can be used to downsample the multispectral image signal and the three-channel image signal, thereby downsampling the original resolution multispectral image signal and the three-channel image signal into low-resolution image signals to reduce the computational load of subsequent processing. The downsampling module can also provide the downsampled multispectral image signal and the downsampled three-channel image signal to a light source estimation module, which can specifically be used to obtain correction parameters and grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal. Using the downsampled multispectral image signal and the three-channel image signal to generate correction parameters and grid coefficients can effectively reduce the computational load of the light source estimation module, enabling real-time video-level image processing. Furthermore, since the grid coefficients are generated based on the multispectral image signal and the three-channel image signal, the spatial accuracy of subsequent operations such as calculating the light source partition map and color correction is higher. Moreover, compared to the embodiment shown in the figure, the embodiment shown in the figure does not require positional alignment of the grid coefficients and the three-channel image signal, which can save computational steps in the light source estimation module and improve the computational speed of the light source estimation module.

[0013] Furthermore, the light source estimation module in this embodiment can also be used to obtain a guide map based on the three-channel image signal. The color correction module can specifically be used to sample the three-channel image signal to obtain a light source partition map based on the grid coefficients and the guide map. The guide map is a reference image associated with the three-channel image signal and can be used to guide the direction during the generation of the light source partition map. In this embodiment, the guide map is used to guide the generation process of the light source partition map and optimize the processing at the boundaries of different light source partitions. This avoids edge false color problems such as blurring and color deviation at the boundaries of different light source partitions, thus making the image corrected by the color correction module closer to the real image seen by the human eye. In specific implementation, the light source estimation module can obtain the guide map based on the three-channel image signal at the original resolution, thereby enriching the image information in the guide map and providing better guidance for the generation process of the light source partition map. Furthermore, if the guide map is calculated using multispectral image signals, the generated light source partition map needs to be aligned to the three-channel image signal. To avoid edge false color problems, high alignment accuracy is required, making the calculation difficult. In this embodiment, the guide map is calculated using the three-channel image signal. The generated light source partition map does not need to be aligned with the three-channel image signal, which can reduce the amount of calculation and speed up the operation of the image processing device.

[0014] In one possible implementation, the aforementioned light source estimation module may include a neural processing unit (NPU) and a data processing unit. For example, the neural processing unit may be a convolutional neural network (CNN). The neural processing unit can be used to perform multi-light source estimation, calculate grid coefficients, and calculate guidance maps based on multispectral image signals. The data processing unit can be used to calculate the correction parameters corresponding to each light source based on the light source information of each light source in the scene.

[0015] In practical implementation, the aforementioned color correction module can be implemented through an image signal processor (ISP). The operations of the color correction module, such as calculating the light source partition map and correcting the three-channel image signals according to the correction parameters, can be hard-coded into the image signal processor, which can reduce the real-time data processing volume of the image correction device, thereby realizing video-level multi-light source image processing.

[0016] Secondly, embodiments of this application also provide an electronic device, which may include any of the image correction devices described in the first aspect above. The image correction device in this application embodiment may be an image processing chip, an image processor, or other devices with processing functions. The electronic device in this application embodiment may further include a three-channel sensor and a multispectral sensor. The three-channel sensor may be a photosensitive sensor of at least one of a main camera, a telephoto camera, and a wide-angle camera. In one possible implementation, the multispectral sensor may have multiple photosensitive units arranged in an array, enabling the multispectral sensor to collect more information about the light source spectrum and provide spectral information close to the spectral response curve of the human eye.

[0017] Thirdly, embodiments of this application also provide an image correction method. The image correction method in embodiments of this application can be executed in any of the image correction devices mentioned in the first aspect above, for example, it can be executed in the processor of the image correction device mentioned above.

[0018] The image correction method provided in this application embodiment may include:

[0019] Step 1: Receive multispectral image signals from the multispectral sensor and three-channel image signals from the three-channel sensor;

[0020] Step 2: Obtain the correction parameters corresponding to each of the multiple light sources based on the multispectral image signal;

[0021] Step 3: Correct the three-channel image signal according to the correction parameters.

[0022] In the image processing method provided in this application embodiment, multi-light source estimation of the scene is performed based on multispectral image signals to obtain correction parameters corresponding to each of the multiple light sources. The three-channel image signal is then corrected according to these correction parameters to obtain an image. Compared to a three-channel sensor, a multispectral sensor has more spectral channels, allowing it to collect spectral information from more light sources. Therefore, the technical solution provided in this application embodiment utilizes multispectral image signals to identify multiple light sources in a scene, achieving more accurate multi-light source estimation and image correction. This effectively improves the accuracy of color correction in multi-light source scenes, enabling more accurate color reproduction of the captured scene.

[0023] In embodiments of this application, step two may specifically include: performing multi-light source estimation on the scene based on the multispectral image signal, determining the light source information of each light source in the scene, and determining the correction parameters corresponding to each light source based on the light source information of each light source in the scene. For example, the light source information of each light source in the scene may include the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. In practical applications, the light sources in the shooting scene can be divided into indoor light sources, outdoor light sources, and display light sources (such as light sources formed by computer and television screens). Since the number of light sources in most shooting scenes will not exceed three, the number of light sources obtained from multi-light source estimation in step two can be less than or equal to three. This reduces the complexity of hardware and algorithms and improves the processing speed of the image correction device while meeting the image correction requirements of most scenes. In one possible implementation, the correction parameters may include at least one of a white balance gain value or a color correction matrix. For example, when there are two light sources in a scene, an indoor light source and an outdoor light source, by performing multi-light source estimation on the multispectral image signal, the white balance gain value and color correction matrix under the indoor light source, as well as the white balance gain value and color correction matrix under the outdoor light source, can be calculated.

[0024] In some embodiments of this application, after step one and before step three, the image correction method may further include: obtaining grid coefficients for calculating partitions based on the multispectral image signal. Since the multispectral image signal contains many spectral channels, it can distinguish image regions corresponding to different light sources. Therefore, the grid coefficients obtained from the multispectral image signal can represent light source partitioning information. For example, the grid coefficients can be bilateral grid coefficients.

[0025] Step three above can specifically include: sampling the three-channel image signal according to the grid coefficient to obtain a light source partition map, and correcting the three-channel image signal using correction parameters and the light source partition map. For example, slicing can be used to process the three-channel image signal to obtain the light source partition map. The light source partition map is essentially a weight map, used to represent the weight of each light source at each pixel. Multispectral image signals have a stronger ability to identify multi-light source scenes compared to three-channel image signals. Therefore, sampling the light source partition information from the multispectral image signal onto the three-channel image signal allows for accurate partitioning of the image according to different light sources. In specific implementation, the weight of each light source at each pixel can be determined based on the light source partition map. Based on the weight of each light source at each pixel and the corresponding correction parameters, color correction can be performed on each pixel. For example, if there are two light sources in the scene, an indoor light source and an outdoor light source, when correcting the color of a certain pixel, the color of the pixel can be corrected according to the weight of the indoor light source at that pixel and the corresponding correction parameters of the indoor light source, the weight of the outdoor light source at that pixel and the corresponding correction parameters of the outdoor light source.

[0026] In some embodiments of this application, after step one, the image correction method may further include: downsampling the multispectral image signal, thereby downsampling the original resolution multispectral image signal into a low resolution image signal to reduce the computational load of subsequent processing. Step two may specifically include: obtaining correction parameters based on the downsampled multispectral image signal. Obtaining grid coefficients for calculating partitions based on the multispectral image signal may specifically include: obtaining grid coefficients based on the downsampled multispectral image signal. Using the downsampled multispectral image signal to generate correction parameters and grid coefficients can effectively reduce the computational load, enabling real-time video-level image processing. In practical applications, due to the different spatial positions of the multispectral sensor and the three-channel sensor, there may be deviations between the multispectral image signal and the three-channel image signal. In some embodiments, the image correction method may further include: aligning the grid coefficients and the three-channel image signal in position before sampling the three-channel image signal according to the grid coefficients to obtain the light source partition map. In specific implementations, the grid coefficients and the three-channel image signal can be aligned in position according to the spatial relationship between the multispectral sensor and the three-channel sensor. Specifically, the spatial relationship between the multispectral sensor and the three-channel sensor can be aligned through calibration or optical flow. By utilizing the relative spatial relationship between the two, operations such as cropping, warping, and resizing can be performed on the grid coefficients to obtain the grid coefficients that align the three-channel image signals.

[0027] In some other embodiments of this application, after step one above, the image correction method may further include: downsampling the multispectral image signal and downsampling the three-channel image signal, thereby downsampling the original resolution multispectral image signal and the three-channel image signal into low-resolution image signals to reduce the computational load of subsequent processing. Step two above may specifically include: obtaining correction parameters based on the downsampled multispectral image signal and the downsampled three-channel image signal. Obtaining the grid coefficients for calculating the partition based on the multispectral image signal may specifically include: obtaining the grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal. In this way, the computational load of calculating the correction parameters and grid coefficients can be effectively reduced to achieve real-time video-level image processing. In addition, since the grid coefficients are generated based on the multispectral image signal and the three-channel image signal, the spatial accuracy of subsequent operations such as calculating the light source partition map and color correction can be higher. Furthermore, there is no need to align the grid coefficients with the three-channel image signal, which can save computational steps and improve computational speed.

[0028] Furthermore, after step one and before step three, the image correction method may further include obtaining a guide map based on the three-channel image signal. The sampling of the three-channel image signal based on the grid coefficient to obtain the light source partition map may specifically include: sampling the three-channel image signal based on the grid coefficient and the guide map to obtain the light source partition map. The guide map is a reference image associated with the three-channel image signal and can be used to guide the direction during the generation of the light source partition map. In this embodiment, the guide map guides the generation process of the light source partition map, optimizing the processing at the boundaries of different light source partitions. This avoids edge false color problems such as blurring and color deviation at the boundaries of different light source partitions, thereby making the corrected image closer to the real image seen by the human eye. In specific implementation, the guide map can be obtained based on the three-channel image signal at the original resolution, thus enriching the image information in the guide map and providing better guidance for the generation process of the light source partition map. Furthermore, if the guide map is calculated using multispectral image signals, the generated light source partition map needs to be aligned with the three-channel image signal. To avoid edge false color issues, high alignment accuracy is required, making the calculation quite difficult. In this embodiment, the guide map is calculated using three-channel image signals, and the generated light source partition map does not need to be aligned with the three-channel image signal, which can reduce the amount of computation and speed up the image processing process. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of the image correction device provided in the embodiments of this application;

[0031] Figure 3 This is another schematic diagram of the image correction device provided in the embodiments of this application;

[0032] Figure 4 This is another schematic diagram of the image correction device provided in the embodiments of this application;

[0033] Figure 5 This is another schematic diagram of the image correction device provided in the embodiments of this application;

[0034] Figure 6 This is another schematic diagram of the image correction device provided in the embodiments of this application;

[0035] Figure 7 This is another schematic diagram of the image correction device provided in the embodiments of this application;

[0036] Figure 8 A flowchart of the image processing method provided in the embodiments of this application;

[0037] Figure 9 This is a schematic flowchart of the image correction method in Embodiment 1 of this application;

[0038] Figure 10 This is a schematic diagram illustrating the preprocessing process of multispectral image signals and three-channel image signals in Embodiment 1 of this application;

[0039] Figure 11 This is a schematic diagram illustrating the process of calculating the multi-source estimation results and grid coefficients in Embodiment 1 of this application;

[0040] Figure 12 This is a schematic diagram illustrating the process of aligning the grid coefficients with the three-channel image signal in Embodiment 1 of this application;

[0041] Figure 13 This is a schematic diagram illustrating the process of generating a light source partition map in Embodiment 1 of this application;

[0042] Figure 14 This is a schematic diagram illustrating the process of correcting three-channel image signals in Embodiment 1 of this application;

[0043] Figure 15 This is a schematic flowchart of the image correction method in Embodiment 2 of this application;

[0044] Figure 16 This is a schematic diagram illustrating the preprocessing process of multispectral image signals and three-channel image signals in Embodiment 2 of this application;

[0045] Figure 17This is a schematic diagram illustrating the process of calculating the multi-source estimation results and grid coefficients in Embodiment 2 of this application.

[0046] Figure label:

[0047] 100 - Electronic device; 10 - Image correction device; 11 - Housing; 12 - Three-channel sensor; 13 - Multispectral sensor; 101 - First input interface; 102 - Second input interface; 103 - Light source estimation module; 104 - Color correction module; 105 - Downsampling module; 200 - Processor. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will now be described in further detail with reference to the accompanying drawings.

[0049] It should be noted that the accompanying drawings in this application are for illustrative purposes only and do not represent actual scale. The same reference numerals in the accompanying drawings denote the same or similar structures, and therefore, repeated descriptions of them will be omitted.

[0050] The terms describing position and direction used in this application, such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," are merely illustrative examples based on the orientation or positional relationships shown in the accompanying drawings. They are intended solely for the convenience of describing this application and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Changes may be made as needed, and all such changes are included within the scope of protection of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0051] To address the problem of inaccurate color correction in multi-light source scenarios, this application provides an image correction method, apparatus, and electronic device. The technical solution provided in this application can be applied to various electronic devices with shooting functions, such as mobile phones, tablets, digital cameras, webcams, and intelligent robots. The primary application scenario of this application is when the shooting scene includes multiple light sources, enabling accurate color correction of multiple light sources in the image. Of course, in some cases, the technical solution provided in this application can also be applied to scenarios with a single-color light source, and this is not a limitation.

[0052] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 1As shown, the electronic device 100 provided in this application embodiment may include an image correction device 10, which may be located within the housing 11 of the electronic device 100. The image correction method in this application embodiment can be executed in the image correction device 10, which may be an image processing chip, an image processor, or other devices with processing functions. The electronic device 100 in this application embodiment may also include a three-channel sensor 12 and a multispectral sensor 13. The three-channel sensor 12 may be a photosensitive sensor of at least one of a main camera, a telephoto camera, and a wide-angle camera. In one possible implementation, the multispectral sensor 13 may have multiple photosensitive units arranged in an array, enabling the multispectral sensor 13 to collect more information about the light source spectrum and provide spectral information close to the spectral response curve of the human eye. Figure 1 The example shown uses a mobile phone as an example of electronic device 100. When electronic device 100 is another device, please refer to the following: Figure 1 Implementation.

[0053] Figure 2 This is a schematic diagram of the structure of the image correction device provided in the embodiments of this application, such as... Figure 2 As shown, the image correction device 10 provided in this embodiment may include: a first input interface 101, a second input interface 102, a light source estimation module 103, and a color correction module 104. The first input interface 101 can be used to receive multispectral image signals from the multispectral sensor 13, and the second input interface 102 can be used to receive three-channel image signals from the three-channel sensor 12. The light source estimation module 103 can be used to obtain correction parameters corresponding to each of the multiple light sources based on the multispectral image signals. The color correction module 104 can be used to correct the three-channel image signals according to the correction parameters.

[0054] In the image correction device 10 provided in this application embodiment, the light source estimation module 103 can estimate multiple light sources in the scene based on the multispectral image signal to obtain the correction parameters corresponding to each light source. The color correction matrix 104 can correct the three-channel image signal according to the correction parameters to obtain an image. Compared with the three-channel sensor 12, the multispectral sensor 13 has more spectral channels and can collect more spectral information from more light sources. Therefore, the technical solution provided in this application embodiment can identify multiple light sources in the scene using multispectral image signals, achieve more accurate multi-light source estimation and image correction, and effectively improve the accuracy of color correction in multi-light source scenes, so as to more accurately restore the colors in the photographed scene.

[0055] Figure 3 This is another schematic diagram of the image correction device provided in the embodiments of this application, as shown below. Figure 3As shown, the image correction device 10 provided in this embodiment may further include a processor 200. The functions of the light source estimation module 103 and the color correction module 104 can be implemented by the integrated logic circuit in the hardware and / or the instructions in the software form of the processor 200. In specific implementations, the processor 200 can be a chip, such as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), system-on-chip (SoC), central processing unit (CPU), network processor (NP), digital signal processor (DSP), microcontroller unit (MCU), programmable logic device (PLD), or any other chip with processing capabilities. Of course, the processor 200 can also be any other device with data processing capabilities, which will not be listed here.

[0056] In embodiments of this application, the light source estimation module 103 can perform multi-light source estimation on the scene based on the multispectral image signal, determine the light source information of each light source in the scene, and determine the correction parameters corresponding to each light source based on the light source information of each light source in the scene. For example, the light source information of each light source in the scene may include the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. In practical applications, the light sources in the shooting scene can be divided into indoor light sources, outdoor light sources, and display light sources (such as light sources formed by computer and television screens). Since the number of light sources in most shooting scenes will not exceed three, the number of light sources obtained by the light source estimation module 103 through multi-light source estimation can be less than or equal to three. This reduces the complexity of hardware and algorithms and improves the computing speed of the image correction device 10 while meeting the image correction requirements of most scenes. In one possible implementation, the correction parameters may include at least one of a white balance gain value or a color correction matrix. For example, when there are two light sources in a scene, an indoor light source and an outdoor light source, by performing multi-light source estimation on the multispectral image signal, the white balance gain value and color correction matrix under the indoor light source, as well as the white balance gain value and color correction matrix under the outdoor light source, can be calculated.

[0057] Figure 4 This is another schematic diagram of the image correction device provided in the embodiments of this application, as shown below. Figure 4As shown, in some embodiments of this application, the light source estimation module 103 can also be used to obtain grid coefficients for calculating partitions based on the multispectral image signal. Since the multispectral image signal has many spectral channels, it can distinguish image regions corresponding to different light sources. Therefore, the grid coefficients obtained from the multispectral image signal can represent light source partitioning information. For example, the grid coefficients can be bilateral grid coefficients. The color correction module 104 can specifically be used to sample the three-channel image signal based on the grid coefficients to obtain a light source partitioning map. For example, slicing can be used to process the three-channel image signal. The light source partitioning map is essentially a weight map, used to represent the weight of each light source at each pixel. Compared to the three-channel image signal, the multispectral image signal has a stronger ability to identify multi-light source scenes. Therefore, by sampling the light source partitioning information from the multispectral image signal onto the three-channel image signal, the image can be accurately partitioned according to different light sources.

[0058] Furthermore, the color correction module 104 can also be used to correct the three-channel image signal using correction parameters and a light source zoning map. In specific implementation, the weight of each light source at each pixel can be determined according to the light source zoning map. Based on the weight of each light source at each pixel and the corresponding correction parameters, color correction can be performed on each pixel. For example, taking a scene with both indoor and outdoor light sources as an example, when correcting the color of a certain pixel, the color correction can be performed on that pixel based on the weight of the indoor light source at that pixel and its corresponding correction parameters, and the weight of the outdoor light source at that pixel and its corresponding correction parameters.

[0059] In some embodiments of this application, such as Figure 5 As shown, Figure 5This is another schematic diagram of the image correction device provided in this application embodiment. The image correction device 10 may further include a downsampling module 105, which can be used to downsample the multispectral image signal, thereby downsampling the original resolution multispectral image signal into a low resolution image signal to reduce the computational load of subsequent processing. The downsampling module 105 can also provide the downsampled multispectral image signal to the light source estimation module 103, which can be specifically used to obtain correction parameters and grid coefficients based on the downsampled multispectral image signal. By using the downsampled multispectral image signal to generate correction parameters and grid coefficients, the computational load of the light source estimation module 103 can be effectively reduced to achieve real-time video-level image processing. In practical applications, due to the different spatial positions of the multispectral sensor and the three-channel sensor, there may be a deviation between the multispectral image signal and the three-channel image signal. In some embodiments, the color correction module 104 may also be used to: align the grid coefficients and the three-channel image signal in position before sampling the three-channel image signal according to the grid coefficients to obtain the light source partition map. In practical implementation, the grid coefficients and the three-channel image signals can be aligned based on the spatial relationship between the multispectral sensor and the three-channel sensor. Specifically, the spatial relationship between the multispectral sensor and the three-channel sensor can be aligned through calibration or optical flow. Utilizing their relative spatial positions, operations such as cropping, warping, and resizing can be performed on the grid coefficients to obtain the aligned grid coefficients for the three-channel image signals. In this embodiment, the computational power required for aligning the grid coefficients is relatively low, enabling real-time multi-source image processing on video.

[0060] In other embodiments of this application, such as Figure 6 As shown, Figure 6This is another schematic diagram of the image correction device provided in this application embodiment. The image correction device 10 may further include: a downsampling module 105, which can be used to downsample the multispectral image signal and the three-channel image signal, thereby downsampling the original resolution multispectral image signal and the three-channel image signal into low-resolution image signals to reduce the computational load of subsequent processing. The downsampling module 105 can also provide the downsampled multispectral image signal and the downsampled three-channel image signal to the light source estimation module 103. The light source estimation module 103 can specifically be used to obtain correction parameters and grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal. By using the downsampled multispectral image signal and the three-channel image signal to generate correction parameters and grid coefficients, the computational load of the light source estimation module 103 can be effectively reduced to achieve real-time video-level image processing. In addition, since the grid coefficients are generated based on the multispectral image signal and the three-channel image signal, the spatial accuracy of subsequent operations such as calculating the light source partition map and color correction is higher. Moreover, compared to Figure 5 The embodiment shown, Figure 6 In the illustrated embodiment, there is no need to align the grid coefficients with the three-channel image signals, which can save the calculation steps of the light source estimation module 103 and improve the calculation speed of the light source estimation module 103.

[0061] Furthermore, such as Figure 7 As shown, Figure 7This is another schematic diagram of the image correction device provided in this application embodiment. The light source estimation module 103 in this application embodiment can also be used to obtain a guide map based on the three-channel image signal. The color correction module 104 can specifically be used to sample the three-channel image signal to obtain a light source partition map based on the grid coefficients and the guide map. The guide map is a reference image associated with the three-channel image signal and can be used to guide the direction in the light source partition map generation process. In this application embodiment, the guide map is used to guide the generation process of the light source partition map and optimize the processing at the boundaries of different light source partitions. This avoids edge false color problems such as blurring and color deviation at the boundaries of different light source partitions, thereby making the image corrected by the color correction module 104 closer to the real image seen by the human eye. In specific implementation, the light source estimation module 103 can obtain a guide map based on the three-channel image signal at the original resolution, thus enriching the image information in the guide map and providing better guidance for the generation process of the light source partition map. Furthermore, if the guide map is calculated using multispectral image signals, the generated light source partition map needs to be aligned with the three-channel image signals. To avoid edge false color problems, high alignment accuracy is required, making the calculation quite difficult. In this embodiment, the guide map is calculated using three-channel image signals, and the generated light source partition map does not need to be aligned with the three-channel image signals, which can reduce the amount of computation and speed up the operation of the image processing device 10.

[0062] In practical implementation, the light source estimation module 103 can train the parameters of the grid coefficients and the parameters of the guide map. The labels in the training set can be point-by-point light source partition maps and multiple light source information. The labels in the training set can be used to perform a loss function with the light source partition maps.

[0063] In one possible implementation, the aforementioned light source estimation module 103 may include a neural processing unit (NPU) and a data processing unit. For example, the neural processing unit may be a convolutional neural network (CNN). The neural processing unit can be used to perform multi-light source estimation, calculate grid coefficients, and calculate guidance maps based on multispectral image signals. The data processing unit can be used to calculate correction parameters corresponding to each light source based on the light source information of each light source in the scene.

[0064] In specific implementation, the color correction module 104 can be implemented by an image signal processor (ISP). The operations of the color correction module 104, such as calculating the light source partition map and correcting the three-channel image signals according to the correction parameters, can be hard-coded into the image signal processor, which can reduce the real-time data processing volume of the image correction device 10, thereby realizing video-level multi-light source image processing.

[0065] Based on the same technical concept, this application also provides an image correction method. The image correction method provided in this application can be executed in the above-mentioned image correction device, for example, it can be executed in the processor of the above-mentioned image correction device.

[0066] Figure 8 A flowchart of the image correction method provided in the embodiments of this application is shown below. Figure 8 As shown, the image correction method provided in this application embodiment may include:

[0067] S201, Receive multispectral image signals from a multispectral sensor and receive three-channel image signals from a three-channel sensor;

[0068] S202. Obtain the correction parameters corresponding to each of the multiple light sources based on the multispectral image signal;

[0069] S203. Correct the three-channel image signal according to the correction parameters.

[0070] In the image processing method provided in this application embodiment, multi-light source estimation of the scene is performed based on multispectral image signals to obtain correction parameters corresponding to each of the multiple light sources. The three-channel image signal is then corrected according to these correction parameters to obtain an image. Compared to a three-channel sensor, a multispectral sensor has more spectral channels, allowing it to collect spectral information from more light sources. Therefore, the technical solution provided in this application embodiment utilizes multispectral image signals to identify multiple light sources in a scene, achieving more accurate multi-light source estimation and image correction. This effectively improves the accuracy of color correction in multi-light source scenes, enabling more accurate color reproduction of the captured scene.

[0071] In embodiments of this application, step S202 may specifically include: performing multi-light source estimation on the scene based on the multispectral image signal, determining the light source information of each light source in the scene, and determining the correction parameters corresponding to each light source based on the light source information of each light source in the scene. For example, the light source information of each light source in the scene may include the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. In practical applications, the light sources in the shooting scene can be divided into indoor light sources, outdoor light sources, and display light sources (such as light sources formed by computer and television screens). Since the number of light sources in most shooting scenes will not exceed three, the number of light sources obtained by multi-light source estimation in step S202 can be less than or equal to three. This reduces the complexity of hardware and algorithms and improves the computing speed of the image correction device while meeting the image correction requirements of most scenes. In one possible implementation, the correction parameters may include at least one of a white balance gain value or a color correction matrix. For example, when there are two light sources in a scene, an indoor light source and an outdoor light source, by performing multi-light source estimation on the multispectral image signal, the white balance gain value and color correction matrix under the indoor light source, as well as the white balance gain value and color correction matrix under the outdoor light source, can be calculated.

[0072] In some embodiments of this application, after step S201 and before step S203, the image correction method may further include: obtaining grid coefficients for calculating partitions based on the multispectral image signal. Since the multispectral image signal contains many spectral channels, it can distinguish image regions corresponding to different light sources. Therefore, the grid coefficients obtained from the multispectral image signal can represent light source partitioning information. For example, the grid coefficients can be bilateral grid coefficients.

[0073] Step S203 can specifically include: sampling the three-channel image signal according to the grid coefficient to obtain a light source partition map, and correcting the three-channel image signal using correction parameters and the light source partition map. For example, slicing can be used to process the three-channel image signal to obtain the light source partition map. The light source partition map is essentially a weight map, used to represent the weight of each light source at each pixel. Multispectral image signals have a stronger ability to identify multi-light source scenes compared to three-channel image signals. Therefore, sampling the light source partition information from the multispectral image signal onto the three-channel image signal allows for accurate partitioning of the image according to different light sources. In specific implementation, the weight of each light source at each pixel can be determined based on the light source partition map. Based on the weight of each light source at each pixel and the corresponding correction parameters, color correction can be performed on each pixel. For example, if there are two light sources in the scene, an indoor light source and an outdoor light source, when correcting the color of a certain pixel, the color of the pixel can be corrected according to the weight of the indoor light source at that pixel and the corresponding correction parameters of the indoor light source, the weight of the outdoor light source at that pixel and the corresponding correction parameters of the outdoor light source.

[0074] In some embodiments of this application, after step S201, the image correction method may further include: downsampling the multispectral image signal, thereby downsampling the original resolution multispectral image signal into a low resolution image signal to reduce the computational load of subsequent processing. Step S202 may specifically include: obtaining correction parameters based on the downsampled multispectral image signal; obtaining the grid coefficients for calculating the partitions based on the multispectral image signal may specifically include: obtaining the grid coefficients based on the downsampled multispectral image signal. Using the downsampled multispectral image signal to generate correction parameters and grid coefficients can effectively reduce the computational load, enabling real-time video-level image processing. In practical applications, due to the different spatial positions of the multispectral sensor and the three-channel sensor, there may be deviations between the multispectral image signal and the three-channel image signal. In some embodiments, the image correction method may further include: aligning the grid coefficients and the three-channel image signal in position before sampling the three-channel image signal according to the grid coefficients to obtain the light source partition map. In specific implementations, the grid coefficients and the three-channel image signal can be aligned in position according to the spatial relationship between the multispectral sensor and the three-channel sensor. Specifically, the spatial relationship between the multispectral sensor and the three-channel sensor can be aligned through calibration or optical flow. By utilizing the relative spatial relationship between the two, operations such as cropping, warping, and resizing can be performed on the grid coefficients to obtain the grid coefficients that align the three-channel image signals.

[0075] In some other embodiments of this application, after step S201, the image correction method may further include: downsampling the multispectral image signal and downsampling the three-channel image signal, thereby downsampling the original resolution multispectral image signal and the three-channel image signal into low-resolution image signals to reduce the computational load of subsequent processing. Step S202 may specifically include: obtaining correction parameters based on the downsampled multispectral image signal and the downsampled three-channel image signal. Obtaining the grid coefficients for calculating the partitions based on the multispectral image signal may specifically include: obtaining the grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal. This effectively reduces the computational load of calculating the correction parameters and grid coefficients, enabling real-time video-level image processing. Furthermore, since the grid coefficients are generated based on the multispectral image signal and the three-channel image signal, the spatial accuracy of subsequent operations such as calculating the light source partition map and color correction is higher. Moreover, there is no need to align the grid coefficients with the three-channel image signal, saving computational steps and improving computational speed.

[0076] Furthermore, after step S201 and before step S203, the image correction method may further include obtaining a guide map based on the three-channel image signal. The sampling of the three-channel image signal based on the grid coefficient to obtain the light source partition map may specifically include: sampling the three-channel image signal based on the grid coefficient and the guide map to obtain the light source partition map. The guide map is a reference image associated with the three-channel image signal and can be used to guide the direction during the generation of the light source partition map. In this embodiment, the guide map guides the generation process of the light source partition map, optimizing the processing at the boundaries of different light source partitions. This avoids edge false color problems such as blurring and color deviation at the boundaries of different light source partitions, thereby making the corrected image closer to the real image seen by the human eye. In specific implementation, the guide map can be obtained based on the three-channel image signal at the original resolution, thus enriching the image information in the guide map and providing better guidance for the generation process of the light source partition map. Furthermore, if the guide map is calculated using multispectral image signals, the generated light source partition map needs to be aligned with the three-channel image signal. To avoid edge false color issues, high alignment accuracy is required, making the calculation quite difficult. In this embodiment, the guide map is calculated using three-channel image signals, and the generated light source partition map does not need to be aligned with the three-channel image signal, which can reduce the amount of computation and speed up the image processing process.

[0077] The inventive concept of this application has been described in detail above. The technical solutions of the embodiments of this application will be described in detail below with reference to specific examples.

[0078] Example 1:

[0079] Figure 9 This is a flowchart illustrating the image correction method in Embodiment 1 of this application, as shown below. Figure 9 As shown, the image correction method in Embodiment 1 of this application may specifically include:

[0080] S301, Receive multispectral image signals from a multispectral sensor and receive three-channel image signals from a three-channel sensor.

[0081] Figure 10 This is a schematic diagram illustrating the preprocessing of multispectral image signals and three-channel image signals in Embodiment 1 of this application, as shown below. Figure 10 As shown, preprocessing operations such as black level correction and lens shading correction can be performed on the received multispectral image signal, as well as downsampling the multispectral image signal to reduce the computational load of subsequent processing. Preprocessing operations such as black level correction and lens shading correction can be performed on the received three-channel image signal to obtain the original resolution three-channel image signal.

[0082] S302. Obtain multi-source estimation results and grid coefficients for calculating partitions based on multispectral image signals.

[0083] Figure 11 This is a schematic diagram illustrating the process of calculating the multi-source estimation results and grid coefficients in Embodiment 1 of this application, as shown below. Figure 11As shown, the multispectral image signal can be processed by a neural network processing unit to obtain the multi-source estimation results and grid coefficients of the scene. For example, the grid coefficients can be bilateral grid coefficients. Step S302 can be based on the downsampled multispectral image signal, thereby effectively reducing computational load, improving image processing speed, and achieving real-time video-level image processing. The aforementioned multi-source estimation results can include the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. Since the number of light sources in most shooting scenes does not exceed three, the number of light sources in the multi-source estimation results is usually less than or equal to three. This can reduce the complexity of hardware and algorithms and improve the speed of image correction while meeting the image correction requirements of most scenes. Furthermore, multispectral image signals have stronger recognition capabilities than three-channel image signals. Calculating the multi-source estimation results and grid coefficients using multispectral image signals can make the correction parameters and light source partitioning maps obtained in subsequent steps more accurate.

[0084] S303. Based on the multi-source estimation results, obtain the correction parameters corresponding to each source.

[0085] In one possible implementation, the aforementioned correction parameters may include at least one of a white balance gain value or a color correction matrix. For example, when there are two light sources in the scene, an indoor light source and an outdoor light source, the white balance gain value and color correction matrix under the indoor light source, as well as the white balance gain value and color correction matrix under the outdoor light source, can be calculated by performing multi-light source estimation on the multispectral image signal.

[0086] S304. Align the grid coefficients and the three-channel image signals.

[0087] Figure 12 This is a schematic diagram illustrating the process of aligning the grid coefficients with the three-channel image signal in Embodiment 1 of this application. Figure 12 (1) is a schematic diagram of the spatial location of the multispectral image signal and the three-channel image signal, as shown in Figure 1. Figure 12 As shown in (1), since the spatial positions of the multi-channel sensor and the three-channel sensor are different, there may be a positional deviation between the spatial positions of the multi-channel image signal and the three-channel image signal. Furthermore, since the grid coefficient is calculated based on the multispectral image signal, there may also be a positional deviation between the grid coefficient and the three-channel image signal. Figure 12 (2) is a schematic diagram of aligning the grid coefficients with the three-channel image signal, as shown in Figure 2. Figure 12As shown in (2) above, in practical implementation, the grid coefficients and the three-channel image signals can be aligned according to the spatial positional relationship between the multispectral sensor and the three-channel sensor. Specifically, the spatial positional relationship between the multispectral sensor and the three-channel sensor can be aligned by calibration or optical flow. Using the relative spatial positional relationship between the two, operations such as cropping, warping, and resizing can be performed on the grid coefficients to obtain the grid coefficients aligned with the three-channel image signals.

[0088] S305. Obtain the guide map based on the three-channel image signal.

[0089] Figure 13 This is a schematic diagram illustrating the process of generating a light source partition map in Embodiment 1 of this application, as shown below. Figure 13 As shown, a three-channel image signal can be provided to the neural network processing unit, which then processes the signal to obtain a guide map. The guide map is a reference image associated with the three-channel image signal and can be used to guide the direction during the generation of the light source partition map. In specific implementations, the guide map can be obtained based on the original resolution three-channel image signal, thus enriching the image information in the guide map and providing better guidance for the subsequent generation of the light source partition map. Furthermore, if the guide map is calculated using multispectral image signals, the generated light source partition map needs to be aligned to the three-channel image signal. To avoid edge false color problems, high alignment accuracy is required, making the calculation computationally difficult. In this embodiment, the guide map is calculated using the three-channel image signal, eliminating the need for alignment between the generated light source partition map and the three-channel image signal, reducing computational load and accelerating image processing.

[0090] S306. Based on the grid coefficients and the guide map, sample the three-channel image signals to obtain the light source partition map.

[0091] A light source partitioning map is essentially a weighted map used to represent the weight of each light source at each pixel. Multispectral image signals have a stronger ability to identify multi-light source scenes compared to three-channel image signals. Therefore, by sampling the light source partitioning information from the multispectral image signal onto the three-channel image signal, the image can be accurately partitioned according to different light sources. For example, a slicing method can be used to process the three-channel image signal to obtain the light source partitioning map.

[0092] Continue to refer to Figure 13This method generates a light source partition map based on the grid coefficients of aligned three-channel image signals, enabling accurate image partitioning according to the light sources in the scene. Furthermore, a guiding map guides the generation process of the light source partition map, optimizing the processing at the boundaries between different light source partitions. This avoids edge artifacts such as blurring and color deviation at these boundaries, resulting in an image that is closer to the real image as seen by the human eye. In practice, the parameters of the grid coefficients and the guiding map can be trained. The labels in the training set can be point-by-point light source partition maps and information from multiple light sources. These labels can be used to calculate the loss function between the training set and the light source partition map.

[0093] S307. Correct the three-channel image signal using correction parameters and a light source zoning diagram. The following example illustrates the image correction process using correction parameters including white balance gain and color correction matrix.

[0094] A light source partition map is essentially a weighted map used to represent the weight of each light source at each pixel in the image. The size of the light source partition map can be represented by H*W*3, where H and W are the height and width of the three-channel image signal, respectively.

[0095] Using the light source zoning map and the white balance gain values ​​corresponding to each light source, the white balance gain value at each pixel is calculated to obtain the white balance gain map. The white balance gain value at each pixel can be calculated using the following formula:

[0096] AWB_Gain (h,w,c)

[0097] =AWB0_Gain (c) *Mask0 (h,w) +AWB1_Gain (c) *Mask1 (h,w) +AWB2_Gain (c)

[0098] *Mask2 (h,w)

[0099] h = 0, 1, 2, ..., H

[0100] w = 0, 1, 2, ..., W

[0101] c = 0, 1, 2

[0102] Among them, AWB_Gain (h,w,c) This represents the white balance gain value at each pixel, where h and w represent the coordinates, and c represents the channel of the three-channel image signal, typically the RGB three channels. AWB0_Gain (c) AWB1_Gain(c) AWB2_Gain (c) These represent the white balance gain values ​​corresponding to different light sources, and Mask represents the weight of different light sources in the light source partitioning map. When the scene includes a single light source, Mask1... (h,w) and Mask2 (h,w) The value can be 0. When the scene includes two light sources, Mask2... (h,w) The value can be 0, and the sum of the three channels of Mask is 1, i.e., Mask0. (h,w) +Mask1 (h,w) +Mask2 (h,w) =1.

[0103] Using the light source partitioning map and the color correction matrix corresponding to each light source, the color correction matrix at each pixel is calculated to obtain the color correction matrix map. The color correction matrix at each pixel can be calculated using the following formula:

[0104] CCM (h,w,b) =CCM0 (b) *Mask0 (h,w) +CCM1 (b) *Mask1 (h,w) +CCM2 (b) *Mask2 (h,w)

[0105] h = 0, 1, 2, ..., H

[0106] w = 0, 1, 2, ..., W

[0107] b = 0, 1, 2, ..., 9

[0108] Among them, CCM (h,w,b) This represents the color correction matrix at each pixel, where h and w represent the coordinate positions and b represents the index of the color correction matrix. Typically, the color correction matrix is ​​a 3x3 matrix, and CCM0, CCM1, and CCM2 represent the color correction matrices corresponding to different light sources.

[0109] Figure 14 This is a schematic diagram illustrating the process of correcting three-channel image signals in Embodiment 1 of this application, as shown below. Figure 14 As shown, the obtained white balance gain map and color correction matrix map are applied to the three-channel image signals respectively to obtain the corrected image map. Figure 14 The three-channel image signal in the above step S301 can be the original resolution three-channel image signal obtained after preprocessing. By correcting the original resolution three-channel image signal, the resolution of the corrected image can be increased.

[0110] Example 2:

[0111] Figure 15 This is a schematic flowchart of the image correction method in Embodiment 2 of this application, as shown below. Figure 15 As shown, the image correction method in Embodiment 2 of this application may specifically include:

[0112] S401, Receive multispectral image signals from a multispectral sensor and receive three-channel image signals from a three-channel sensor.

[0113] Figure 16 This is a schematic diagram illustrating the preprocessing of multispectral image signals and three-channel image signals in Embodiment 2 of this application, as shown below. Figure 16 As shown, preprocessing operations such as black level correction and lens shading correction can be performed on the received multispectral image signal, and downsampling can be performed on the multispectral image signal to reduce the computational load of subsequent processing.

[0114] The received three-channel image signal can undergo preprocessing operations such as black level correction and lens shading correction to obtain the original resolution three-channel image signal. Furthermore, the preprocessed three-channel image signal is downsampled, reducing the original resolution three-channel image signal to a lower resolution three-channel image signal. In other words, by processing the received three-channel image signal, two three-channel image signals with different resolutions can be obtained. The original resolution three-channel image signal can be used for subsequent calculation of the guide map, while the lower resolution three-channel image signal can be used for subsequent calculation of the grid coefficients.

[0115] S402. Based on the multispectral image signal and the three-channel image signal, obtain the multi-source estimation results and the grid coefficients used to calculate the partition.

[0116] Figure 17 This is a schematic diagram illustrating the process of calculating the multi-source estimation results and grid coefficients in Embodiment 2 of this application, as shown below. Figure 17As shown, the multispectral image signal and the three-channel image signal can be processed by the neural network processing unit to obtain the multi-source estimation results and grid coefficients of the scene. For example, the grid coefficients can be bilateral grid coefficients. In a specific implementation, in step S402, the downsampled multispectral image signal and the downsampled three-channel image signal can be provided to the neural network processing unit. Processing based on the low-resolution multispectral image signal and the low-resolution three-channel image signal can effectively reduce the computational load and achieve real-time video-level image processing. Furthermore, since the grid coefficients are generated based on the multispectral image signal and the three-channel image signal, the spatial accuracy of subsequent operations such as calculating the light source partition map and color correction is higher. Moreover, there is no need to align the grid coefficients with the three-channel image signal, saving computational steps and improving computational speed.

[0117] The multi-light source estimation results mentioned above can include light source information such as the number of light sources in the scene, the color temperature information of each light source, and the spectral information of each light source. Since the number of light sources in most shooting scenes will not exceed three, the number of light sources in the multi-light source estimation results is usually less than or equal to three. This can reduce the complexity of hardware and algorithms and improve the processing speed of image correction while meeting the image correction requirements of most scenes.

[0118] S403. Based on the multi-source estimation results, obtain the correction parameters corresponding to each source.

[0119] S404. Obtain the guide map based on the three-channel image signal.

[0120] S405. Based on the grid coefficients and the guide map, sample the three-channel image signals to obtain the light source partition map.

[0121] S406. Correct the three-channel image signal using correction parameters and light source partitioning diagram.

[0122] The difference between Embodiment 2 of this application and Embodiment 1 above is that: in step S401, both the multispectral image signal and the three-channel image signal are downsampled; in step S402, based on the multispectral image signal and the three-channel image signal, the multi-source estimation result and the grid coefficient for calculating the partition are obtained; there is no need to align the grid coefficient with the three-channel image signal.

[0123] Therefore, compared to Example 1, Example 2 calculates the multi-source estimation results and grid coefficients based on the multispectral image signal and the three-channel image signal. Example 2 exhibits higher spatial accuracy, resulting in better image correction. Furthermore, Example 2 eliminates the need for positional alignment between the grid coefficients and the three-channel image signal, saving computational steps and improving computational speed.

[0124] The technical solutions of the embodiments of this application have been described in detail above with reference to Embodiment 1 and Embodiment 2. In specific implementation, corresponding modifications can be made based on the above embodiments, which will not be illustrated here.

[0125] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0126] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. An image correction device, characterized in that, include: The first input interface is used to receive multispectral image signals from a multispectral sensor; The second input interface is used to receive three-channel image signals from a three-channel sensor; The light source estimation module is used to obtain the correction parameters corresponding to each of the multiple light sources based on the multispectral image signal. The color correction module is used to correct the three-channel image signal according to the correction parameters.

2. The image correction device as described in claim 1, characterized in that, The light source estimation module is also used to obtain grid coefficients for calculating partitions based on the multispectral image signal; The color correction module is specifically used to sample the three-channel image signal according to the grid coefficient to obtain a light source partition map, and to correct the three-channel image signal using the correction parameters and the light source partition map.

3. The image correction device as described in claim 2, characterized in that, Also includes: A downsampling module is used to downsample the multispectral image signal; The light source estimation module is specifically used to obtain the correction parameters and the grid coefficients based on the downsampled multispectral image signal. The color correction module is further configured to: align the grid coefficients and the three-channel image signals in position before sampling the three-channel image signals according to the grid coefficients to obtain the light source partition map.

4. The image correction device as described in claim 2, characterized in that, Also includes: The downsampling module is used to downsample the multispectral image signal and the three-channel image signal; The light source estimation module is specifically used to: obtain the correction parameters and the grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal.

5. The image correction apparatus according to any one of claims 2 to 4, characterized in that, The light source estimation module is also used to obtain a guide map based on the three-channel image signal; The color correction module is specifically used to sample the three-channel image signal according to the grid coefficient and the guide map to obtain the light source partition map.

6. The image correction apparatus according to any one of claims 1 to 5, characterized in that, The correction parameters include at least one of the white balance gain value or the color correction matrix.

7. An electronic device, characterized in that, include: The image correction apparatus as described in any one of claims 1 to 6.

8. An image correction method, characterized in that, include: Receive multispectral image signals from a multispectral sensor and three-channel image signals from a three-channel sensor; Based on the multispectral image signal, the correction parameters corresponding to each of the multiple light sources are obtained; The three-channel image signal is corrected according to the correction parameters.

9. The image correction method as described in claim 8, characterized in that, After receiving the multispectral image signal from the multispectral sensor, the method further includes: obtaining grid coefficients for calculating the partition based on the multispectral image signal; The step of correcting the three-channel image signal according to the correction parameters specifically includes: sampling the three-channel image signal according to the grid coefficient to obtain a light source partition map, and using the correction parameters and the light source partition map to correct the three-channel image signal.

10. The image correction method as described in claim 9, characterized in that, After receiving the multispectral image signal from the multispectral sensor, the image correction method further includes: downsampling the multispectral image signal; The step of obtaining the correction parameters corresponding to each of the multiple light sources based on the multispectral image signal specifically includes: obtaining the correction parameters based on the downsampled multispectral image signal; The step of obtaining the grid coefficients for calculating the partitions based on the multispectral image signal specifically includes: obtaining the grid coefficients based on the downsampled multispectral image signal; Before sampling the three-channel image signal according to the grid coefficient to obtain the light source partition map, the method further includes: aligning the grid coefficient and the three-channel image signal in position.

11. The image correction method as described in claim 9, characterized in that, After receiving the multispectral image signal from the multispectral sensor, the method further includes: downsampling the multispectral image signal; After receiving the three-channel image signal from the three-channel sensor, the method further includes: downsampling the three-channel image signal; The step of obtaining the correction parameters corresponding to each of the multiple light sources based on the multispectral image signal specifically includes: obtaining the correction parameters based on the downsampled multispectral image signal and the downsampled three-channel image signal; The step of obtaining the grid coefficients for calculating the partition based on the multispectral image signal specifically includes: obtaining the grid coefficients based on the downsampled multispectral image signal and the downsampled three-channel image signal.

12. The image correction method according to any one of claims 9 to 11, characterized in that, After receiving the three-channel image signal from the three-channel sensor, the method further includes: obtaining a guide map based on the three-channel image signal; The step of sampling the three-channel image signal according to the grid coefficient to obtain the light source partition map specifically includes: sampling the three-channel image signal according to the grid coefficient and the guide map to obtain the light source partition map.

13. The image correction method according to any one of claims 8 to 12, characterized in that, The correction parameters include at least one of the white balance gain value or the color correction matrix.

Images