A method and apparatus for processing image data

Abstract

The present application provides an image data processing method and apparatus. The method includes obtaining image data and multispectral data of a target scene; determining the proportion of K basic light source components based on the multispectral data, where the proportion of K basic light source components indicates the proportion of the spectrum of each basic light source among the K basic light sources in the scene reflection spectrum; reconstructing a calibrated image to process the image data of the target scene based on the proportion of K basic light source components and K calibrated images, and each of the K calibrated images is an image obtained by capturing the calibrated scene under each of the K basic light sources through an imaging module. The present application more precisely depicts the color changes brought about by scene changes to a specific imaging module, thereby achieving stable and continuous dynamic color modulation.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a method and apparatus for processing image data. Background Art

[0002] The authenticity of color reproduction is a key indicator for evaluating the performance of a photographic system. In a traditional camera imaging path, the spectrum (L) reflected by the scene is first converged by optical devices onto an image sensor. The image sensor responds to form a three-channel raw signal, and then through a series of color processing (such as lens shading correction, white balance, color correction, etc.), it can be restored into an image that conforms to the human eye response.

[0003] Since different camera modules have different responses to the spectrum, even in the same scene, the raw signals formed by them are significantly different. Therefore, in actual engineering, it is necessary to perform characteristic calibration on each camera module separately and adjust the color parameters accordingly. How to accurately characterize the module characteristics and generate dynamic color parameters according to the actual scene is the key to improving the authenticity of color reproduction. Summary of the Invention

[0004] Embodiments of this application provide a method and apparatus for processing image data, which can more precisely characterize the color changes brought about by scene changes to a specific camera module, thereby achieving stable and continuous dynamic color modulation.

[0005] In a first aspect, this application provides a method for processing image data. The method includes obtaining image data and multi-spectral data of a target scene; based on the multi-spectral data, determining the proportions of K base light source components, where the proportions of the K base light source components indicate the proportion of the spectrum of each base light source among the K base light sources in the scene reflection spectrum, and the scene reflection spectrum indicates the reflection spectrum distribution corresponding to the target shooting scene; based on the proportions of the K base light source components and the K calibrated images, processing the image data of the target scene to reconstruct the calibrated images, and each of the K calibrated images is an image obtained by shooting the calibrated scene under each of the K base light sources through a camera module to reconstruct the calibrated images.

[0006] The method for processing image data provided by this application processes the image data through multi-spectral data and the characteristic calibration data of the camera module (including K base light sources and K calibrated images), fully considering the influence of the response differences of the camera module on the image color processing parameters, achieving unified color effects for different camera modules. At the same time, the image processing method of this application does not have tasks such as discrete threshold screening and classification, and can more precisely characterize the color changes brought about by scene changes to a specific camera module, thereby achieving stable and continuous dynamic color modulation and significantly improving the accuracy of image color.

[0007] In a possible implementation, a specific implementation for determining the component ratios of K basic light sources based on multispectral data is as follows: Based on the multispectral data, determine the scene reflection spectrum; split the scene reflection spectrum based on the scene reflection spectrum and the spectra of each of the K basic light sources to obtain a splitting result, where the splitting result indicates the component ratios of the spectra of each basic light source in the scene reflection spectrum; based on the splitting result, determine the component ratios of the K basic light sources.

[0008] The multispectral signal of the real shooting scene is collected by a multispectral sensor, and then the multispectral signal is inversely solved to obtain the scene reflection spectrum L of the real shooting scene. Then, the component analysis of the scene reflection spectrum L is performed to analyze and obtain the component ratios of each basic light source in the scene reflection spectrum L. That is to say, the scene reflection spectrum L of the real shooting scene can be represented by the reflection spectra of multiple basic light sources, so as to facilitate the subsequent steps of using the component ratios of each basic light source and K calibrated scene images to determine the reconstructed calibrated image, that is, to reconstruct the calibrated image of the calibrated scene under the illumination of the light source in the current real shooting scene.

[0009] In this possible implementation, a specific implementation for determining the component ratios of K basic light sources based on multispectral data is as follows: Use the multispectral data as the input of the trained neural network model, and output the component ratios of the K basic light sources.

[0010] Through the trained neural network model, the multispectral data is mapped to the component ratios of each of the K basic light sources in the scene reflection spectrum L, implementing an end-to-end solution for determining the component ratios of the K basic light sources, and quickly obtaining the component ratio information of the K basic light sources.

[0011] In another possible implementation, a specific implementation for processing the image data of the target scene based on the component ratios of the K basic light sources and the K calibrated images is as follows: Based on the component ratios of the K basic light sources and the K calibrated images, determine the reconstructed calibrated image; based on the reconstructed calibrated image, process the image data of the target scene.

[0012] In another possible implementation, a specific implementation for determining the reconstructed calibrated image based on the component ratios of the K basic light sources and the K calibrated images is as follows: Use the component ratios of the K basic light sources as the weights of each of the K calibrated images in the K calibrated images respectively, and perform a weighted summation operation on the K calibrated images to obtain the reconstructed calibrated image.

[0013] In another possible implementation, a specific implementation for processing the image data of the target scene based on the reconstructed calibrated image is as follows: Based on the reconstructed calibrated image, determine the image color processing parameters; based on the image processing parameters, process the image data of the target scene.

[0014] In another possible implementation, the image color processing parameters include lens shading correction parameters; the calibration scene is a uniform illumination scene; a specific implementation for determining the image color processing parameters based on the reconstructed calibration image is: determining the lens shading correction parameters based on the color distribution in the reconstructed calibration image and the color distribution in the image data of the target scene.

[0015] This application uses multi-spectral signals to analyze the scene reflection spectrum L, dynamically reconstructs the calibration image according to the component ratios of each base light source in the scene reflection spectrum L, alleviates the metamerism problem, and at the same time, the adaptive parameter calculation improves the calibration accuracy and can more precisely depict the scene changes. Meanwhile, the algorithm is forward-modulated by physical signals (i.e., multi-spectral signals) and is not affected by the content of the shooting scene, thus improving the robustness.

[0016] In another possible implementation, the calibration scene is a standard color card scene; a specific implementation for determining the image color processing parameters based on the reconstructed calibration image is: determining the module color transfer parameters based on the reconstructed calibration image and the target reconstructed calibration image, where the target reconstructed calibration image is the reconstructed calibration image corresponding to the target camera module; determining the image color processing parameters based on the transfer matrix and the target image color processing parameters, where the target image color processing parameters are the image color processing parameters corresponding to the target camera module.

[0017] This application converts the traditional discrete classification task into a continuous spectral component regression task, and dynamically reconstructs the calibration image according to the component ratios of each base light source in the scene reflection spectrum L, which can accurately model the continuous color changes brought about by scene changes and significantly improve the upper limit of the color transfer accuracy.

[0018] In another possible implementation, the image color processing parameters include color correction parameters; the calibration scene is a standard color card scene; a specific implementation for determining the image color processing parameters based on the reconstructed calibration image is: determining the standard color distribution based on the scene reflection spectrum, the reflectivity of the standard color card, and the standard human eye response curve, where the standard color distribution indicates the color distribution of the standard color card in the human eye response space; determining the color correction parameters based on the color distribution in the reconstructed calibration image and the standard color distribution.

[0019] This application converts the traditional discrete classification task into a continuous spectral component regression task, and dynamically reconstructs the calibration image according to the component ratios of each base light source in the scene reflection spectrum L, which can accurately model the continuous color changes brought about by scene changes and significantly improve the upper limit of the color restoration accuracy. At the same time, the determination of the end-to-end color correction parameters avoids the color loss caused by the gamut truncation of the intermediate module, and the optimization direction of the color can be controlled by modulating the fitting target.

[0020] In another possible implementation, the image data processing method provided by the embodiments of the present application further includes: reading K calibration images from a target memory. For example, the K calibration images are stored in the memory corresponding to the ISP. When image data needs to be processed, the K calibration images are read from the memory corresponding to the ISP to facilitate subsequent image processing using the K calibration images.

[0021] In another possible implementation, the image data processing method provided by the embodiments of the present application is applied to a camera system. The camera system includes a camera module and a multispectral sensor module. The image data of the target scene is obtained by the camera module capturing the target scene, and the multispectral data of the target scene is obtained by the multispectral sensor module collecting spectral signals of the target scene; the K calibration images are obtained by the camera module.

[0022] In another possible implementation, the camera module includes an image sensor. The image sensor has N response channels, where N is a positive integer; the K basic light sources are the K light sources used by the camera module in the feature calibration stage. The overlap degree of the wavelength ranges corresponding to each of the K basic light sources is less than a preset threshold, and the wavelength range corresponding to the K basic light sources is greater than or equal to the wavelength response range of the image sensor. K is an integer greater than N to facilitate combining the K basic light sources to form a light source spectrum within the response range of the image sensor.

[0023] In a second aspect, the present application provides an image data processing device. The device includes an acquisition module, a determination module, and a processing module. Among them, the acquisition module is used to acquire the image data and multispectral data of the target scene; the determination module is used to determine the proportion of K basic light source components based on the multispectral data, where the proportion of K basic light source components indicates the proportion of the spectrum of each of the K basic light sources in the scene reflection spectrum, and the scene reflection spectrum indicates the reflection spectrum distribution corresponding to the target shooting scene; the processing module is used to process the image data of the target scene based on the proportion of K basic light source components and the K calibration images. Each of the K calibration images is one of the K images obtained by shooting the calibration scene under each of the K basic light sources through the camera module.

[0024] In a possible implementation, the determination module is specifically configured to: determine the scene reflection spectrum based on the multispectral data; split the scene reflection spectrum based on the scene reflection spectrum and the spectrum of each of the K basic light sources to obtain a splitting result, where the splitting result indicates the proportion of the spectrum of each basic light source in the scene reflection spectrum; and determine the proportion of K basic light source components based on the splitting result.

[0025] In another possible implementation, the determination module is specifically configured to: use the multispectral data as the input of a trained neural network model and output the proportion of K basic light source components.

[0026] In another possible implementation, the processing module is specifically configured to: determine a reconstructed calibration image based on the K base light source component ratios and the K calibration images; and process the image data of the target scene based on the reconstructed calibration image.

[0027] In another possible implementation, a specific implementation of determining a reconstructed calibration image based on the K base light source component ratios and the K calibration images is: using the K base light source component ratios as the weights of the respective calibration images among the K calibration images, and performing a weighted summation operation on the K calibration images to obtain the reconstructed calibration image.

[0028] In another possible implementation, a specific implementation of determining image color processing parameters based on the reconstructed calibration image is: determining image color processing parameters based on the reconstructed calibration image; and processing the image data of the target scene based on the image processing parameters.

[0029] In another possible implementation, the image color processing parameters include lens shadow correction parameters; the calibration scene is a uniform illumination scene; and a specific implementation of determining image color processing parameters based on the reconstructed calibration image is: determining the lens shadow correction parameters based on the color distribution in the reconstructed calibration image and the color distribution in the image data of the target scene.

[0030] In another possible implementation, the calibration scene is a standard color card scene; and a specific implementation of determining image color processing parameters based on the reconstructed calibration image is: determining module color transfer parameters based on the reconstructed calibration image and the target reconstructed calibration image, where the target reconstructed calibration image is the reconstructed calibration image corresponding to the target camera module; and determining the image color processing parameters based on the module color transfer parameters and the target image color processing parameters, where the target image color processing parameters are the image color processing parameters corresponding to the target camera module.

[0031] In another possible implementation, the image color processing parameters include color correction parameters; the calibration scene is a standard color card scene; and a specific implementation of determining image color processing parameters based on the reconstructed calibration image is: determining a standard color distribution based on the scene reflection spectrum, the reflectance of the standard color card, and the standard human eye response curve, where the standard color distribution indicates the color distribution of the standard color card in the human eye response space; and determining the color correction parameters based on the color distribution in the reconstructed calibration image and the standard color distribution.

[0032] In another possible implementation, the image data processing apparatus provided in this application further includes a reading module, and this reading module is configured to read the K calibration images from the target memory.

[0033] In another possible implementation, the processing device for image data provided by the embodiments of the present application can be deployed in an image signal processor of a camera system. The camera system includes a camera module and a multispectral sensor module. The image data of the target scene is collected by the camera module for the target scene, and the multispectral data of the target scene is obtained by the multispectral sensor module collecting spectral signals for the target scene; the K calibration images are collected by the camera module.

[0034] In another possible implementation, the camera module includes an image sensor, and the image sensor has N response channels, where N is a positive integer; the K basic light sources are the K light sources used by the camera module in the feature calibration stage. The overlap degree of the wavelength ranges corresponding to each of the K basic light sources is less than a preset threshold, and the wavelength ranges corresponding to the K basic light sources are greater than or equal to the wavelength response range of the image sensor, where K is an integer greater than N.

[0035] In a third aspect, an embodiment of the present application provides a terminal device, including a camera module, a multispectral sensor, a memory, and a processor. The camera module is configured to capture image data of a target shooting scene for the target shooting scene; the multispectral sensor is configured to collect multispectral data by collecting multispectral signals for the target shooting scene; instructions are stored in the memory, and when the instructions are executed by the processor, the image data of the target scene is processed based on the method described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0037] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product. The computer program or the computer program product includes instructions. When the instructions are executed, the computer is made to execute the method described in the first aspect.

[0038] In a sixth aspect, an embodiment of the present application further provides a chip, including at least one processor and a communication interface. The processor is configured to execute the method described in the first aspect.

[0039] For example, the chip can be an Image signal processing (ISP) chip, and can implement the method for processing image data described in the first aspect to process image data. Description of the Drawings

[0040] Figure 1 A schematic diagram of a mobile phone applying the method for processing image data provided by the embodiments of the present application is shown;

[0041] Figure 2Shows an implementation architecture diagram of an image data processing method provided by an embodiment of the present application;

[0042] Figure 3 Is a method flowchart of an image data processing method provided by an embodiment of the present application;

[0043] Figure 4 Shows a schematic flow diagram of a method for determining the proportion of K base light source components using multispectral data;

[0044] Figure 5 Shows another schematic flow diagram of a method for determining the proportion of K base light source components using multispectral data;

[0045] Figure 6 Shows a schematic diagram of a camera system performing lens shadow correction processing using the image data processing method provided by an embodiment of the present application;

[0046] Figure 7 Shows a comparison schematic diagram of an image processed by a traditional lens shadow correction scheme and a lens shadow correction scheme using the image data processing method provided by an embodiment of the present application;

[0047] Figure 8 Shows a schematic diagram of a camera system performing module color migration processing using the image data processing method provided by an embodiment of the present application;

[0048] Figure 9 Shows a comparison schematic diagram of an image processed by a module color migration scheme using the image data processing method provided by an embodiment of the present application and a traditional module color migration scheme;

[0049] Figure 10 Shows a schematic diagram of a camera system performing color correction processing using the image data processing method provided by an embodiment of the present application;

[0050] Figure 11 Shows a comparison schematic diagram of an image processed by a traditional color correction scheme and a color correction scheme using the image data processing method provided by an embodiment of the present application;

[0051] Figure 12 Is a schematic structural diagram of a processing device for image data provided by an embodiment of the present application;

[0052] Figure 13 Is a schematic structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0053] The term "and / or" mentioned in this article is an associative relationship describing associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In this article, the symbol " / " indicates that the associated objects are in an "or" relationship. For example, A / B means A or B.

[0054] In the description of the specification and claims of this application, terms such as "first" and "second" are used to distinguish different objects, rather than to describe the specific order of the objects.

[0055] In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of words such as "exemplary" or "for example" aims to present relevant concepts in a specific way.

[0056] In the description of the embodiments of this application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units, etc.; a plurality of elements refers to two or more elements, etc.

[0057] To facilitate the understanding of the solutions of the embodiments of this application, the following first explains the technical terms involved in this article.

[0058] Lens Shading Correction (LSC): A key link in the image processing process, aiming to correct the differences in imaging color and brightness between the center and the periphery caused by the imperfections of the lens optical characteristics.

[0059] Color Correction (CC): A key link in the image processing process, aiming to correct the color response of the imaging module to the human eye color response, so as to restore the imaging color and make it consistent with the human eye perception.

[0060] Metamerism: The imaging signals (colors) are the same, but the spectral compositions are different.

[0061] Spectral Reflectance of the Scene (L): The equivalent spectrum after the light source is reflected by the scene, which is equal to the product of the light source intensity and the scene reflectance in each band.

[0062] Image spatial / spectral resolution: An image is represented by H×W×C, where H×W represents the spatial resolution of the image and C represents the spectral resolution of the image. For example, for a 3-channel color image of a scene, its spectral resolution is 3, while for a multispectral image, usually C>3.

[0063] In related technologies, various solutions have been adopted to achieve the restoration accuracy of image colors. For example, in related technology one, a color scheme based on scene statistics and color temperature, including performing segmentation processing on the image to be processed to obtain image blocks of the image to be processed; obtaining the target color temperature of the image to be processed according to the image data of the image block and the relevant color temperature obtained by the color temperature sensor; determining the light source type of the image to be processed according to the infrared parameters obtained by the color temperature sensor and the target color temperature; and performing image processing on the image to be processed based on the light source type. That is to say, this solution performs block processing on the image, determines the light source type according to the original data of the image block, the color temperature recorded by the color temperature sensor, and the infrared parameters, and performs image processing accordingly, and then obtains the color parameters of the image.

[0064] This solution cannot specifically distinguish different light sources only relying on color temperature and infrared ratio, so there are classification errors, which affect the calculation of color parameters. At the same time, this solution uses discrete light source classification tasks to establish the connection between dynamic scenes and calibration parameters. Since the limited calibration environment cannot fully represent the rich real scenes, there are errors in its color modeling of scenes and it cannot achieve true dynamic color modeling.

[0065] Related technology two, a color approximation scheme based on scene light source classification, including obtaining the multispectral data of the image to be processed; estimating the light source category in the image to be processed according to the multispectral data; and performing color correction on the image to be processed according to the color-related characteristics corresponding to the light source category. That is, this solution classifies the light source according to the multispectral signal, indexes the calibration data under the preset standard light source, and thus calculates the relevant color parameters.

[0066] Although this solution introduces multispectral signals, it is only used to improve the discrimination of the type of scene light source. The calculation of the image correction parameters still depends on the discrete classification results and cannot achieve end-to-end dynamic color modeling; in addition, the scene color characteristics under extreme light sources (such as monochromatic stage lights, etc.) are difficult to be characterized by the linear combination of the preset calibration under the standard light source.

[0067] In view of the above problems, an embodiment of the present application provides an image data processing method and apparatus. By using a multispectral sensor to collect the multispectral signals of the actual shooting scene, and then using the multispectral signals and the characteristic calibration data of the camera module (including K basic light sources and K calibration images), the component ratios of each basic light source of the scene reflection spectrum L of the actual shooting scene are determined. Then, the calibration images are reconstructed by using the component ratios and the K calibration images, and the image color processing parameters are determined by using the reconstructed calibration images. Furthermore, the image signals collected by the main camera module are processed according to the image color processing parameters, fully considering the influence of the response difference of the camera module on the image color processing parameters, achieving unified color effects for different camera modules. At the same time, the image processing method of the embodiment of the present application does not involve discrete tasks such as threshold screening and classification, and can more precisely depict the color changes brought by scene changes to a specific camera module, thereby realizing stable and continuous dynamic color modulation and significantly improving the accuracy of image color.

[0068] The technical solution of the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0069] The image color processing method provided by the embodiment of the present application can be applied to any terminal device with a shooting function, including but not limited to mobile phones, tablet computers, laptop computers, smart screens, cameras, etc.

[0070] The terminal device applying the image data processing method provided by the embodiment of the present application can improve the quality of image color restoration.

[0071] Taking the terminal device as a mobile phone as an example, the image data processing method provided by the embodiment of the present application will be introduced below.

[0072] Figure 1 The schematic diagram of a mobile phone applying the image data processing method provided by the embodiment of the present application is shown. As Figure 1 shown, the mobile phone 10 includes a camera module 11 and a multispectral sensor module 12. By adding the multispectral sensor module 12 to the camera system of the mobile phone, the multispectral signals of the shooting scene collected by the multispectral sensor module 12 and the characteristic calibration data of the camera module by the basic light sources are used to reconstruct the calibration images, and then the reconstructed calibration images are used to assist in guiding the color processing of the image signals collected by the camera module 11, so that the finally output image color is closer to the true color of the photographed scene and the color accuracy of the captured images is improved.

[0073] The camera module 11 may include one or more camera modules, such as one or more of a main camera module, an ultra-wide-angle module, and a telephoto module, etc.

[0074] It can be understood that Figure 1This is only an example of a mobile phone that can apply the image data processing method of the embodiments of the present application, and does not constitute a limitation. For example, the camera module 11 and the multispectral sensor module 12 can be set independently, or the multispectral sensor module 12 can be integrated into the camera module 11 for setting.

[0075] The image sensor of the camera module 11 can be an RGB image sensor with RGB three channels, or other unconventional image sensors, such as an image sensor with unconventional color channels such as RYYB.

[0076] The multispectral sensor module 12 can be a single-point multispectral sensor, which can only measure at a certain spatial point, and the collected multispectral signal is (1*1*N); it can also be an array multispectral sensor with spatial resolution, and the collected multispectral signal is (H*W*N), where N>3, and N represents that the multispectral sensor has N channel responses, which can also be called the frequency domain resolution of the multispectral signal.

[0077] Taking the image sensor of the camera module 11 as an RGB image sensor as an example below, the detailed implementation of the image data processing method provided by the embodiments of the present application will be introduced.

[0078] After the user triggers the camera of the mobile phone to take a picture of the target shooting scene, the RGB image sensor on the mobile phone generates the original image of the current shooting scene (that is, the unprocessed image signal, which can also be called the image signal to be processed). At the same time, the multispectral sensor on the mobile phone collects the multispectral signal of the shooting scene, then uses the multispectral signal and the characteristic calibration data of the camera module to reconstruct the calibration scene map under the current illumination scene, and then calculates the parameters of the corresponding image processing module in the ISP, such as the LSC module, the CC module, etc., and sends the parameters to the corresponding module, so that the ISP processes the raw image data captured by the camera module to obtain an image with high color accuracy.

[0079] Figure 2 Shows an implementation architecture diagram of an image data processing method provided by an embodiment of the present application. As Figure 2As shown in the figure, the implementation architecture of the image data processing method provided by the embodiments of the present application mainly includes four modules: a module color response characterization module, a scene reflection spectrum analysis and component splitting module, a feature scene dynamic reconstruction module, and a parameter calculation module. Among them, the module color response characterization module is used to characterize the characteristics of the color response of the camera module, including the spectral information of K base light sources and K base calibrations (i.e., calibration images). The K base light sources are K light sources with low overlap, low correlation coefficient, and independent of each other, and cover the response range of the camera module; the K base calibrations are the raw images of the K calibration scenes captured by the camera module under the illumination of the K base light sources respectively. The scene reflection spectrum analysis and component splitting module is used to output the component ratio of the scene reflection spectrum of each base light source in the K base light sources in the current shooting scene according to the input multi-spectral signal. The feature scene dynamic reconstruction module is used to reconstruct the calibration image under the current illumination scene according to the component ratio of the scene reflection spectrum of each base light source in the K base light sources and the K calibration images. The parameter calculation module calculates the parameters of the corresponding modules in the ISP according to the reconstructed calibration image, and sends the parameters to the corresponding modules, so that each module in the ISP processes the raw image data captured by the camera module to obtain an image with high color accuracy. For the specific implementation of each module in the module color response characterization module, the scene reflection spectrum analysis and component splitting module, the feature scene dynamic reconstruction module, and the parameter calculation module, please refer to the following description.

[0080] Figure 3 It is a flowchart of the image data processing method provided by the embodiments of the present application. This method can be implemented by the ISP of the camera system to process the image data collected by the camera module, so that the color accuracy of the finally presented image is high, realizing "what you see is what you get". As Figure 3 shown, the image data processing method provided by the embodiments of the present application at least includes steps S301 to S303.

[0081] In step S301, the image data and multi-spectral data of the target scene are acquired.

[0082] Trigger the shooting function of the terminal device, such as the shooting function of a mobile phone, to shoot the shooting scene, trigger the shooting button (whether it is a virtual button or a physical entity button), and the RGB image sensor in the camera module generates the original image of the current shooting scene. For example, for each pixel, the RGB image sensor converts the intensity of the light received by the pixel into the color value of the corresponding color channel of the pixel, and uses the color values of the corresponding color channels of the surrounding pixels as the color values of the other color channels of the pixel, so as to obtain the RGB value of the pixel. The RGB values of all pixels on all RGB image sensors constitute the image data of the target scene, that is, the raw image of the current shooting scene.

[0083] While pressing the shooting button, the multispectral sensor senses the multispectral data of the shooting scene. The resolution of the multispectral data is determined by the number of channels of the multispectral sensor. For example, if the multispectral sensor has 9 response channels, the resolution of the multispectral data collected by the multispectral sensor is 9.

[0084] In step S302, based on the multispectral data, determine the proportion of K basic light source components.

[0085] After obtaining the multispectral data of the current shooting scene, use the multispectral data to determine the proportion of K basic light source components, that is, the proportion of the spectrum of each basic light source in the K basic light sources in the scene reflection spectrum L. The scene reflection spectrum L is the reflection spectrum distribution corresponding to the current shooting scene.

[0086] It should be noted that the basic light sources are a series of light sources with low coincidence and low correlation coefficient used in the calibration stage of the camera module. Any light source or common lighting light sources such as sunlight, incandescent lamp, LED lamp, and fluorescent lamp can be combined through the basic light sources. In other words, the K basic light sources are the K light sources used by the camera module in the feature calibration stage. The overlap degree of the wavelength ranges corresponding to each of the K basic light sources is less than a preset threshold, and the wavelength range corresponding to the K basic light sources is greater than or equal to the wavelength response range of the RGB image sensor. For example, if the wavelength response range of the RGB image sensor is from 400nm to 750nm, the coverage range of the K basic light sources is at least from 400nm to 750nm. The K basic light sources can be respectively: basic light source k1, basic light source k2, basic light source k3, basic light source k4, basic light source k5, basic light source k6, and basic light source k7. And the wavelength coverage ranges of the K basic light sources are respectively: the wavelength coverage range of basic light source k1 is from 400nm to 450nm, the wavelength coverage range of basic light source k2 is from 450nm to 500nm, the wavelength coverage range of basic light source k3 is from 500nm to 550nm, the wavelength coverage range of basic light source k4 is from 550nm to 600nm, the wavelength coverage range of basic light source k5 is from 600nm to 650nm, the wavelength coverage range of basic light source k6 is from 650nm to 700nm, and the wavelength coverage range of basic light source k7 is from 700nm to 750nm. Specifically, the K basic light sources can also be called K monochromatic light sources or K narrow-band light sources.

[0087] The calibration schemes of traditional camera modules are all carried out under standard illumination sources, such as sunlight and LED light, etc. However, both of these two light sources themselves cover all wavelength bands and have overlapping wavelength ranges. In contrast, the embodiments of the present application do not perform calibration under broad-spectrum common light sources, but under a series of characteristic light sources, namely base light sources. In this way, we have a higher degree of freedom. Through the free combination of base light sources, some common light sources can be combined, and some special light sources can also be combined. Thus, the adaptation range of the characteristic calibration of the camera module is wider. The illumination light source of the current shooting scene can be combined by K base light sources with different component ratios, making the characteristic calibration information of the camera module more accurate.

[0088] The following details the characteristic calibration scheme of the camera module.

[0089] First of all, it should be noted that the characteristic calibration stage of the camera module is executed offline. During the offline stage, when the camera module of the terminal device performs calibration, the following operations are carried out:

[0090] S1. Turn on the base light source k1, and use a spectrometer to measure the spectrum of the base light source k1.

[0091] S2. Use the base light source k1 to construct a characteristic calibration scene (which can be simply referred to as a calibration scene or a characteristic scene).

[0092] According to needs, different characteristic calibration scenes can be constructed for the characteristic calibration scene. For example, in LSC, the constructed characteristic calibration scene can be a uniform illumination scene, and in CC, the constructed characteristic calibration scene can be a standard color card scene.

[0093] Taking the uniform illumination scene as an example, turn on the base light source k1, and then set a light homogenizing lens assembly (such as a frosted glass) on the propagation path of the light beam generated by the base light source k1. The light homogenizing assembly homogenizes the light beam emitted by the base light source k1 to construct a uniform illumination scene.

[0094] Taking the standard color card scene as an example, place a standard color card under the illumination of the base light source k1. The standard color card and the light beam emitted by the base light source k1 form a shooting scene, and this shooting scene is the standard color card scene.

[0095] S3. Use the camera module to shoot the characteristic calibration scene to obtain a calibration image.

[0096] Exemplarily, adjust the field of view angle of the camera module of the terminal device so that the characteristic calibration shooting scene constructed in the above steps is within the field of view angle coverage range, trigger the shooting, and the RGB image sensor generates a raw image of the current shooting scene, that is, the calibration image under the illumination of the base light source k1, which can also be called the calibration image corresponding to the base light source k1.

[0097] After the operations of S1 and S3, a set of characteristic calibration data of the camera module is obtained. This set of data includes the spectrum of the base light source k1 and the calibration image corresponding to the base light source k1.

[0098] S4. Change the base light source k1, and repeat S1 to S3 to obtain the characteristic calibration data of the camera module under other base light sources.

[0099] Through the above steps, K sets of data under K base light sources can be obtained. For example, the K base light sources include: base light source k1, base light source k2, base light source k3, base light source k4, base light source k5, base light source k6, and base light source k7. The characteristic calibration data of the camera module includes 7 sets of characteristic calibration data, namely the spectrum of the base light source k1 and the calibration image corresponding to the base light source k1, the spectrum of the base light source k2 and the calibration image corresponding to the base light source k2, the spectrum of the base light source k3 and the calibration image corresponding to the base light source k3, the spectrum of the base light source k4 and the calibration image corresponding to the base light source k4, the spectrum of the base light source k5 and the calibration image corresponding to the base light source k5, the spectrum of the base light source k6 and the calibration image corresponding to the base light source k6, and the spectrum of the base light source k7 and the calibration image corresponding to the base light source k7.

[0100] Of course, in some other examples, the spectrum of the base light source can also be known (for example, using a base light source with a known spectrum for characteristic calibration), and there is no need to measure it.

[0101] After the characteristic calibration of the camera module is completed, optionally, the characteristic calibration data of the camera module can be stored in the memory corresponding to the LSP, so as to process the captured image data using the characteristic calibration data every time the terminal device takes a picture.

[0102] There are various methods for determining the proportion of K base light source components using multispectral data. Exemplarily, the multispectral data can be mapped to the proportion of each base light source in the K base light sources in the scene reflection spectrum L through a trained neural network model, realizing the component analysis of the end-to-end scene reflection spectrum L.

[0103] Figure 4 Shows a schematic flowchart of a method for determining the proportion of K base light source components using multispectral data. As Figure 4As shown, after the multispectral sensor collects the multispectral data of the current shooting scene, the multispectral data is used as the input of the trained neural network model. The trained neural network model performs inference and outputs the proportion of K base light source components. For example, the proportion of the K base light source components output is that the base light source k1 is 0.2, the base light source k2 is 0, the base light source k3 is 0.3, the base light source k4 is 0.4, the base light source k5 is 0, the base light source k6 is 0, and the base light source k7 is 0.1, which means that the base light source k1, the base light source k3, the base light source k4, and the base light source k7 can form the scene reflection spectrum L of the current shooting scene in a ratio of 2:3:4:1.

[0104] The trained neural network model may include a convolutional layer and one or more of a pooling layer, an activation layer, an upsampling layer, a downsampling layer, a self-attention layer, and a residual connection layer. Or the trained neural network model may include a fully connected layer and one or more of a pooling layer, an activation layer, an upsampling layer, a downsampling layer, a self-attention layer, and a residual connection layer.

[0105] In some other examples, the multispectral data needs to be preprocessed before being input into the trained neural network model. For example, preprocessing operations such as dimension elevation operations or encoding operations are performed on the multispectral data to facilitate the inference of the trained neural network model.

[0106] The training of the neural network model is a very mature technology, and appropriate training methods can be selected according to needs, such as supervised training, self-supervised training, semi-supervised training, etc. For the sake of brevity, it will not be elaborated here.

[0107] For another example, the proportion of K base light source components can be obtained by analyzing the multispectral data and K base light sources. For example, based on the multispectral data, the scene reflection spectrum L is determined; based on the scene reflection spectrum L and the spectra of each base light source among the K base light sources, the scene reflection spectrum L is split to obtain a splitting result, and the splitting result indicates the proportion of the spectra of each base light source in the scene reflection spectrum L; based on the splitting result, the proportion of K base light source components is determined.

[0108] The multispectral signal of the real shooting scene is collected by the multispectral sensor, and then the multispectral signal is inversely solved to obtain the scene reflection spectrum L of the real shooting scene (for example, the scene reflection spectrum L is obtained by inversely solving the multispectral signal through an interpolation algorithm), and then the component analysis of the scene reflection spectrum L is performed to analyze the proportion of each base light source in the scene reflection spectrum L. That is to say, the scene reflection spectrum L can be obtained by weighted summation of the proportion of each base light source and the spectra of each base light source.

[0109] For example, by analyzing the L component of the scene reflection spectrum, the proportion of each basic light source among the K basic light sources is as follows: the proportion of basic light source k1 is 0.2, the proportion of basic light source k2 is 0, the proportion of basic light source k3 is 0.3, the proportion of basic light source k4 is 0.4, the proportion of basic light source k5 is 0, the proportion of basic light source k6 is 0, and the proportion of basic light source k7 is 0.1. That is to say, the combination of basic light sources k1, k3, k4, and k7 in the ratio of 2:3:4:1 can form the scene reflection spectrum L of the current captured scene. For example, the spectrum of basic light source k1 * 0.2 + the spectrum of basic light source k2 * 0 + the spectrum of basic light source k3 * 0.3 + the spectrum of basic light source k4 * 0.4 + the spectrum of basic light source k5 * 0 + the spectrum of basic light source k6 * 0 + the spectrum of basic light source k7 * 0.1 = the scene reflection spectrum L.

[0110] Figure 5 FIG. shows a schematic flowchart of another method for determining the proportion of components of K basic light sources using multispectral data. As Figure 5 shown, feature extraction is respectively performed on the multispectral data and the spectral data of each basic spectrum to obtain a feature space with a common physical meaning, and then the proportion information of the basic light source components is obtained by splitting in this feature space.

[0111] The frequency domain resolution of the spectra of each of the K basic light sources is M, and the resolution of the multispectral data is N. Usually, N is less than M, that is to say, the frequency domain resolution of the multispectral data is less than the frequency domain resolution of the basic light source spectrum. For example, the resolution N of the multispectral data is 9, and the spectral resolution M of the basic light source is 30. Therefore, it is necessary to raise the multispectral data to the M dimension so that the frequency domain resolution of the multispectral data is the same as the frequency domain resolution of the basic light source spectrum.

[0112] In some other examples, it is also possible not to perform feature extraction processing on the basic light source spectral data. For example, the feature space can be a sampling space with a fixed interval that can represent the spectral continuous signal. Since the basic light source spectrum itself is recorded in such a feature space, there is no need to perform feature extraction processing on the basic spectrum spectral data anymore. Only the multispectral data needs to be spectrally inverse-solved to raise its dimension to the same dimension feature space as the basic spectrum spectral data, and coefficient splitting is performed in this feature space, and then projected onto the spectral dimension to obtain the proportion of the basic light source components. The proportion of each basic light source component is used as K coefficients, so that k coefficients * the spectra of k basic light sources = the scene reflection spectrum L obtained by inverse-solving the multispectral data.

[0113] In step S303, based on the proportion of K basic light source components and K calibration images, the image data of the target scene is processed.

[0114] After obtaining the K base light source component ratios and the K calibration images through step S302, the K base light source component ratios and the K calibration images can be used to process the image data of the target scene to obtain an image with accurate colors, achieving "what you see is what you get".

[0115] For example, first, based on the K base light source component ratios and the K calibration images, a reconstructed calibration image is determined. Then, based on the reconstructed calibration image, image color processing parameters are determined. Finally, the ISP uses the image color processing parameters to perform color processing on the image data of the target scene.

[0116] Optionally, a specific implementation of determining the reconstructed calibration image based on the K base light source component ratios and the K calibration images can be: taking the K base light source component ratios as the weights of the respective calibration images among the K calibration images, performing a weighted summation operation on the K calibration images to obtain the reconstructed calibration image, and the reconstructed calibration image is the calibration scene image under the current lighting scene.

[0117] For example, the K base light sources include: base light source k1, base light source k2, base light source k3, base light source k4, base light source k5, base light source k6, and base light source k7, and the calibration images corresponding to each base light source are respectively: calibration image k1 corresponding to base light source k1, calibration image k2 corresponding to base light source k2, calibration image k3 corresponding to base light source k3, calibration image k4 corresponding to base light source k4, calibration image k5 corresponding to base light source k5, calibration image k6 corresponding to base light source k6, and calibration image k7 corresponding to base light source k7. The component ratios of each base light source are respectively: base light source k1 is 0.2, base light source k2 is 0, base light source k3 is 0.3, base light source k4 is 0.4, base light source k5 is 0, base light source k6 is 0, and base light source k7 is 0.1. Then the reconstructed calibration image is: calibration image k1 * 0.2 + calibration image k2 * 0 + calibration image k3 * 0.3 + calibration image k4 * 0.4 + calibration image k5 * 0 + calibration image k6 * 0 + calibration image k7 * 0.1.

[0118] The reconstructed calibration image can be used for subsequent processing of the image data of the currently captured scene, such as processing of image color parameters, to ensure the accuracy of the colors of the processed image.

[0119] Using the reconstructed calibration image, image color processing parameters can be determined, such as lens shading correction parameters, module color transfer parameters, and color correction parameters, etc. Then, the image data of the current shooting scene (i.e., the image data of the target scene) captured by the camera module is processed using the image color processing parameters. For example, the lens shading correction parameters are sent to the LSC module, and the LSC module uses the lens shading correction parameters to perform lens shading correction processing on the image data of the target scene; for another example, the module color transfer parameters are sent to the module color transfer module, and the module color transfer module uses the module color transfer parameters to perform color transfer processing on the image data set of the target scene; for another example, the color correction parameters are sent to the CC module, and the CC module uses the color correction parameters to perform color correction on the image data of the target scene.

[0120] The following details the determination of the lens shading correction parameters, module color transfer parameters, and color correction parameters based on the reconstructed calibration image; and the specific implementation of performing lens shading correction processing on the image data of the target scene based on the lens shading correction parameters, color transfer processing on the image data of the target scene based on the module color transfer parameters, and color correction on the image data of the target scene based on the color correction parameters.

[0121] I. Regarding lens shading correction

[0122] When using the reconstructed calibration image to determine the lens shading correction parameters, the calibration scene image is a uniformly illuminated scene, and the reconstructed calibration image is the reconstructed uniformly illuminated scene.

[0123] Figure 6 Fig. shows a schematic diagram of a camera system performing lens shading correction processing using the image data processing method provided in an embodiment of the present application. As Figure 6 shown, after the user triggers the camera of the mobile phone to shoot the target shooting scene, the RGB image sensor on the mobile phone generates the original image of the current shooting scene. At the same time, the multispectral sensor on the mobile phone collects the multispectral signals of the shooting scene, and reads the characteristic calibration data of the camera module stored in advance, including the spectra of K basic light sources and the calibration scene images taken under the K basic light sources (at this time, the calibration scene image is a uniformly illuminated scene).

[0124] The multispectral signal is inversely solved using an inverse solution algorithm to obtain the scene reflection spectrum L of the current shooting scene. The scene reflection spectrum L is split to obtain the component ratios of each of the K basic light sources. The component ratios of each of the K basic light sources are used as the weights of the K uniform illumination scene graphs respectively for weighted processing to obtain the reconstructed uniform illumination scene graph. For example, the K basic light sources are: basic light source k1, basic light source k2, basic light source k3, basic light source k4, basic light source k5, basic light source k6, and basic light source k7, and the calibration images corresponding to each basic light source are: calibration image k1 corresponding to basic light source k1, calibration image k2 corresponding to basic light source k2, calibration image k3 corresponding to basic light source k3, calibration image k4 corresponding to basic light source k4, calibration image k5 corresponding to basic light source k5, calibration image k6 corresponding to basic light source k6, and calibration image k7 corresponding to basic light source k7. The component ratios of each basic light source are: 0.2 for basic light source k1, 0 for basic light source k2, 0.3 for basic light source k3, 0.4 for basic light source k4, 0 for basic light source k5, 0 for basic light source k6, and 0.1 for basic light source k7. Then the reconstructed uniform illumination scene graph is: calibration image k1 * 0.2 + calibration image k2 * 0 + calibration image k3 * 0.3 + calibration image k4 * 0.4 + calibration image k5 * 0 + calibration image k6 * 0 + calibration image k7 * 0.1.

[0125] Finally, the lens shadow correction table is determined using the reconstructed uniform illumination image and the currently captured image, and the currently captured image is subjected to lens shadow correction processing using the lens shadow correction table. For example, the color values of each pixel in the reconstructed uniform illumination image are compared with the color values of each pixel in the currently captured image to obtain the lens shadow correction table, and then the color values of each pixel in the currently captured image are multiplied by the corresponding values in the lens shadow correction table to obtain the image after lens shadow correction.

[0126] Figure 7 A comparison schematic diagram of an image processed by a traditional lens shadow correction scheme and an image processed by a lens shadow correction scheme using the image data processing method provided in the embodiments of the present application is shown.

[0127] After applying the image data processing method provided in the embodiments of the present application for lens shadow correction in an actual mobile phone path, compared with the traditional scheme calculated based on the end-to-end compensation idea, after evaluation, there are significant improvements in both the spatial uniformity and the temporal stability after correction.

[0128] The image data processing method provided in the embodiments of the present application uses multispectral signals to analyze the reflection spectrum distribution of the scene, dynamically reconstructs the characteristic scene based on the spectral composition, alleviates the problem of metamerism, and at the same time, the adaptive parameter calculation improves the correction accuracy and can more finely depict the scene changes. In addition, the algorithm is modulated by the physical signal forward and is not affected by the scene statistical content, improving the robustness.

[0129] II. Regarding the color shift of the module

[0130] When using the reconstructed calibration image to determine the module color shift parameters, the calibration scene image is a standard color card scene, and the reconstructed calibration image is the reconstructed standard color card scene.

[0131] The goal of the module color shift is to shift the color response of the camera module 2 to be consistent with that of the camera module 1 (which can be a camera module for which the image color processing parameters have been determined), so as to unify the color effect and achieve unified parameter adjustment.

[0132] Figure 8 The figure shows a schematic diagram of a camera system performing module color shift processing using the image data processing method provided in the embodiment of the present application. As Figure 8 shown, after the user triggers the camera of the mobile phone to shoot the target shooting scene, the RGB image sensor on the mobile phone generates the original image of the current shooting scene. At the same time, the multispectral sensor on the mobile phone collects the multispectral signal of the shooting scene, reads the characteristic calibration data of the first camera module (the target camera module) and the characteristic calibration data of the second camera module (the camera module of this mobile phone) stored in advance. The characteristic calibration data includes the spectra of K basic light sources and the calibration scene images taken under the K basic light sources (at this time, the calibration scene image is a standard color card scene).

[0133] Use the inverse solution algorithm to inverse solve the multispectral signal to obtain the scene reflection spectrum L of the current shooting scene, split the scene reflection spectrum L to obtain the proportion of each basic light source in the K basic light sources, and use the obtained proportion of the basic light source components as weights to perform weighted reconstruction on the standard color card scenes corresponding to the two camera modules (including the camera module of this mobile phone and the target camera module) to obtain two reconstructed standard color card scenes.

[0134] For example, the K basic light sources of the calibration data of the target camera module are respectively: basic light source k1, basic light source k2, basic light source k3, basic light source k4, basic light source k5, basic light source k6, and basic light source k7. And the calibration images corresponding to each basic light source are respectively: calibration image k1.1 corresponding to basic light source k1, calibration image k1.2 corresponding to basic light source k2, calibration image k1.3 corresponding to basic light source k3, calibration image k1.4 corresponding to basic light source k4, calibration image k1.5 corresponding to basic light source k5, calibration image k1.6 corresponding to basic light source k6, and calibration image k1.7 corresponding to basic light source k7. The proportion of each basic light source component is respectively: 0.2 for basic light source k1, 0 for basic light source k2, 0.3 for basic light source k3, 0.4 for basic light source k4, 0 for basic light source k5, 0 for basic light source k6, and 0.1 for basic light source k7. Then the reconstructed standard color card scene diagram corresponding to the target camera module is: calibration image k1.1 * 0.2 + calibration image k1.2 * 0 + calibration image k1.3 * 0.3 + calibration image k1.4 * 0.4 + calibration image k1.5 * 0 + calibration image k1.6 * 0 + calibration image k1.7 * 0.1. The K basic light sources of the calibration data of the camera module of this mobile phone are respectively: basic light source k1, basic light source k2, basic light source k3, basic light source k4, basic light source k5, basic light source k6, and basic light source k7. And the calibration images corresponding to each basic light source are respectively: calibration image k2.1 corresponding to basic light source k1, calibration image k2.2 corresponding to basic light source k2, calibration image k2.3 corresponding to basic light source k3, calibration image k2.4 corresponding to basic light source k4, calibration image k2.5 corresponding to basic light source k5, calibration image k2.6 corresponding to basic light source k6, and calibration image k2.7 corresponding to basic light source k7. The proportion of each basic light source component is respectively: 0.2 for basic light source k1, 0 for basic light source k2, 0.3 for basic light source k3, 0.4 for basic light source k4, 0 for basic light source k5, 0 for basic light source k6, and 0.1 for basic light source k7. Then the reconstructed standard color card scene diagram corresponding to the camera module of this mobile phone is: calibration image k2.1 * 0.2 + calibration image k2.2 * 0 + calibration image k2.3 * 0.3 + calibration image k2.4 * 0.4 + calibration image k2.5 * 0 + calibration image k2.6 * 0 + calibration image k2.7 * 0.1.

[0135] Finally, the color transfer parameters of the module are determined by using the reconstructed standard color card scene map corresponding to the camera module of this mobile phone and the reconstructed standard color card scene corresponding to the target camera module (which can be called the target reconstructed calibration image), and the color transfer processing of the currently captured image is performed by using the color transfer parameters of the module. For example, for the reconstructed standard color card scene maps obtained by weighted reconstruction corresponding to the two camera modules, that is, the reconstructed standard color card scene map corresponding to the target camera module can be called the reconstructed standard color card scene map raw1, and the reconstructed standard color card scene map corresponding to the camera module of this mobile phone can be called the reconstructed standard color card scene map raw2. The reconstructed standard color card scene map raw1 and the reconstructed standard color card scene map raw2 can be regarded as a raw pair obtained by shooting a standard color card under the current lighting scene. Then, a 3×3 response conversion matrix (i.e., the transfer matrix) of the two camera modules is obtained through least squares fitting of the color blocks corresponding to the reconstructed standard color card scene map raw1 and the reconstructed standard color card scene map raw2. The original image of the currently captured scene captured by this mobile phone is color transferred through the response conversion matrix to obtain an image with the same response as the target camera model, which is convenient for subsequent processing. For example, the target camera module already has determined target color processing parameters. The target color processing parameters are used as the color processing parameters of the image after color transfer processing, and finally the color of the image after color transfer processing is processed by using the target color.

[0136] Figure 9 FIG. shows a comparison diagram of images processed by the module color transfer scheme of the image data processing method provided by the embodiment of the present application and the traditional module color transfer scheme.

[0137] After applying the embodiment of the present application for dynamic color transfer in the actual mobile phone channel, compared with the traditional transfer scheme based on scene statistics and color temperature: in 78% of the evaluation scenes, the color transfer accuracy of the implementation scheme of the present application is higher, and the white point transfer error is reduced by 45%. In difficult scenes such as blue sky, pure color, and large-area neutral colors where it is difficult to transfer accurately, the color similarity between the transferred image and the transfer target is significantly improved after applying the embodiment of the present application.

[0138] The image data processing method provided by the embodiment of the present application is only affected by the forward modulation of the multispectral physical signal, avoiding the color influence of the front-end module (such as lens correction, etc.), and is more accurate and stable; and converting the traditional discrete classification task into a continuous spectral component regression task. The characteristic scene can be dynamically reconstructed according to the spectral composition, and the continuous color change brought by the scene change can be accurately modeled, significantly improving the upper limit of the color transfer accuracy.

[0139] III. For color correction

[0140] When using the reconstructed calibration image to determine the color correction parameters, the calibration scene image is a standard color card scene, and the reconstructed calibration image is the reconstructed standard color card scene.

[0141] The goal of color correction is to transfer the color response of the camera module to be consistent with the human eye color response, so that the color effect of the image conforms to human eye observation and perception, and the imaging effect achieves "what you see is what you get".

[0142] Figure 10 The figure shows a schematic diagram of a camera system performing color correction processing using the image data processing method provided in the embodiments of the present application. As Figure 10 shown, after the user triggers the camera of the mobile phone to take a picture of the target shooting scene, the RGB image sensor on the mobile phone generates the original image of the current shooting scene. At the same time, the multispectral sensor on the mobile phone collects the multispectral signal of the shooting scene, and reads the characteristic calibration data of the camera module stored in advance, including the spectra of K basic light sources and the calibration scene images taken under the K basic light sources (at this time, the calibration scene image is a standard color card scene).

[0143] The multispectral signal is inversely solved using the inverse solution algorithm to obtain the scene reflectance spectrum L of the current shooting scene. The scene reflectance spectrum L is split to obtain the component ratios of each of the K basic light sources. The component ratios of each of the K basic light sources are used as the weights of the K uniform illumination scene images for weighted processing to obtain the reconstructed standard color card scene image. For example, the K basic light sources are: basic light source k1, basic light source k2, basic light source k3, basic light source k4, basic light source k5, basic light source k6, and basic light source k7, and the corresponding calibration images of each basic light source are: calibration image k1 corresponding to basic light source k1, calibration image k2 corresponding to basic light source k2, calibration image k3 corresponding to basic light source k3, calibration image k4 corresponding to basic light source k4, calibration image k5 corresponding to basic light source k5, calibration image k6 corresponding to basic light source k6, and calibration image k7 corresponding to basic light source k7. The component ratios of each basic light source are: 0.2 for basic light source k1, 0 for basic light source k2, 0.3 for basic light source k3, 0.4 for basic light source k4, 0 for basic light source k5, 0 for basic light source k6, and 0.1 for basic light source k7. Then the reconstructed standard color card scene image is: calibration image k1 * 0.2 + calibration image k2 * 0 + calibration image k3 * 0.3 + calibration image k4 * 0.4 + calibration image k5 * 0 + calibration image k6 * 0 + calibration image k7 * 0.1.

[0144] Then, based on the scene reflectance spectrum L and the reflectance fef of the standard color card SG140The color card signal corresponding to the current shooting scene in the human eye space is calculated from the standard human eye response curve CMF, that is, the standard human eye response signal (which can also be called the standard color distribution). Finally, based on the reconstructed standard color card image and the represented human eye response signal, the color correction parameters are determined, and the color correction parameters are used to perform color correction processing on the currently captured image to ensure the accuracy of the color values of the final image. For example, the color values of each pixel in the reconstructed standard color card image are minimally fitted with the color values of each pixel in the quasi-human eye response signal to obtain a 3*3 dynamic color correction matrix, and the color correction matrix is obtained. Then, the color values of each pixel in the currently captured image are multiplied by the corresponding values in the color correction matrix to obtain the image after color correction processing.

[0145] Figure 11 The figure shows a comparison diagram of an image processed by a traditional color correction scheme and an image processed by the color correction scheme using the image data processing method provided in the embodiments of the present application.

[0146] By calculating the color angle error of the color patches on the 24-color card after the reduction algorithm under the standard light source, compared with the traditional algorithm based on color temperature, the color reduction error is reduced by 15%-32%. Under extreme light sources such as bars and stages, the degree of color overflow is alleviated, and the retention of scene details is significantly improved.

[0147] The image data processing method provided in the embodiments of the present application uses the fitting of an end-to-end color correction matrix to avoid color loss caused by gamut truncation of intermediate modules, and can control the optimization direction of colors by modulating the fitting target; and converts the traditional discrete classification task into a continuous spectral component regression task. Dynamically reconstructing the feature scene according to the spectral composition can accurately model the continuous color changes brought about by scene changes, and significantly improve the upper limit of color reduction accuracy.

[0148] In summary, for the image data processing method provided in the embodiments of the present application, due to the innovative feature calibration method of the camera module (that is, calibration using the base light source), it is possible to use fewer and more independent calibrations to cover a richer dynamic scene, significantly reducing the calibration redundancy, and it can be quickly realized automatically. This feature calibration method of the embodiments of the present application significantly reduces the time and labor costs of calibration and the storage space of feature files; secondly, the color parameter optimization scheme based on multi-spectral signals in the embodiments of the present application can accurately restore key color parameters using higher-dimensional spectral signals, avoiding color differences caused by metamerism, thereby improving the authenticity and accuracy of color reduction; and the embodiments of the present application can more finely depict the color changes brought about by scene changes to a specific module by performing dynamic scene feature modeling and optimizing color parameters, so as to achieve stable and continuous dynamic color modulation.

[0149] Based on the same concept as the foregoing method embodiments, an image data processing apparatus 1200 is further provided in an embodiment of the present application. The image data processing apparatus 1200 includes units or means for implementing Figures 3 to 11 each step executed by the terminal device in the method shown.

[0150] Figure 12 FIG. 6 is a schematic structural diagram of an image data processing apparatus provided in an embodiment of the present application. The image data processing apparatus can be applied to any terminal device having a camera module and a multispectral sensor module. The terminal device includes, but is not limited to, a mobile phone, a tablet computer, a laptop computer, a smart screen, a camera, etc., to improve the color reproduction accuracy of the captured image.

[0151] As Figure 12 shown, the image data processing apparatus 1200 at least includes an acquisition module 1201, a determination module 1202, and a processing module 1203. Among them, the acquisition module 1201 is used to acquire image data and multispectral data of a target scene. The image data of the target scene is obtained by the camera module shooting the target shooting scene, and the multispectral data is obtained by the multispectral sensor module collecting spectral signals of the target shooting scene; the determination module 1202 is used to determine the proportion of K basic light source components based on the multispectral data, where the proportion of K basic light source components indicates the proportion of the spectrum of each basic light source among the K basic light sources in the scene reflection spectrum, and the scene reflection spectrum indicates the reflection spectrum distribution corresponding to the target shooting scene; the processing module 1203 is used to process the image data of the target scene based on the proportion of K basic light source components and K calibration images. Each of the K calibration images is an image obtained by shooting the calibration scene under each of the K basic light sources through the camera module.

[0152] In a possible implementation, the determination module 1202 is specifically configured to: determine the scene reflection spectrum based on the multispectral data; split the scene reflection spectrum based on the scene reflection spectrum and the spectrum of each basic light source among the K basic light sources to obtain a splitting result, where the splitting result indicates the proportion of the spectrum of each basic light source in the scene reflection spectrum; determine the proportion of K basic light source components based on the splitting result.

[0153] In another possible implementation, the determination module 1202 is specifically configured to: use the multispectral data as the input of a trained neural network model and output the proportion of K basic light source components.

[0154] In another possible implementation, the processing module 1203 is specifically configured to: determine a reconstructed calibration image based on the proportion of K basic light source components and K calibration images; process the image data of the target scene based on the reconstructed calibration image.

[0155] In another possible implementation, a specific implementation for determining a reconstructed calibration image based on the proportion of K base light source components and K calibration images is as follows: using the proportion of the K base light source components as the weights for each of the K calibration images respectively, performing a weighted summation operation on the K calibration images to obtain the reconstructed calibration image.

[0156] In another possible implementation, a specific implementation for determining image color processing parameters based on the reconstructed calibration image is as follows: determining image color processing parameters based on the reconstructed calibration image; processing the image data of the target scene based on the image processing parameters.

[0157] In another possible implementation, the image color processing parameters include lens shadow correction parameters; the calibration scene is a uniformly illuminated scene; a specific implementation for determining image color processing parameters based on the reconstructed calibration image is as follows: determining the lens shadow correction parameters based on the color distribution in the reconstructed calibration image and the color distribution in the image data of the target scene.

[0158] In another possible implementation, the calibration scene is a standard color card scene; a specific implementation for determining image color processing parameters based on the reconstructed calibration image is as follows: determining the module color transfer parameters based on the reconstructed calibration image and the target reconstructed calibration image, where the target reconstructed calibration image is the reconstructed calibration image corresponding to the target camera module; determining the image color processing parameters based on the module color transfer parameters and the target image color processing parameters, where the target image color processing parameters are the image color processing parameters corresponding to the target camera module.

[0159] In another possible implementation, the image color processing parameters include color correction parameters; the calibration scene is a standard color card scene; a specific implementation for determining image color processing parameters based on the reconstructed calibration image is as follows: determining the standard color distribution based on the scene reflection spectrum, the reflectivity of the standard color card, and the standard human eye response curve, where the standard color distribution indicates the color distribution of the standard color card in the human eye response space; determining the color correction parameters based on the color distribution in the reconstructed calibration image and the standard color distribution.

[0160] In another possible implementation, the image data processing apparatus 1200 provided in this application further includes a reading module 1204, and this reading module 1204 is used to read the K calibration images from the target memory.

[0161] In another possible implementation, the camera module includes an image sensor, the image sensor has N response channels, N is a positive integer; the K base light sources are the K light sources used by the camera module in the feature calibration stage, the overlap degree of the wavelength ranges corresponding to each of the K base light sources is less than a preset threshold, and the wavelength ranges corresponding to the K base light sources are greater than or equal to the wavelength response range of the image sensor, and K is an integer greater than N.

[0162] The image data processing device 1200 according to an embodiment of the present invention may correspond to executing the method described in the embodiments of the present application, and the above and other operations and / or functions of each module in the image data processing device 1200 are respectively for implementing Figures 3 to 11 the corresponding processes of each method in, for the sake of brevity, will not be elaborated here.

[0163] It should be understood that the "module" mentioned in the embodiments of the present application may be implemented in the form of software and / or hardware, and the embodiments of the present application do not limit this. For example, the "module" may be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a combined logic circuit, and / or other suitable components that support the described functions.

[0164] Based on the same concept as the foregoing method embodiments, an embodiment of the present application also provides a terminal device, which at least includes a processor and a memory. When the processor executes the program stored on the memory, it can implement Figures 3 - 11 the units or modules of each step in the method shown.

[0165] Figure 13 FIG. is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

[0166] As Figure 13 shown, the terminal device 1300 includes at least one processor 1301, a memory 1302, a communication interface 1303, an RGB image sensor 1304, and a multispectral sensor 1305. Among them, the processor 1301, the memory 1302, the communication interface 1303, the RGB image sensor 1304, and the multispectral sensor 1305 are communicatively connected, and the communication connection can be achieved by wire (such as a bus) or wirelessly. The communication interface 1303 is used to receive or send data sent by other devices. The memory 1302 stores computer instructions, and the processor 1301 executes the computer instructions to execute the method in the foregoing method embodiments.

[0167] It should be understood that in the embodiments of the present application, the processor 1301 may be a central processing unit (CPU), and the processor 1301 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0168] The memory 1302 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1301. The memory 1302 may also include a non-volatile random access memory.

[0169] The memory 1302 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0170] It should be understood that the terminal device 1300 according to the embodiments of the present application may execute the method implemented in the embodiments of the present application Figures 3 - 11 as shown. For a detailed description of the implementation of this method, please refer to the above. For the sake of brevity, it will not be repeated here.

[0171] Embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer instructions are executed by a processor, the above-mentioned method is implemented.

[0172] Embodiments of the present application provide a chip, which includes at least one processor and an interface. The at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the above-mentioned method.

[0173] Embodiments of the present application provide a computer program or a computer program product, which includes instructions. When the instructions are executed, the computer is made to execute the above-mentioned method.

[0174] Those of ordinary skill in the art should further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are implemented in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0175] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0176] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present application. It should be understood that the above description is only the specific embodiments of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing image data, characterized in that: Applied to a camera system, the camera system includes a camera module and a multispectral sensor module, the camera module includes an image sensor, the image sensor has N response channels, N is a positive integer, and the method includes: Acquire image data and multispectral data of the target scene; Based on the multispectral data, determining the proportion of K basic light source components, wherein the proportion of the K basic light source components indicates the proportion of the spectrum of each basic light source among the K basic light sources in the scene reflection spectrum, the K basic light sources are the K light sources used by the camera module in the feature calibration stage, the overlap of the wavelength range corresponding to each basic light source among the K basic light sources is less than a preset threshold, and the wavelength range corresponding to the K basic light sources is greater than or equal to the wavelength response range of the image sensor, K is an integer greater than N, and the scene reflection spectrum indicates the reflection spectrum distribution of the target scene; Determine a reconstructed calibration image based on the K basic light source component proportions and K calibration images, where the K calibration images are K images of the calibration scene respectively collected under the illumination of the K basic light sources; Based on the reconstructed calibration image, image data of the target scene is processed.

2. The method according to claim 1, characterized in that The determining, based on the multispectral data, the composition ratios of the K basic light sources comprises: Determining the scene reflectance spectrum based on the multispectral data; Splitting the scene reflection spectrum based on the scene reflection spectrum and the frequency spectrum of each of the K basic light sources to obtain a splitting result, wherein the splitting result indicates the proportion of the frequency spectrum of each basic light source in the scene reflection spectrum; Based on the splitting result, the proportions of the K basic light source components are determined.

3. The method according to claim 1, characterized in that The determining, based on the multispectral data, the composition ratios of the K basic light sources comprises: The multispectral data is used as the input of the trained neural network model, and the proportions of the K basic light source components are output.

4. The method according to claim 1, characterized in that The step of determining the reconstructed calibration image based on the K basic light source component proportions and the K calibration images includes: The proportions of the K basic light source components are respectively used as the weights of each calibration image in the K calibration images, and a weighted sum operation is performed on the K calibration images to obtain the reconstructed calibration image.

5. The method according to claim 4, characterized in that The processing of the image data of the target scene based on the reconstructed calibration image includes: Determining image color processing parameters based on the reconstructed calibration image; Based on the image color processing parameters, the image data of the target scene is processed.

6. The method according to claim 5, characterized in that The image color processing parameters include lens shading correction parameters; The calibration scene is a uniform illumination scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: The lens shading correction parameters are determined based on the color distribution in the reconstructed calibration image and the color distribution in the image data of the target scene.

7. The method according to claim 5, characterized in that The calibration scene is a standard color card scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: Determine the module color migration parameters based on the reconstructed calibration image and the target reconstructed calibration image, wherein the target reconstructed calibration image is a reconstructed calibration image corresponding to the target camera module; Based on the module color migration parameters and the target image color processing parameters, the image color processing parameters are determined, and the target image color processing parameters are the image color processing parameters corresponding to the target camera module.

8. The method according to claim 5, characterized in that The image color processing parameters include color correction parameters; The calibration scene is a standard color card scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: Determine a standard color distribution based on the scene reflectance spectrum, the reflectance of the standard color card and a standard human eye response curve, wherein the standard color distribution indicates the color distribution of the standard color card in a human eye response space; The color correction parameters are determined based on the color distribution of the reconstructed calibration image and the standard color distribution.

9. The method according to claim 1, characterized in that: Also includes: The K calibration images are read from the target memory.

10. The method according to any one of claims 1 to 9, characterized in that: The image data of the target scene is obtained by the camera module collecting the target scene, and the multispectral data of the target scene is obtained by the multispectral sensor module collecting spectral signals of the target scene; The K calibration images are acquired by the camera module.

11. An image data processing device, characterized in that: Applied to a camera system, the camera system includes a camera module and a multispectral sensor module, the camera module includes an image sensor, the image sensor has N response channels, N is a positive integer, and the device includes: An acquisition module, used to acquire image data and multispectral data of a target scene; A determination module, configured to determine a proportion of K basic light source components based on the multispectral data, wherein the proportion of the K basic light source components indicates a proportion of a spectrum of each of the K basic light sources in a scene reflectance spectrum, the K basic light sources are the K light sources used by the camera module in a feature calibration phase, an overlap of wavelength ranges corresponding to each of the K basic light sources is less than a preset threshold, and the wavelength ranges corresponding to the K basic light sources are greater than or equal to a wavelength response range of the image sensor, K is an integer greater than N, and the scene reflectance spectrum indicates a reflectance spectrum distribution corresponding to the target scene; A processing module, which determines a reconstructed calibration image based on the proportions of the K basic light sources and the K calibration images, where the K calibration images are K images of the calibration scene respectively collected under the illumination of the K basic light sources; Based on the reconstructed calibration image, image data of the target scene is processed.

12. The device according to claim 11, characterized in that The determination module is specifically used for: Determining the scene reflectance spectrum based on the multispectral data; Splitting the scene reflection spectrum based on the scene reflection spectrum and the frequency spectrum of each of the K basic light sources to obtain a splitting result, wherein the splitting result indicates the proportion of the frequency spectrum of each basic light source in the scene reflection spectrum; Based on the splitting result, the proportions of the K basic light source components are determined.

13. The device according to claim 11, characterized in that The determination module is specifically used for: The multispectral data is used as the input of the trained neural network model, and the proportions of the K basic light source components are output.

14. The device according to claim 11, characterized in that The step of determining the reconstructed calibration image based on the K basic light source component proportions and the K calibration images includes: The proportions of the K basic light source components are respectively used as the weights of each calibration image in the K calibration images, and a weighted sum operation is performed on the K calibration images to obtain the reconstructed calibration image.

15. The device according to claim 14, characterized in that The processing of the image data of the target scene based on the reconstructed calibration image includes: Determining image color processing parameters based on the reconstructed calibration image; Based on the image color processing parameters, the image data of the target scene is processed.

16. The device according to claim 15, characterized in that The image color processing parameters include lens shading correction parameters; The calibration scene is a uniform illumination scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: The lens shading correction parameters are determined based on the color distribution in the reconstructed calibration image and the color distribution in the image data of the target scene.

17. The device according to claim 15, characterized in that The calibration scene is a standard color card scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: Determine the module color migration parameters based on the reconstructed calibration image and the target reconstructed calibration image, wherein the target reconstructed calibration image is a reconstructed calibration image corresponding to the target camera module; Based on the module color migration parameters and the target image color processing parameters, the image color processing parameters are determined, and the target image color processing parameters are the image color processing parameters corresponding to the target camera module.

18. The device according to claim 15, characterized in that The image color processing parameters include color correction parameters; The calibration scene is a standard color card scene; The step of determining image color processing parameters based on the reconstructed calibration image includes: Determine a standard color distribution based on the scene reflectance spectrum, the reflectance of the standard color card and a standard human eye response curve, wherein the standard color distribution indicates the color distribution of the standard color card in a human eye response space; The color correction parameters are determined based on the color distribution of the reconstructed calibration image and the standard color distribution.

19. The device according to claim 11, characterized in that Also includes: The reading module is used to read the K calibration images from the target memory.

20. The device according to any one of claims 11 to 19, characterized in that The image data of the target scene is obtained by the camera module collecting the target scene, and the multispectral data of the target scene is obtained by the multispectral sensor module collecting spectral signals of the target scene; The K calibration images are acquired by the camera module.

21. A terminal device, comprising a camera module, a multispectral sensor, a memory and a processor, characterized in that: The camera module is used to shoot a target shooting scene to obtain image data of the target scene; The multispectral sensor is used to collect multispectral signals of the target shooting scene to obtain multispectral data; The memory stores instructions, and when the instructions are executed by the processor, the image data of the target scene is processed based on the method according to any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images