Image processing method and related device

Abstract

The embodiment of the application discloses an image processing method, which can be applied to color processing scenes such as white balance and color restoration, and the method comprises the following steps: acquiring a first to-be-processed image through a color camera; acquiring first ambient spectrum information through a multispectral sensor, the first ambient spectrum information corresponding to a same shooting scene as the first to-be-processed image; acquiring a white balance gain based on the first to-be-processed image and the first ambient spectrum information; and performing first processing on the first to-be-processed image to obtain a first target image, wherein the first processing comprises white balance processing based on the white balance gain. By introducing the multispectral sensor to collect the ambient spectrum information corresponding to the to-be-processed image, the to-be-processed image can be adjusted in real time. In addition, compared with the light source estimation mode in the prior art, the adjustment quality of the target image can be improved.

Description

[0001] This application is a divisional application. The original application has the application number 202110867203.0 and the original application date is July 29, 2021. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to an image processing method and related equipment. Background Technology

[0003] As users' demand for photography increases, the requirements for the quality of images captured by electronic devices are also rising. When users take pictures with electronic devices, differences in shooting environments lead to discrepancies between the image produced and the actual object. For camera imaging, color processing is a crucial aspect affecting the final result, and color shading, white balance, and color reproduction are key factors influencing color accuracy.

[0004] Currently, the most common color processing method in the industry is to calibrate multiple light sources in offline scenarios to obtain correction parameters for different light sources, and then adjust the original image captured by the color camera according to the correction parameters to obtain the target image to be displayed to the user.

[0005] However, the above-mentioned multiple light source calibration methods in offline scenarios can be understood as estimating the light source. The correction parameters obtained in this way are not accurate enough, which affects subsequent color processing. Summary of the Invention

[0006] This application provides an image processing method and related apparatus. By introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image can be adjusted in real time. Furthermore, since the environmental spectral information corresponding to the image to be processed is collected, the adjustment quality of the target image can be improved compared to the existing method of estimating the light source.

[0007] The first aspect of this application provides an image processing method applicable to color processing scenarios such as white balance, color reproduction, and color uniformity. This method can be applied to an image processing device, which includes a color camera and a multispectral sensor. The method includes: acquiring a first image to be processed via the color camera; acquiring first environmental spectral information via the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed; acquiring a white balance gain based on the first image to be processed and the first environmental spectral information; and performing a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes white balance processing based on the white balance gain.

[0008] In this embodiment, by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image can be adjusted in real time. Furthermore, since the environmental spectral information corresponding to the image to be processed is collected, the adjustment quality of the target image can be improved compared to the existing method of estimating the light source.

[0009] Optionally, in one possible implementation of the first aspect, the first image to be processed and the first target image are solid color images or color images with a large area of ​​solid color.

[0010] In this possible implementation, compared to the existing technology of using grayscale world algorithms for white balance, the method of using the first environmental spectral information collected by a multispectral sensor to perform white balance on a solid color image can improve the adjustment quality of the target image.

[0011] Optionally, in one possible implementation of the first aspect, the above step of obtaining white balance gain based on the first image to be processed and the first environmental spectral information includes: inputting the first environmental spectral information and the first image to be processed into a trained neural network to obtain white balance gain; the trained neural network is obtained by training the neural network with training data as input and aiming to make the loss function value less than a threshold, the training data includes the original training image and training spectral information, the original training image and training spectral information correspond to the same shooting scene, the output of the neural network includes white balance gain, the loss function is used to indicate the difference between the white balance gain output by the neural network and the actual white balance gain, and the actual white balance gain is obtained by processing the response value of the gray card in the shooting scene.

[0012] In this possible implementation, compared to the gray-world algorithm used in the prior art for white balance, the method of using the first environmental spectral information collected by a multispectral sensor and the white balance gain obtained by a neural network to perform white balance on the image can improve the adjustment quality of the target image.

[0013] Optionally, in one possible implementation of the first aspect, the above steps further include: acquiring multiple spectral response functions of the color camera; acquiring multiple compensation values ​​based on the first environmental spectral information and the multiple spectral response functions; the first processing further includes: color shading processing based on the multiple compensation values.

[0014] In this possible implementation, compared to the existing technology that requires offline calibration to perform color uniformity on the image, real-time calculation can be achieved, avoiding problems caused by potential errors in offline table selection. Furthermore, since the color uniformity compensation value generated from the first environmental spectral information collected by a multispectral sensor can improve the quality of color uniformity compared to offline calibration in existing technologies.

[0015] Optionally, in one possible implementation of the first aspect, the above steps further include: obtaining the tristimulus value curve and the reflectance of the color card; obtaining a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and the tristimulus value curve; the first processing further includes: color space conversion processing based on the color correction matrix.

[0016] This possible implementation, compared to existing technologies that require offline calibration for image color restoration, allows for real-time calculation and avoids potential errors caused by offline table selection. Furthermore, since the color space transformation matrix is ​​generated from the first environmental spectral information collected by a multispectral sensor, the quality of color restoration can be improved compared to offline calibration in existing technologies.

[0017] Optionally, in one possible implementation of the first aspect, the above steps, based on first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves, to obtain a color correction matrix, include: converting the first environmental spectral information into a light source curve; obtaining a first response value of a color card to a color camera based on multiple spectral response functions, light source curves, and reflectance; obtaining a second response value of the color card to a first human eye color space based on the tristimulus value curves, light source curves, and reflectance, wherein the first human eye color space is the response space corresponding to the human eye matching function; and obtaining a color correction matrix based on the first response value and the second response value, wherein the color correction matrix is ​​used to represent the correlation between the first response value and the second response value.

[0018] In this possible implementation, the color space conversion matrix is ​​obtained by acquiring two response values. Since the color space conversion matrix is ​​generated by the first environmental spectral information collected by the multispectral sensor, the quality of color reproduction can be improved compared to offline calibration in the prior art.

[0019] Optionally, in one possible implementation of the first aspect, the first processing described above further includes: post-processing the image after white balance processing to obtain the first target image.

[0020] In this possible implementation, the method can be understood as adjusting the image based on the conversion relationship between the human eye color response space (the response space constituted by the CIE1931 human eye matching function) and other human eye color response spaces (such as the response space calculated by color adaptation CAT02 in the color appearance model CIECAM02), which is beneficial for subsequent white balance processing.

[0021] Optionally, in one possible implementation of the first aspect, the above steps further include: displaying the first target image to the user.

[0022] In this possible implementation, the first image to be processed is adjusted using the first environmental spectral information collected by a multispectral sensor, and the adjusted image is displayed to the user, thereby improving the color processing effect of the image and enhancing the user experience.

[0023] Optionally, in one possible implementation of the first aspect, the above steps further include: acquiring a second image to be processed via a color camera; acquiring second environmental spectral information via a multispectral sensor, wherein the second environmental spectral information and the second image to be processed correspond to the same shooting scene; determining filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information; filtering the first target image and the second image to be processed based on the filtering parameters to obtain correction parameters; and adjusting the second image to be processed based on the correction parameters to obtain the second target image.

[0024] In this possible implementation, the correction parameters of the second image to be processed are determined by similarity, which improves the temporal stability of color processing while taking into account sensitivity. That is, it avoids the flickering of color effects in the temporal domain, and at the same time can respond to changes in the environment in a timely manner to adjust the parameters.

[0025] Optionally, in one possible implementation of the first aspect, the above steps determine the filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information, including: generating a filtering intensity function based on the similarity; and determining the filtering parameters based on the filtering intensity function. Similarity and filtering intensity are positively correlated; that is, the greater the similarity, the stronger the filtering intensity. In other words, if the difference between the first environmental spectral information and the second environmental spectral information is small, historical correction parameters can be used, or the historical correction parameters (i.e., correction parameters obtained by comparing the first environmental spectral information with the color channels in the first image to be processed) can have a larger weight, while the new correction parameters (i.e., correction parameters obtained by comparing the second environmental spectral information with the second image to be processed) have a smaller weight, thereby obtaining the correction parameters for the second image to be processed. If the difference between the first environmental spectral information and the second environmental spectral information is large (e.g., the difference between indoor and outdoor environments), new correction parameters can be used, or the new correction parameters can have a larger weight, while the historical correction parameters have a smaller weight, thereby obtaining the correction parameters for the second image to be processed.

[0026] In this possible implementation, a filter intensity function is generated through similarity. The higher the similarity, the greater the weight of the correction parameters of historical frames. This achieves both improved temporal stability of color processing and improved sensitivity, that is, avoiding flickering of color effects in the temporal domain, while also responding promptly to changes in the environment that lead to parameter adjustments.

[0027] A second aspect of this application provides an image processing method applicable to color processing scenarios such as white balance, color restoration, and color uniformity. This method can be applied to an image processing device, which includes a color camera and a multispectral sensor. The method includes: acquiring a first image to be processed via the color camera; acquiring first environmental spectral information via the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed; acquiring multiple spectral response functions of the color camera; acquiring multiple compensation values ​​based on the multiple first environmental spectral information and the multiple spectral response functions; and performing a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color uniformity color shading processing based on the multiple compensation values.

[0028] In this embodiment, compared to the prior art which requires offline calibration to perform color uniformity on images, real-time calculation can be achieved, avoiding problems caused by potential errors in offline table selection. Furthermore, the color uniformity compensation value generated from the first environmental spectral information collected by a multispectral sensor can improve the quality of color uniformity compared to offline calibration in the prior art.

[0029] A third aspect of this application provides an image processing method applicable to color processing scenarios such as white balance, color reproduction, and color uniformity. This method can be applied to an image processing device, which includes a color camera and a multispectral sensor. The method includes: acquiring a first image to be processed based on the color camera; acquiring first environmental spectral information based on the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed; acquiring multiple spectral response functions of the color camera; acquiring tristimulus value curves and the reflectance of a color chart; acquiring a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves; and performing a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color space conversion processing based on the color correction matrix.

[0030] In this embodiment, compared to the prior art which requires offline calibration for image color restoration, real-time calculation can be achieved, avoiding potential errors caused by offline table selection. Furthermore, since the color space transformation matrix is ​​generated from the first environmental spectral information collected by a multispectral sensor, the quality of color restoration can be improved compared to offline calibration in the prior art.

[0031] Optionally, in one possible implementation of the third aspect, the above steps: obtaining a color correction matrix based on first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves, include: converting the first environmental spectral information into a light source curve; obtaining a first response value of a color card to a color camera based on multiple spectral response functions, light source curves, and reflectance; obtaining a second response value of the color card to a first human eye color space based on the tristimulus value curves, light source curves, and reflectance, wherein the first human eye color space is the response space corresponding to the human eye matching function; and obtaining a color correction matrix based on the first response value and the second response value, wherein the color correction matrix is ​​used to represent the conversion relationship between the first response value and the second response value.

[0032] In this possible implementation, the color space conversion matrix is ​​obtained by acquiring two response values. Since the color space conversion matrix is ​​generated by the first environmental spectral information collected by the multispectral sensor, the quality of color reproduction can be improved compared to offline calibration in the prior art.

[0033] Optionally, in one possible implementation of the third aspect, the above steps further include: adjusting the image after color space conversion based on the conversion relationship between the first human eye color space and the second human eye color space, wherein the second human eye color space is the response space corresponding to the color appearance model when performing color adaptation.

[0034] In this possible implementation, the method can be understood as adjusting the image based on the conversion relationship between the human eye color response space (the response space constituted by the CIE1931 human eye matching function) and other human eye color response spaces (such as the response space calculated by color adaptation CAT02 in the color appearance model CIECAM02), which is beneficial for subsequent white balance processing.

[0035] A fourth aspect of this application provides an image processing apparatus that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing apparatus includes: The first acquisition unit is used to acquire a first image to be processed through a color camera; The second acquisition unit is used to acquire first environmental spectral information through a multispectral sensor. The first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The processing unit is used to perform a first processing on the first image to be processed to obtain a first target image. The first processing includes white balance processing based on white balance gain.

[0036] Optionally, in one possible implementation of the fourth aspect, the first image to be processed and the first target image are solid color images or color images with a large area of ​​solid color.

[0037] Optionally, in one possible implementation of the fourth aspect, the third acquisition unit described above is specifically used to input the first environmental spectral information and the first image to be processed into a trained neural network to obtain white balance gain; the trained neural network is obtained by training the neural network with training data as input and aiming to make the loss function value less than a threshold. The training data includes the original training image and training spectral information, the original training image and the training spectral information correspond to the same shooting scene, the output of the neural network includes white balance gain, the loss function is used to indicate the difference between the white balance gain output by the neural network and the actual white balance gain, and the actual white balance gain is obtained by processing the response value of the gray card in the shooting scene.

[0038] Alternatively, in one possible implementation of the fourth aspect, the aforementioned device further includes: The fourth acquisition unit is used to acquire multiple spectral response functions of the color camera; The fourth acquisition unit is also used to acquire multiple estimated values ​​based on the first environmental spectral information and multiple spectral response functions; The fourth acquisition unit is also used to calculate multiple compensation values ​​based on multiple estimates; The processing unit is also used for color shading processing based on multiple compensation values.

[0039] Alternatively, in one possible implementation of the fourth aspect, the aforementioned device further includes: The fifth acquisition unit is used to acquire the tristimulus value curve and the reflectance of the color card; The fifth acquisition unit is also used to acquire a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves; The processing unit is also used for color space conversion processing based on the color correction matrix.

[0040] Optionally, in one possible implementation of the fourth aspect, the fifth acquisition unit described above is specifically used to convert the first environmental spectral information into a light source curve; The fifth acquisition unit is specifically used to acquire the first response value of the color card to the color camera based on multiple spectral response functions, light source curves, and reflectance. The fifth acquisition unit is specifically used to acquire the second response value of the color card to the first human eye color space based on the tristimulus value curve, the light source curve and the reflectance. The first human eye color space is the response space corresponding to the human eye matching function. The fifth acquisition unit is specifically used to acquire a color correction matrix based on the first response value and the second response value. The color correction matrix is ​​used to represent the correlation between the first response value and the second response value.

[0041] Optionally, in one possible implementation of the fourth aspect, the aforementioned processing unit is further configured to perform post-processing on the image after white balance processing to obtain the first target image.

[0042] Alternatively, in one possible implementation of the fourth aspect, the aforementioned device further includes: The display unit is used to display the first target image to the user.

[0043] Optionally, in one possible implementation of the fourth aspect, the first acquisition unit described above is further configured to acquire a second image to be processed via a color camera; The second acquisition unit is also used to acquire second environmental spectral information through a multispectral sensor, wherein the second environmental spectral information corresponds to the same shooting scene as the second image to be processed; The equipment also includes: The determining unit is used to determine the filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information; The filtering unit is used to filter the first target image and the second image to be processed based on the filtering parameters to obtain the correction parameters; The processing unit is also used to adjust the second image to be processed based on the correction parameters to obtain the second target image.

[0044] Alternatively, in one possible implementation of the fourth aspect, the aforementioned determining unit is specifically used to generate a filter intensity function based on similarity; The determination unit is specifically used to determine the filtering parameters based on the filtering intensity function.

[0045] A fifth aspect of this application provides an image processing apparatus that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing apparatus includes: The first acquisition unit is used to acquire a first image to be processed through a color camera; The second acquisition unit is used to acquire first environmental spectral information through a multispectral sensor. The first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The third acquisition unit is used to acquire multiple spectral response functions of the color camera; The third acquisition unit is also used to acquire multiple compensation values ​​based on the first environmental spectral information and multiple spectral response functions; The processing unit is used to perform a first processing on the first image to be processed to obtain a first target image. The first processing includes color shading processing based on multiple compensation values.

[0046] A sixth aspect of this application provides an image processing apparatus that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing apparatus includes: The first acquisition unit is used to acquire a first image to be processed through a color camera; The second acquisition unit is used to acquire first environmental spectral information through a multispectral sensor. The first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The third acquisition unit is used to acquire multiple spectral response functions of the color camera; The third acquisition unit is also used to acquire the tristimulus value curve and the reflectance of the color card; The third acquisition unit is also used to acquire a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves; The processing unit is used to perform a first processing on the first image to be processed to obtain a first target image. The first processing includes color space conversion processing based on the color correction matrix.

[0047] Optionally, in one possible implementation of the sixth aspect, the third acquisition unit described above is specifically used to convert the first environmental spectral information into a light source curve; The third acquisition unit is specifically used to acquire the first response value of the color card to the color camera based on multiple spectral response functions, light source curves, and reflectance. The third acquisition unit is specifically used to acquire the second response value of the color card to the first human eye color space based on the tristimulus value curve, the light source curve, and the reflectance. The first human eye color space is the response space corresponding to the human eye matching function. The third acquisition unit is specifically used to acquire a color correction matrix based on the first response value and the second response value. The color correction matrix is ​​used to represent the conversion relationship between the first response value and the second response value.

[0048] Optionally, in one possible implementation of the sixth aspect, the aforementioned processing unit is further configured to adjust the image after color space conversion based on the conversion relationship between the first human eye color space and the second human eye color space, wherein the second human eye color space is the response space corresponding to the color appearance model when performing color adaptation.

[0049] A seventh aspect of this application provides an image processing device that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing device includes a color camera, a multispectral sensor, and an image processor. A color camera is used to acquire the first image to be processed. A multispectral sensor is used to acquire first environmental spectral information, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed; An image processor is used to obtain a white balance gain based on a first image to be processed and first environmental spectral information; and to perform a first processing on the first image to be processed to obtain a first target image, wherein the first processing includes white balance processing based on the white balance gain.

[0050] An eighth aspect of this application provides an image processing device that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing device includes a color camera, a multispectral sensor, and an image processor. A color camera is used to acquire the first image to be processed. A multispectral sensor is used to acquire first environmental spectral information, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed; An image processor is used to acquire multiple spectral response functions from a color camera; The image processor is also used to obtain multiple compensation values ​​based on the first environmental spectral information and multiple spectral response functions; The image processor is further configured to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color shading processing based on multiple compensation values.

[0051] A ninth aspect of this application provides an image processing device that can be applied to color processing scenarios such as white balance, color reproduction, and color uniformity in images. The image processing device includes a color camera, a multispectral sensor, and an image processor. A color camera is used to acquire the first image to be processed. A multispectral sensor is used to acquire first environmental spectral information corresponding to a first image to be processed; An image processor is used to acquire multiple spectral response functions from a color camera; The image processor is also used to obtain tristimulus value curves and the reflectance of color cards; The image processor is also used to obtain a color correction matrix based on first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves; The image processor is further configured to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color space conversion processing based on a color correction matrix.

[0052] The tenth aspect of this application provides an image processing apparatus that performs the method in the first aspect or any possible implementation thereof, or performs the method in the second aspect or any possible implementation thereof, or performs the method in the third aspect or any possible implementation thereof.

[0053] The eleventh aspect of this application provides an image processing apparatus, including: a processor coupled to a memory, the memory being used to store programs or instructions, which, when executed by the processor, cause the image processing apparatus to implement the methods of the first aspect or any possible implementation thereof, or cause the image processing apparatus to implement the methods of the second aspect or any possible implementation thereof, or cause the image processing apparatus to implement the methods of the third aspect or any possible implementation thereof.

[0054] The twelfth aspect of this application provides a computer-readable medium having a computer program or instructions stored thereon, which, when run on a computer, cause the computer to perform the method of the first aspect or any possible implementation thereof, or cause the computer to perform the method of the second aspect or any possible implementation thereof, or cause the computer to perform the method of the third aspect or any possible implementation thereof.

[0055] The thirteenth aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the methods in the first aspect or any possible implementation of the first aspect, the second aspect or any possible implementation of the second aspect, or the third aspect or any possible implementation of the third aspect.

[0056] The technical effects of aspects four, seven, ten, eleven, twelfth, and thirteen, or any of their possible implementations, can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.

[0057] The technical effects of aspects five, eight, ten, eleven, twelfth, and thirteen, or any one of their possible implementations, can be found in aspect two or different possible implementations of aspect two, and will not be repeated here.

[0058] The technical effects of aspects six, nine, ten, eleven, twelfth, and thirteen, or any one of their possible implementations, can be found in aspect two or different possible implementations of aspect two, and will not be repeated here.

[0059] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. Furthermore, compared with the existing method of estimating the light source, the adjustment quality of the target image can be improved. Attached Figure Description

[0060] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of the present invention; Figure 2 and Figure 3 Two example diagrams of the first environmental spectral information provided in the embodiments of this application; Figure 4 An example image of the first image to be processed provided in an embodiment of this application; Figure 5 An example image of a white-balanced image provided in an embodiment of this application; Figure 6 An example image showing the image before and after color uniformity processing provided in an embodiment of this application; Figure 7 Another example image of the image before color space conversion processing provided in this application embodiment; Figure 8 An example image of an image after color space conversion processing provided in an embodiment of this application; Figure 9 This is another schematic flowchart of the image processing method provided in an embodiment of the present invention; Figure 10 This is another schematic flowchart of the image processing method provided in an embodiment of the present invention; Figures 11-14 These are some structural example diagrams of the image processing device in the embodiments of this application. Detailed Implementation

[0061] This application provides an image processing method and related apparatus. By introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image can be adjusted in real time. Furthermore, since the environmental spectral information corresponding to the image to be processed is collected, the adjustment quality of the target image can be improved compared to the existing method of estimating the light source.

[0062] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0063] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of this application will be introduced below.

[0064] 1. White Balance

[0065] White balance, simply put, means "to reproduce white objects as white under any light source." It compensates for color casts that occur when shooting under specific light sources by enhancing the corresponding complementary colors. If white objects are reproduced as white, then the images of other objects will approximate the color perception habits of the human eye. The "balance" in white balance can be understood as correcting the color differences caused by different color temperatures, thus making white objects appear truly white.

[0066] 2. Color unevenness

[0067] Color shading refers to the uneven distribution of color across a single plane. For example, when taking photos with a mobile phone camera, the center of the image often appears reddish, and there are dark corners. The core reason is that the limited space in a mobile phone forces trade-offs in the optical system design. Parallel light, after passing through a convex lens, focuses after a certain distance. While cameras, due to their ample space, can have a long focal length, mobile phones must minimize the focal length, focusing the light very close behind the lens. Although both focusing methods achieve the goal of forming an image on the sensor, the results are vastly different. Furthermore, because cameras have different refractive indices for different wavelengths of light, the direction of light passing through the lens will differ. With a very short focal length, the scattered light from the periphery cannot completely overlap due to premature focusing, resulting in a greater amount of light in the center and less around the edges. This is the root cause of the reddish center in mobile phone photos mentioned earlier, the color shading phenomenon.

[0068] Currently, the most common color processing method in the industry is to calibrate multiple light sources in offline scenarios to obtain correction parameters under different light sources, and then adjust the original image captured by the color camera according to the correction parameters to obtain the target image to be displayed to the user.

[0069] However, the above-mentioned multiple light source calibration methods in offline scenarios can be understood as estimating the light source. The correction parameters obtained in this way are not accurate enough, which affects subsequent color processing.

[0070] To address the aforementioned issues, this application provides an image processing method that uses a multispectral sensor to acquire environmental spectral information corresponding to the image to be processed, enabling real-time adjustments to the image. Furthermore, since the method uses acquired environmental spectral information corresponding to the image to be processed, it improves the adjustment quality of the target image compared to the prior art's method of estimating the light source.

[0071] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0072] The image processing method provided in this application can be applied to color processing scenarios such as white balance, color uniformity, and color reproduction.

[0073] Please see Figure 1 This application provides an embodiment of an image processing method that can be applied to an image processing device, which includes a color camera and a multispectral sensor. This embodiment includes steps 101 to 104. Figure 1 The illustrated embodiment can be understood as performing white balance processing on the image to be processed.

[0074] Step 101: Acquire the first image to be processed using a color camera.

[0075] The color camera in this embodiment can be understood as an RGB sensor capable of capturing the colors of a scene and taking color photos. Specifically, the color camera can be a monocular or binocular camera, positioned at the front (front-facing camera) or rear (rear-facing camera) of the image processing device's main body. Furthermore, the color camera can be an ultra-wide-angle color camera, a wide-angle color camera, or a telephoto color camera, etc., and is not specifically limited here.

[0076] A first image to be processed is acquired using a color camera, which may be a raw RAW image captured by the color camera.

[0077] The color camera in this embodiment is used to capture color images or solid color images. The specific structure of the color camera is not limited here.

[0078] Optionally, the first image to be processed is a solid color image (or monochrome image) or an image with a large area of ​​a single color.

[0079] Optionally, the first image to be processed can be a raw RAW domain image (also called a RAW image). A RAW image can be the raw data from a complementary metal-oxide semiconductor (CMOS) or charge-coupled device (CCD) image sensor, where the light source signal captured by the camera is converted into a digital signal. This raw data has not yet been processed by an image signal processor (ISP). Specifically, the RAW image can be a Bayer image in Bayer format.

[0080] Step 102: Obtain first environmental spectral information through a multispectral sensor.

[0081] The multispectral sensor in this embodiment is used to collect spectra. A spectrum (or optical spectrum) can be understood as a pattern in which monochromatic light, after being dispersed by a dispersion system (such as a prism or grating), is arranged sequentially according to wavelength or frequency.

[0082] For example, a multispectral sensor can acquire spectra in the 350-1000 nm wavelength range with a field of view (FOV) of ±35 degrees. A multispectral sensor may also include eight visible light bands and multiple special bands (e.g., a full-spectrum channel, a scintillation frequency detection channel, and / or an infrared channel), or ten visible light bands and multiple special bands. It is understood that the number of visible light bands mentioned above is merely an example; in practical applications, there may be fewer or more visible light bands. This article only uses eight visible light bands as an example for illustrative purposes.

[0083] The first environmental spectral information is obtained by a multispectral sensor, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed.

[0084] The same shooting scene in the embodiments of this application can be understood as satisfying at least one of the following properties: 1. The same shooting scene can refer to the distance between the position of the color camera when acquiring the first image to be processed and the position of the multispectral sensor when acquiring the first environmental spectral information being less than a certain threshold (for example, if the distance between the position of the color camera when acquiring the first image to be processed and the position of the multispectral sensor when acquiring the first environmental spectral information is 1 meter and the threshold is 2 meters, that is, if the distance is less than the threshold, then the first image to be processed and the first environmental spectral information can be determined to be the same shooting scene).

[0085] The location mentioned above can be a relative location or a geographic location. If the location is a relative location, it can be determined by establishing a scene model or other means. If the location is a geographic location, it can be the location of the first device and the location of the second device determined by the Global Positioning System (GPS) or the Beidou Navigation System, and then the distance between the two locations can be obtained.

[0086] 2. The same shooting scene can also be judged based on light intensity. For example, the similarity between the weather type when the first image to be processed was acquired and the weather type when the first environmental spectral information was acquired can be used to determine whether the first environmental spectral information and the first image to be processed belong to the same shooting scene. For example, if both the first image to be processed and the first environmental spectral information were acquired on a sunny day, then it can be determined that the first environmental spectral information and the first image to be processed belong to the same shooting scene. If both the first image to be processed and the first environmental spectral information were acquired on a sunny day, then it can be determined that the first environmental spectral information and the first image to be processed do not belong to the same shooting scene.

[0087] It is understandable that the above-mentioned determination that the first image to be processed and the first environmental spectral information are from the same shooting scene is just an example. In actual applications, there may be other methods, which are not limited here.

[0088] The first environmental spectral information can be either the light source spectrum or the reflection spectrum. In other words, the light source spectrum is the spectrum corresponding to the light source illuminating the first image to be processed, and the reflection spectrum is the spectrum corresponding to the light reflected from the object in the first image to be processed.

[0089] In addition, the first environmental spectral information can be sampling points of a multispectral sensor or environmental spectral maps, etc., which can characterize the environmental spectrum. The number of sampling points (or the number of channels of the multispectral sensor) is related to the design of the multispectral sensor (e.g., the number of visible light bands, special bands, etc.), and can be 8, 10, or even fewer or more sampling points. Continuing the above example, this application embodiment only describes the example of a multispectral sensor collecting 8 sampling points.

[0090] For example, Figure 2 and Figure 3 Here are two examples of the first environmental spectral information. It can be understood that the first environmental information can be an eight-dimensional array, such as (color temperature, light intensity).

[0091] Step 103: Obtain white balance gain based on the first image to be processed and the first environmental spectral information.

[0092] In this application embodiment, there are multiple ways to obtain white balance gain based on the first image to be processed and the first environmental spectral information. It can be based on obtaining the light source white point or obtaining the white balance gain based on the first environmental spectral information. Here, the light source white point can be understood as 1 / white balance gain, and the white balance gain can be understood as red gain (Rgain) and blue gain (Bgain).

[0093] It is understandable that Rgain and Bgain, or the white point of the light source, can be obtained through grayscale world algorithms, total internal reflection algorithms, or input neural networks. Of course, in a solid color scene, grayscale world algorithms can be avoided to prevent white balance failure caused by the assumption of grayscale world (i.e., for an image with a large number of color changes, the average value of the three components RGB tends to the same grayscale value).

[0094] Optionally, the description is taken as an example using a neural network: the first image to be processed is 16 Size 16. This allows upsampling of values ​​from 8 sampling points to 8. Size 8, and downsample the first image to be processed to 8. The image is set to size 8. The upsampled 8 sampling points and the downsampled first image to be processed are then input into the neural network to obtain Rgain and Bgain. Alternatively, the upsampling or downsampling steps can be performed after the neural network, using the first environmental spectral information and the first image to be processed as input. The neural network can be a deep neural network, convolutional neural network, etc., and is not specifically limited here. The trained neural network is obtained by training the network with training data as input, aiming for a loss function value less than a threshold. The training data includes the original training image and training spectral information, corresponding to the same shooting scene. The neural network output includes white balance gain. The loss function indicates the difference between the white balance gain output by the neural network and the actual white balance gain. The actual white balance gain is obtained by processing the response value of the gray card in the shooting scene. Since the RGB channel values ​​of the gray card are equal or approximately equal, the response value of the gray card in the shooting scene is helpful in determining the white balance gain.

[0095] Step 104: Perform a first processing on the first image to be processed to obtain the first target image.

[0096] After obtaining the white balance gain, the first image to be processed is subjected to a first processing step to obtain the first target image. This first processing step includes white balance processing based on the white balance gain.

[0097] Optionally, Rgain is multiplied by the value of the red channel in the first image to be processed, and Bgain is multiplied by the value of the blue channel in the first image to be processed to obtain the adjusted values ​​of each channel, thereby realizing the white balance processing of the first image to be processed.

[0098] Alternatively, the adjustment can be done by directly multiplying Rgain and Bgain by the pixel values ​​in the first image to be processed. Another approach is to adjust multiple compensation values ​​based on RGGB and then multiply them by the pixel values ​​in the Bayer domain of the first image to be processed; the specific method is not limited here.

[0099] For example, Figure 4 This is an example image of the first image to be processed. Figure 5 This is an example image of the first target.

[0100] The first processing in this embodiment includes white balance processing based on white balance gain, and may also include, but is not limited to, one or more of the following post-processing algorithms: automatic exposure control (AEC), automatic gain control (AGC), color correction, lens correction, noise removal / reduction, bad pixel removal, linear correction, color interpolation, image downsampling, level compensation, etc. Furthermore, in some instances, it may also include image enhancement algorithms, such as gamma correction, contrast enhancement and sharpening, color noise removal and edge enhancement in the YUV color space, color enhancement, color space conversion (e.g., RGB to YUV), etc. The first target image is, for example, an image in YUV or RGB format.

[0101] Optionally, after acquiring the first target image, the first target image can be displayed to the user.

[0102] Optionally, the image processing apparatus further includes an image processor, which is used to perform steps 103 and 104.

[0103] In this embodiment, on the one hand, by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. On the other hand, since it contains the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. Furthermore, for scenes with solid color images (i.e., the image scene has limited color or contains large areas of monochromatic objects), compared to the method of using the grayscale world algorithm for white balance in the prior art, the method of performing white balance on solid color images using the first environmental spectral information collected by the multispectral sensor can improve the adjustment quality of the target image.

[0104] In one possible implementation, the first processing also includes any combination (or at least one) of various color processing methods such as color restoration and color uniformity, which are described below.

[0105] The first method is color shading.

[0106] In this embodiment, color uniformity processing can be performed before or after white balance processing. If color uniformity processing is performed before white balance processing, the object of color uniformity processing is the first image to be processed. If color uniformity processing is performed after white balance processing, the object of color uniformity processing is the first image to be processed after white balance processing. The following description only uses color uniformity processing on the first image to be processed as an example. Of course, color uniformity processing can also be performed on images after white balance processing or other color processing methods; the specific method is not limited here.

[0107] In this case Figure 1 The steps of the illustrated embodiment may further include: acquiring multiple spectral response functions of the color camera; acquiring multiple estimated values ​​based on the first environmental spectral information and the multiple spectral response functions; calculating multiple compensation values ​​based on the multiple estimated values; and the first processing further includes: color shading processing based on the multiple compensation values. The number of compensation values ​​may correspond one-to-one with the number of pixels in the first image to be processed, or the number of compensation values ​​may be less than the number of pixels in the first image to be processed (it can also be understood that one compensation value corresponds to one region, and one region includes multiple pixels in the first image to be processed). Specifically, the above step of acquiring multiple spectral response functions of the color camera may be done by measuring the spectral response of the pixel positions of the color camera using a monochromator to obtain multiple spectral response functions. Alternatively, the response function of the color camera may be determined by adjusting different light intensities of the light source offline. The above step of acquiring multiple estimated values ​​based on the first environmental spectral information and the multiple spectral response functions may be done by upsampling the values ​​of 8 sampling points and integrating the upsampled values ​​with the multiple spectral response functions to obtain multiple estimated values. The steps for obtaining multiple compensation values ​​described above can be as follows: Using the center pixel of the first image to be processed as a reference, obtain the compensation values ​​for all pixels in the first image to be processed except for the center pixel. Then, upsample the size of the compensation values ​​to the spatial size of the camera image. Finally, use the upsampled compensation values ​​for color shading processing.

[0108] Alternatively, color shading can be achieved by directly multiplying multiple compensation values ​​by the number of pixels in the first image to be processed. Another approach is to adjust multiple compensation values ​​according to RGGB, and then multiply them by the number of pixels in the first image to be processed in the Bayer domain to complete the color shading process. The specific method is not limited here.

[0109] For example, continuing the previous example, 8 multispectral sample values ​​are upsampled to obtain 256 multispectral sample values. Multiple spectral response functions are 256 (the vertical dimension of the image). 256 (image's horizontal dimension) 3 (pixel channels) 256 (number of multispectral channels) are used to obtain multiple estimated values ​​by integrating multiple upsampled values ​​with multiple spectral response functions using the following formula. The compensation values ​​for pixels other than the center pixel in the first image to be processed are obtained using Formula 2, with the center pixel as the reference. The size of the compensation values ​​is upsampled to the spatial size of the camera image using Formula 3. Color shading is then performed using the upsampled compensation values ​​using Formula 4.

[0110] Formula 1: .

[0111] in, There are multiple estimates, which can be 256 (the vertical dimension of the image). 256 (image's horizontal dimension) 3 (pixel channels). The 8 multispectral samples were upsampled to obtain 256 sample values. These are multiple spectral response functions corresponding to x and y. x and y are the spatial dimensions of the color camera's spectral response, for example, 256. 256. c represents the pixel channel of the color camera. Here, we only use 3 pixel channels as an example for description. It can be understood that in practical applications, the number of pixel channels of a color camera can be more, but this is not limited here. It is the response wavelength of a color camera. Generally, the response wavelength ranges from 380 nanometers (nm) to 780 nanometers (nm), which is the wavelength range of visible light to the human eye.

[0112] Formula 2: .

[0113] Formula 3: .

[0114] Formula 4: .

[0115] in, These are multiple compensation values ​​corresponding to x and y. It is the center pixel of the first image to be processed. Used to indicate upsampling of x and y. It involves upsampling x and y to obtain multiple compensation values ​​corresponding to the camera image size. , It is the spatial size of the camera image, for example, 3000. 4000. It refers to the vertical and horizontal dimensions of the image after color uniformity processing, as well as the number of pixel channels. It is the vertical dimension, horizontal dimension, and pixel channels of the first image to be processed.

[0116] It is understood that Formula 1, Formula 2, Formula 3 and Formula 4 above are just examples. In practical applications, there may be other forms of Formula 1, Formula 2, Formula 3 and Formula 4, which are not limited here.

[0117] For example, Figure 6 This is an example image showing the first image to be processed (i.e., the image before color shading) and the image after color uniform processing (i.e., the image after color shading).

[0118] This method, compared to existing technologies that require offline calibration for image color uniformity, enables real-time calculation and avoids potential errors caused by offline table selection. Furthermore, the color uniformity compensation value generated from the first environmental spectral information collected by a multispectral sensor improves the quality of color uniformity compared to offline calibration in existing technologies.

[0119] The second method is color restoration processing (also known as color space conversion processing).

[0120] In this embodiment, the color restoration processing is not sequentially related to the aforementioned white balance processing and color uniformity processing. The following explanation uses the example of the first processing including color restoration and white balance processing. If the color restoration processing precedes the white balance processing, the object of the color restoration processing is the first image to be processed. If the color restoration processing follows the white balance processing, the object of the color restoration processing is the first image to be processed after white balance processing. The following description uses color uniformity processing on the first image to be processed as an example; however, color uniformity processing can also be performed on images after white balance processing or other color processing methods, and is not limited here.

[0121] In this case Figure 1 The steps of the illustrated embodiment may further include: obtaining the tristimulus value curve and the reflectance of the color chart; obtaining a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and the tristimulus value curve; and the first processing further includes: color space conversion processing based on the color correction matrix.

[0122] The above steps, specifically obtaining the color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves, may include: converting the first environmental spectral information into a light source curve; obtaining the first response value of the color card to the color camera based on multiple spectral response functions, the light source curve, and reflectance (which can also be understood as the image of the color card in a space composed of RGB as the three axes); obtaining the second response value of the color card to the first human eye color space based on the tristimulus value curve, the light source curve, and reflectance; and obtaining the color correction matrix based on the first and second response values. The first human eye color space can be the response space corresponding to the human eye matching function. The color correction matrix is ​​used to represent the correlation between the first and second response values.

[0123] This color correction matrix can also be understood as a transformation matrix between two color spaces. Optionally, this color space transformation matrix is ​​a 3x3 matrix. This is equivalent to using one color space as the target and the other as the source, and then using the least squares method to obtain the transformation matrix.

[0124] It is understandable that the reflectance of the color chart mentioned above can be the reflectance of a standard 24-color chart, or it can be replaced with a regular rectangular wave, a custom curve, etc., without any specific restrictions here.

[0125] Optionally, the eye-matching function can be the eye-matching function under the International Commission on Illumination (CIE) 1931 or other standards. The tristimulus curves can be tristimulus curves under CIE 1931 or other standards. Specific limitations are not specified here.

[0126] For example, the first response value of the color card to the color camera can be obtained using Formula 5, multiple spectral response functions, light source curves, and reflectivity. The second response value of the color card to the first human eye color space can be obtained using Formula 6, tristimulus curves, light source curves, and reflectivity.

[0127] Formula 5: .

[0128] Formula Six: .

[0129] in, These are the response curves corresponding to multiple spectral response functions. It is the light source curve. It is the reflectance of the color chart. It is a tristimulus value curve.

[0130] It is understandable that Formulas 5 and 6 above are just examples. In practical applications, there may be other forms of Formulas 5 and 6, which are not limited here.

[0131] For example, Figure 7 Example image of an image before color space conversion. Figure 8 This is an example image that has undergone color space conversion.

[0132] Optionally, after obtaining the first human-eye color space, the image after color space conversion can be adjusted according to the conversion relationship between the first and second human-eye color spaces. This second human-eye color space is the response space corresponding to the color appearance model during color adaptation, which helps improve the quality of white balance subsequently.

[0133] This method can be understood as adjusting the third target image based on the conversion relationship between the human eye color response space (the response space constructed by the CIE1931 human eye matching function) and other human eye color response spaces (such as the response space calculated by color adaptation CAT02 in the color appearance model CIECAM02).

[0134] This method, compared to existing technologies that require offline calibration for image color restoration, enables real-time calculation and avoids potential errors caused by offline table selection. Furthermore, since the color space transformation matrix is ​​generated from the first environmental spectral information collected by a multispectral sensor, the quality of color restoration is improved compared to offline calibration in existing technologies.

[0135] The third method is time-domain stability processing.

[0136] In this case Figure 1 The steps of the illustrated embodiment may further include: acquiring a second image to be processed via a color camera; acquiring second environmental spectral information corresponding to the second image to be processed via a multispectral sensor; determining filtering parameters based on the similarity between the first and second environmental spectral information; filtering the first target image and the second image to be processed based on the filtering parameters to obtain correction parameters; and adjusting the second image to be processed based on the correction parameters to obtain the second target image.

[0137] Specifically, the step described above, determining the filtering parameters based on the similarity between the first and second environmental spectral information, may include: generating a filtering intensity function based on the similarity; and determining the filtering parameters based on the filtering intensity function.

[0138] Optionally, the time interval between the acquisition of the first image to be processed and the second image to be processed by the color camera is less than a preset time interval. Further, the first image to be processed and the second image to be processed are two frames captured by the color camera at adjacent moments.

[0139] The specific method for obtaining the similarity between the first and second environmental spectral information can be as follows: A first spectral curve is determined using multiple sampled values ​​corresponding to the first environmental spectral information, and a second spectral curve is determined using multiple sampled values ​​corresponding to the second environmental spectral information. The similarity between the first and second spectral curves is calculated using a curve similarity algorithm (e.g., cosine similarity). A filter intensity function is generated based on the similarity, where the similarity and filter intensity are positively correlated; that is, the greater the similarity, the stronger the filter intensity.

[0140] For example, this description uses a three-segment filter intensity function as an example. It can be understood that this filter intensity function can be set as a first-order or higher-order function; no specific limitation is made here. An example of a three-segment filter intensity function is as follows: First paragraph: If the similarity is greater than the first threshold, the filter weight is 1, that is, the above correction parameters are used. In other words, the correction parameters used in the first process (e.g., Rgain and Bgain, white point, estimated value, color correction matrix mentioned above). The second paragraph states that if the similarity is less than or equal to the first threshold and greater than or equal to the second threshold, the filter weight is less than 1 and greater than 0. Linear interpolation is performed within the range of 0-1 based on the difference between the similarity and the first threshold. The closer the similarity is to the first threshold, the closer the filter weight is to 1; the farther the similarity is from the first threshold, the closer the filter weight is to 0.

[0141] The third paragraph: If the similarity is less than the second threshold, the filter weight is 0, that is, the correction parameters of the second image to be processed are recalculated (the solution method is similar to the first processing, which is called the second processing here. The only difference between the first processing and the second processing is that the first environmental spectral information of the first processing is replaced with the second environmental spectral information in the second processing, and the first image to be processed is replaced with the second image to be processed).

[0142] It is understandable that the first and second thresholds mentioned above can be set according to actual needs, and no specific restrictions are imposed here. For example, the first threshold could be 90%, and the second threshold could be 10%.

[0143] Optionally, the correction parameters obtained by filtering the first target image and the second image to be processed based on the filtering weights can be Rgain and Bgain used in white balance, color correction matrix in color restoration, or estimated values ​​in color uniformity; no specific limitation is made here. After obtaining the correction parameters, the second image to be processed is adjusted using the correction parameters to obtain the second target image (the adjustment method is similar to that described above and will not be repeated here).

[0144] For example, the first threshold is 90%, the second threshold is 10%, and the similarity is 70%, which means the correction parameters are determined using the second segment of the filtering function. The determination of the correction parameters for the second image to be processed (here, the estimated value of color uniformity) is described using Equation 7 as an example.

[0145] Formula 7: .

[0146] in, It is the estimated value related to the first image to be processed in the above color restoration (or it can be understood as the estimated value of the first target image), for example, the filter weight is 0.5. It is the estimated value related to the second image to be processed in the above color restoration (the solution method is similar to the above color restoration, except that the first environmental spectral information is replaced with the second environmental spectral information, and the first image to be processed is replaced with the second image to be processed).

[0147] In other words, if the difference between the first and second environmental spectral information is small, historical correction parameters can be used, or the historical correction parameters (i.e., correction parameters obtained by comparing the first environmental spectral information with the color channels in the first image to be processed, or the correction parameters used in the first processing) can be given a larger weight, while the new correction parameters (i.e., correction parameters obtained by comparing the second environmental spectral information with the second image to be processed, or the correction parameters used in the second processing) can be given a smaller weight, thus obtaining the correction parameters for the second image to be processed. If the difference between the first and second environmental spectral information is large (e.g., the difference between indoor and outdoor environments), new correction parameters can be used, or the new correction parameters can be given a larger weight, while the historical correction parameters can be given a smaller weight, thus obtaining the correction parameters for the second image to be processed.

[0148] This approach achieves both improved temporal stability of color processing and enhanced sensitivity, avoiding flickering of color effects in the temporal domain while responding promptly to changes in the environment that lead to parameter adjustments.

[0149] It is understandable that the above three color processing methods can be combined arbitrarily, and no specific restrictions are imposed here.

[0150] Please see Figure 9 Another embodiment of the image processing method provided in this application can be applied to an image processing device, which includes a color camera and a multispectral sensor. This embodiment includes steps 901 to 905. Figure 9 The illustrated embodiment can be understood as performing color uniform processing on the image to be processed.

[0151] Step 901: Acquire the first image to be processed using a color camera.

[0152] Step 902: Obtain first environmental spectral information through a multispectral sensor.

[0153] Steps 901 and 902 in this implementation are the same as those described above. Figure 1 Step 101 in the illustrated embodiment is similar to step 102, and will not be described again here.

[0154] Step 903: Obtain multiple spectral response functions of the color camera.

[0155] One specific way to obtain multiple spectral response functions of a color camera is to measure the spectral response of the pixel positions of the color camera using a monochromator. Alternatively, after determining the color camera, its photosensitivity properties can be measured. For example, the response functions of the color camera can be adjusted offline under different light intensities from different light sources.

[0156] Step 904: Obtain multiple compensation values ​​based on the first environmental spectral information and multiple spectral response functions.

[0157] The compensation values ​​for all pixels in the first image to be processed, excluding the center pixel, are obtained based on the center pixel. The number of compensation values ​​can correspond one-to-one with the number of pixels in the first image to be processed, or the number of compensation values ​​can be less than the number of pixels in the first image to be processed (it can also be understood that one compensation value corresponds to one region, and one region includes multiple pixels in the first image to be processed).

[0158] Step 905: Perform a first processing on the first image to be processed to obtain a first target image.

[0159] After obtaining the compensation values, the first image to be processed is subjected to a first processing step to obtain the first target image. This first processing step includes color shading based on multiple compensation values.

[0160] The first processing in this embodiment includes color shading based on multiple compensation values, and may also include, but is not limited to, one or more of the following post-processing algorithms: automatic exposure control (AEC), automatic gain control (AGC), color correction, lens correction, noise removal / reduction, bad pixel removal, linear correction, color interpolation, image downsampling, level compensation, etc. Furthermore, in some instances, it may include image enhancement algorithms such as gamma correction, contrast enhancement and sharpening, color noise removal and edge enhancement in the YUV color space, white balance, color space conversion (e.g., RGB to YUV), etc. The first target image is, for example, an image in YUV or RGB format.

[0161] Optionally, after acquiring the first target image, the first target image can be displayed to the user.

[0162] Optionally, the image processing apparatus further includes an image processor, which is used to perform steps 903 and 904.

[0163] The compensation value can be upsampled to the spatial size of the camera image. The upsampled compensation value is then used to adjust the first image to be processed.

[0164] For a detailed description, please refer to the description of the first type of uniform color in the foregoing embodiments, which will not be repeated here.

[0165] It is understood that, based on this embodiment, the first target image can also be subjected to color processing such as white balance and color restoration. The processing method can be referred to the description in the foregoing embodiment, and will not be repeated here.

[0166] In this embodiment, on the one hand, by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. On the other hand, since it contains the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. Compared to the prior art that requires offline calibration to restore the color of the image, real-time calculation can be achieved, and the problems caused by possible errors in offline table selection can be avoided. In addition, since the color uniformity compensation value generated by the first environmental spectral information collected by the multispectral sensor can improve the quality of color uniformity compared to the offline calibration in the prior art.

[0167] Please see Figure 10Another embodiment of the image processing method provided in this application can be applied to an image processing device, which includes a color camera and a multispectral sensor. This embodiment includes steps 1001 to 1006. Figure 10 The illustrated embodiment can be understood as performing color restoration processing on the image to be processed.

[0168] Step 1001: Acquire the first image to be processed using a color camera.

[0169] Step 1002: Obtain first environmental spectral information through a multispectral sensor.

[0170] Steps 1001 and 1002 in this implementation are the same as those described above. Figure 1 Step 101 in the illustrated embodiment is similar to step 102, and will not be described again here.

[0171] Step 1003: Obtain multiple spectral response functions of the color camera.

[0172] One specific way to obtain multiple spectral response functions of a color camera is to measure the spectral response of the pixel positions of the color camera using a monochromator. Alternatively, the response function of the color camera can be determined by adjusting different light intensities of the light source offline.

[0173] Step 1004: Obtain the tristimulus value curve and the reflectance of the color card.

[0174] Optionally, the reflectance of the color chart can be the reflectance of a standard 24-color chart, or it can be replaced with a regular rectangular wave, a custom curve, etc., without any specific limitations here.

[0175] Optionally, the tristimulus value curve can be a tristimulus value curve according to CIE 1931 or other standards. Specific limitations are not specified here.

[0176] Step 1005: Obtain the color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves.

[0177] The first environmental spectral information is converted into a light source curve. Based on multiple spectral response functions, the light source curve, and reflectance, the first response value of the color card to the color camera is obtained (which can also be understood as the image of the color card in a space composed of RGB as the three axes). Based on the tristimulus value curve, the light source curve, and reflectance, the second response value of the color card to the first human eye color space is obtained. The color correction matrix is ​​obtained based on the first and second response values. The aforementioned first human eye color space can be the response space corresponding to the human eye matching function. The color correction matrix is ​​used to represent the correlation between the first and second response values.

[0178] This color correction matrix can also be understood as a transformation matrix between two color spaces. Optionally, this color space transformation matrix is ​​a 3x3 matrix. This is equivalent to using one color space as the target and the other as the source, and then using the least squares method to obtain the transformation matrix.

[0179] Optionally, the eye-matching function can be the eye-matching function under the International Commission on Illumination (CIE) 1931 or other standards. The tristimulus curves can be tristimulus curves under CIE 1931 or other standards. Specific limitations are not specified here.

[0180] For example, the first response value of the color card to the color camera can be obtained using the aforementioned Formula 5, multiple spectral response functions, light source curves, and reflectivity. The second response value of the color card to the first human eye color space can be obtained using the aforementioned Formula 6, tristimulus curves, light source curves, and reflectivity.

[0181] Step 1006: Perform a first processing on the first image to be processed to obtain a first target image.

[0182] After obtaining the color correction matrix, the first image to be processed is subjected to a first processing step to obtain the first target image. This first processing step includes color space conversion based on the color correction matrix.

[0183] The first processing in this embodiment includes color space conversion based on the color correction matrix, and may also include, but is not limited to, one or more of the following post-processing algorithms: automatic exposure control (AEC), automatic gain control (AGC), color correction, lens correction, noise removal / reduction, bad pixel removal, linear correction, color interpolation, image downsampling, level compensation, etc. Furthermore, in some instances, it may also include image enhancement algorithms, such as gamma correction, contrast enhancement and sharpening, color noise removal and edge enhancement in the YUV color space, white balance, color restoration, etc. The first target image is, for example, an image in YUV or RGB format.

[0184] Optionally, after acquiring the first target image, the first target image can be displayed to the user.

[0185] Optionally, the image processing apparatus further includes an image processor, which is used to perform steps 1003 to 1006.

[0186] For example, the color correction matrix is ​​a 3x3 matrix.

[0187] Optionally, after obtaining the first human-eye color space, the image after color space conversion can be adjusted according to the conversion relationship between the first and second human-eye color spaces. This second human-eye color space is the response space corresponding to the color appearance model during color adaptation, which helps improve the quality of white balance subsequently.

[0188] This method can be understood as adjusting the first target image based on the conversion relationship between the human eye color response space (the response space constructed by the CIE1931 human eye matching function) and other human eye color response spaces (such as the response space calculated by color adaptation CAT02 in the color appearance model CIECAM02).

[0189] For a detailed description, please refer to the relevant description of the first color reproduction in the foregoing embodiments, which will not be repeated here.

[0190] It is understood that, based on this embodiment, the image after color restoration can also be subjected to color processing such as white balance and color uniformity. The processing method can be referred to the description in the foregoing embodiment, and will not be repeated here.

[0191] In this embodiment, on the one hand, by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. On the other hand, since it contains the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. Compared to the prior art that requires offline calibration to restore the color of the image, real-time calculation can be achieved, and the problems caused by possible errors in offline table selection can be avoided. In addition, since the color space transformation matrix is ​​generated by the first environmental spectral information collected by the multispectral sensor, the quality of color restoration can be improved compared to the offline calibration in the prior art.

[0192] The image processing method in the embodiments of this application has been described above. The image processing device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 11 One embodiment of the image processing device in this application includes: The first acquisition unit 1101 is used to acquire a first image to be processed through a color camera; The second acquisition unit 1102 is used to acquire first environmental spectral information through a multispectral sensor; The third acquisition unit 1103 is used to acquire white balance gain based on the first image to be processed and the first environmental spectral information. Processing unit 1104 is configured to perform a first processing on the first image to be processed to obtain a first target image. The first processing includes white balance processing based on white balance gain. Optionally, the image processing device may further include the following units: The fourth acquisition unit 1105 is used to acquire multiple spectral response functions of the color camera; The fifth acquisition unit 1106 is used to acquire the tristimulus value curve and the reflectance of the color card; Display unit 1107 is used to display the first target image to the user.

[0193] The determining unit 1108 is used to determine the filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information; The filtering unit 1109 is used to filter the first target image and the second image to be processed based on the filtering parameters to obtain the correction parameters.

[0194] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figure 1 The embodiments shown are similar and will not be repeated here.

[0195] In this embodiment, since it contains the environmental spectral information corresponding to the first image to be processed acquired by the second acquisition unit 1102, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. On the other hand, for scenes with solid color images (i.e., the image scene has limited color or contains large areas of monochromatic objects), compared to the method of using the grayscale world algorithm for white balance in the prior art, the method of white balance for solid color images acquired by the processing unit 1104 through the first environmental spectral information acquired by the multispectral sensor can improve the adjustment quality of the target image.

[0196] Please see Figure 12 Another embodiment of the image processing device in this application includes: a first acquisition unit 1201, a second acquisition unit 1202, a third acquisition unit 1203, and a processing unit 1204.

[0197] In one possible implementation, each unit is specifically used to perform the following functions: The first acquisition unit 1201 is used to acquire a first image to be processed through a color camera; The second acquisition unit 1202 is used to acquire first environmental spectral information through a multispectral sensor; The third acquisition unit 1203 is used to acquire multiple spectral response functions of the color camera; The third acquisition unit 1203 is also used to acquire multiple compensation values ​​based on the first environmental spectral information and multiple spectral response functions; The processing unit 1204 is used to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color shading processing based on multiple compensation values.

[0198] In this possible implementation, the operations performed by each unit in the image processing device are the same as described above. Figure 9 The embodiments shown are similar and will not be repeated here.

[0199] In this possible implementation, since the second acquisition unit 1202 collects the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the existing method of estimating the light source. Compared to the existing method of offline calibration for image color restoration, real-time calculation can be achieved, avoiding problems caused by potential errors in offline table selection. Furthermore, since the third acquisition unit 1203 generates the color uniformity compensation value from the first environmental spectral information collected by the multispectral sensor, the quality of color uniformity can be improved compared to offline calibration in the existing technology.

[0200] In another possible implementation, each unit is specifically used to perform the following functions: The first acquisition unit 1201 is used to acquire a first image to be processed through a color camera; The second acquisition unit 1202 is used to acquire first environmental spectral information through a multispectral sensor; The third acquisition unit 1203 is used to acquire multiple spectral response functions of the color camera; The third acquisition unit 1203 is also used to acquire the tristimulus value curve and the reflectance of the color card; The third acquisition unit 1203 is also used to acquire a color correction matrix based on the first environmental spectral information, multiple spectral response functions, reflectance and tristimulus value curves. The processing unit 1204 is used to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color space conversion processing based on the color correction matrix.

[0201] In this possible implementation, the operations performed by each unit in the image processing device are the same as described above. Figure 10 The embodiments shown are similar and will not be repeated here.

[0202] In this possible implementation, because it includes the environmental spectral information corresponding to the first image to be processed acquired by the second acquisition unit 1202, the adjustment quality of the first target image can be improved compared to the existing method of estimating the light source. Compared to the existing method of offline calibration for image color restoration, real-time calculation can be achieved, avoiding problems caused by potential errors in offline table selection. Furthermore, since the third acquisition unit 1203 generates a color space transformation matrix using the first environmental spectral information acquired by the multispectral sensor, the quality of color restoration can be improved compared to offline calibration in the existing technology.

[0203] Please see Figure 13 Another embodiment of the image processing device in this application includes: a color camera 1301, a multispectral sensor 1302, and an image processor 1303.

[0204] In one possible implementation, each unit is specifically used to perform the following functions: Color camera 1301 is used to acquire the first image to be processed; Multispectral sensor 1302 is used to acquire first environmental spectral information; Image processor 1303 is used to perform a first processing on a first image to be processed to obtain a first target image; wherein the first processing includes white balance processing based on white balance gain.

[0205] In this possible implementation method, for solid color image scenes (i.e., image scenes with limited color or large areas of monochromatic objects), compared to the gray-world algorithm used in the prior art for white balance, the image processor 1303 can improve the adjustment quality of the target image by using the first environmental spectral information collected by the multispectral sensor 1302 to perform white balance on solid color images.

[0206] In another possible implementation, each unit is specifically used to perform the following functions: Color camera 1301 is used to acquire the first image to be processed; Multispectral sensor 1302 is used to acquire first environmental spectral information; Image processor 1303 is used to acquire multiple spectral response functions of a color camera; The image processor 1303 is also used to obtain multiple compensation values ​​based on the first environmental spectral information and multiple spectral response functions; The image processor 1303 is further configured to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color shading processing based on multiple compensation values.

[0207] In this possible implementation, on the one hand, by introducing a multispectral sensor to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. On the other hand, since it contains the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. Compared to the prior art that requires offline calibration to restore the color of the image, real-time calculation can be achieved, and problems caused by possible errors in offline table selection can be avoided. In addition, since the color uniformity compensation value generated by the image processor 1303 through the first environmental spectral information collected by the multispectral sensor 1302 can improve the quality of color uniformity compared to the offline calibration in the prior art.

[0208] In another possible implementation, each unit is specifically used to perform the following functions: Color camera 1301 is used to acquire the first image to be processed; Multispectral sensor 1302 is used to acquire first environmental spectral information; Image processor 1303 is used to acquire multiple spectral response functions of a color camera; The image processor 1303 is also used to acquire the tristimulus value curve and the reflectance of the color card; Image processor 1303 is also used to obtain a color correction matrix based on first environmental spectral information, multiple spectral response functions, reflectance, and tristimulus value curves; The image processor 1303 is further configured to perform a first processing on the first image to be processed to obtain a first target image; wherein the first processing includes color space conversion processing based on a color correction matrix.

[0209] In this possible implementation, on the one hand, by introducing a multispectral sensor 1302 to collect environmental spectral information corresponding to the image to be processed, the image to be processed can be adjusted in real time. On the other hand, since it contains the environmental spectral information corresponding to the first image to be processed, the adjustment quality of the first target image can be improved compared to the method of estimating the light source in the prior art. Compared to the prior art that requires offline calibration to restore the color of the image, real-time calculation can be achieved, and problems caused by possible errors in offline table selection can be avoided. In addition, since the image processor 1303 generates a color space transformation matrix using the first environmental spectral information collected by the multispectral sensor 1302, the quality of color restoration can be improved compared to offline calibration in the prior art.

[0210] Please see Figure 14This application provides another image processing device. For ease of explanation, only the parts related to this application are shown. For specific technical details not disclosed, please refer to the method section of this application. This image processing device can be any image processing device, including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, and in-vehicle computers. Taking a mobile phone as an example: Figure 14 This is a block diagram illustrating a portion of the structure of a mobile phone related to the image processing device provided in an embodiment of this application. (Reference) Figure 14 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a color camera 1451, a multispectral sensor 1452, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 14 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0211] The following is combined Figure 14 A detailed introduction to each component of a mobile phone: RF circuit 1410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 1480; additionally, it transmits uplink data to the base station. Typically, RF circuit 1410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 1410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0212] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0213] The input unit 1430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432. The touch panel 1431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 1431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1480, and can also receive and execute commands sent by the processor 1480. In addition, the touch panel 1431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1431, the input unit 1430 may also include other input devices 1432. Specifically, other input devices 1432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0214] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, a touch panel 1431 may cover the display panel 1441. When the touch panel 1431 detects a touch operation on or near it, it transmits the information to the processor 1480 to determine the type of touch event. Subsequently, the processor 1480 provides corresponding visual output on the display panel 1441 according to the type of touch event. Although in Figure 14 In this embodiment, the touch panel 1431 and the display panel 1441 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1431 and the display panel 1441 can be integrated to realize the input and output functions of the mobile phone.

[0215] The mobile phone may also include a color camera 1451 and a multispectral sensor 1452. The color camera 1451 is specifically used to capture color images or solid color images (or monochrome images). The multispectral sensor 1452 is used to acquire environmental spectral information corresponding to the image. Of course, the mobile phone may also include other types of sensors, such as proximity sensors, motion sensors, and other sensors. Specifically, the proximity sensor can turn off the display panel 1441 and / or backlight when the phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. As for other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described in detail here.

[0216] Audio circuit 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone. Audio circuit 1460 converts received audio data into electrical signals and transmits them to speaker 1461, where speaker 1461 converts them into sound signals for output. On the other hand, microphone 1462 converts collected sound signals into electrical signals, which are received by audio circuit 1460, converted into audio data, and then processed by processor 1480 before being transmitted via RF circuit 1410 to, for example, another mobile phone, or the audio data can be output to memory 1420 for further processing.

[0217] WiFi is a short-range wireless transmission technology. Mobile phones, through the WiFi module 1470, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 14 WiFi module 1470 is shown, but it is understood that it is not an essential component of a mobile phone.

[0218] The processor 1480 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1420, and calls data stored in the memory 1420 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1480 may include one or more processing units; preferably, the processor 1480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1480.

[0219] The mobile phone also includes a power supply 1490 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0220] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0221] In this embodiment of the application, the processor 1480 included in the image processing device can perform the aforementioned... Figures 1 to 10 The functions described in the illustrated embodiments will not be repeated here.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0223] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented wholly or partially through software, hardware, firmware, or any combination thereof.

[0225] When the integrated unit is implemented using software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0226] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

Claims

1. An image processing method, characterized in that, The method is applied to an image processing device, the image processing device including a color camera and a multispectral sensor, and the method includes: The first image to be processed is acquired through the color camera; The first environmental spectral information is acquired by the multispectral sensor, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. White balance gain is obtained based on the first image to be processed and the first environmental spectral information; The first image to be processed is subjected to a first processing to obtain a first target image; The first processing includes white balance processing based on the white balance gain.

2. The method according to claim 1, characterized in that, The first image to be processed and the first target image are both solid color images.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining white balance gain based on the first image to be processed and the first environmental spectral information includes: The first environmental spectral information and the first image to be processed are input into a trained neural network to obtain the white balance gain. The trained neural network is obtained by training the neural network with training data as input and aiming to make the loss function value less than a threshold. The training data includes the original training image and training spectral information, which correspond to the same shooting scene. The output of the neural network includes white balance gain. The loss function is used to indicate the difference between the white balance gain output by the neural network and the actual white balance gain. The actual white balance gain is obtained by processing the response value of the gray card in the shooting scene.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the plurality of spectral response functions of the color camera; Multiple compensation values ​​are obtained based on the first environmental spectral information and the multiple spectral response functions; The first process further includes: color shading processing based on the plurality of compensation values.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the tristimulus value curve and the reflectance of the color card; A color correction matrix is ​​obtained based on the first environmental spectral information, the multiple spectral response functions, the reflectance, and the tristimulus value curves. The first process further includes: color space conversion processing based on the color correction matrix.

6. The method according to claim 5, characterized in that, The process of obtaining the color correction matrix based on the first environmental spectral information, the plurality of spectral response functions, the reflectance, and the tristimulus value curves includes: The first environmental spectral information is converted into a light source curve; The first response value of the color card to the color camera is obtained based on the multiple spectral response functions, the light source curve, and the reflectance. Based on the tristimulus curve, the light source curve, and the reflectance, the second response value of the color card to the first human eye color space is obtained, where the first human eye color space is the response space corresponding to the human eye matching function. The color correction matrix is ​​obtained based on the first response value and the second response value, and the color correction matrix is ​​used to represent the correlation between the first response value and the second response value.

7. The method according to any one of claims 1 to 6, characterized in that, The first process further includes: The image after white balance processing is post-processed to obtain the first target image.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first target image is displayed to the user.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The second image to be processed is acquired through the color camera; The second environmental spectral information is obtained by the multispectral sensor, and the second environmental spectral information corresponds to the same shooting scene as the second image to be processed. The filtering parameters are determined based on the similarity between the first environmental spectral information and the second environmental spectral information. Based on the filtering parameters, the first target image and the second image to be processed are filtered to obtain the correction parameters; The second target image is obtained by adjusting the second image to be processed based on the correction parameters.

10. The method according to claim 9, characterized in that, The step of determining the filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information includes: A filter strength function is generated based on the similarity; The filtering parameters are determined based on the filtering intensity function.

11. An image processing method, characterized in that, The method is applied to an image processing device, the image processing device including a color camera and a multispectral sensor, and the method includes: The first image to be processed is acquired through the color camera; The first environmental spectral information is acquired by the multispectral sensor, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. Obtain multiple spectral response functions of the color camera; Multiple compensation values ​​are obtained based on the multiple first environmental spectral information and the multiple spectral response functions; The first image to be processed is subjected to a first processing to obtain a first target image; The first process includes color shading processing based on the plurality of compensation values.

12. An image processing method, characterized in that, The method is applied to an image processing device, the image processing device including a color camera and a multispectral sensor, and the method includes: The first image to be processed is acquired based on the color camera; The first environmental spectral information is obtained based on the multispectral sensor, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. Obtain multiple spectral response functions of the color camera; Obtain the tristimulus value curve and the reflectance of the color chart; A color correction matrix is ​​obtained based on the first environmental spectral information, the multiple spectral response functions, the reflectance, and the tristimulus value curves. The first image to be processed is subjected to a first processing to obtain a first target image; The first process includes color space conversion based on the color correction matrix.

13. The method according to claim 12, characterized in that, The process of obtaining the color correction matrix based on the first environmental spectral information, the plurality of spectral response functions, the reflectance, and the tristimulus value curves includes: The first environmental spectral information is converted into a light source curve; Based on the multiple spectral response functions, the light source curve, and the reflectance, the first response value of the color card to the color camera is obtained; Based on the tristimulus curve, the light source curve, and the reflectance, the second response value of the color card to the first human eye color space is obtained, where the first human eye color space is the response space corresponding to the human eye matching function. The color correction matrix is ​​obtained based on the first response value and the second response value, and the color correction matrix is ​​used to represent the conversion relationship between the first response value and the second response value.

14. The method according to claim 13, characterized in that, The method further includes: The image after color space conversion is adjusted based on the conversion relationship between the first human eye color space and the second human eye color space, where the second human eye color space is the response space corresponding to the color appearance model when performing color adaptation.

15. An image processing device, characterized in that, The image processing device includes: The first acquisition unit is used to acquire a first image to be processed through the color camera; The second acquisition unit is used to acquire first environmental spectral information through the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The third acquisition unit is used to acquire white balance gain based on the first image to be processed and the first environmental spectral information; The processing unit is configured to perform a first processing on the first image to be processed to obtain a first target image, wherein the first processing includes white balance processing based on the white balance gain.

16. The device according to claim 15, characterized in that, The first image to be processed and the first target image are both solid color images.

17. The device according to claim 15 or 16, characterized in that, The third acquisition unit is specifically used to input the first environmental spectral information and the first image to be processed into a trained neural network to obtain the white balance gain. The trained neural network is obtained by training the neural network with training data as input and aiming to make the loss function value less than a threshold. The training data includes the original training image and training spectral information, which correspond to the same shooting scene. The output of the neural network includes white balance gain. The loss function is used to indicate the difference between the white balance gain output by the neural network and the actual white balance gain. The actual white balance gain is obtained by processing the response value of the gray card in the shooting scene.

18. The device according to any one of claims 15 to 17, characterized in that, The device also includes: The fourth acquisition unit is used to acquire multiple spectral response functions of the color camera; The fourth acquisition unit is further configured to acquire multiple compensation values ​​based on the first environmental spectral information and the multiple spectral response functions; The processing unit is also used for color shading processing based on the plurality of compensation values.

19. The device according to claim 18, characterized in that, The device also includes: The fifth acquisition unit is used to acquire the tristimulus value curve and the reflectance of the color card; The fifth acquisition unit is also used to acquire a color correction matrix based on the first environmental spectral information, the plurality of spectral response functions, the reflectance, and the tristimulus value curve; The processing unit is also used for color space conversion processing based on the color correction matrix.

20. The device according to claim 19, characterized in that, The fifth acquisition unit is specifically used to convert the first environmental spectral information into a light source curve. The fifth acquisition unit is specifically used to acquire the first response value of the color card to the color camera based on the plurality of spectral response functions, the light source curve and the reflectance; The fifth acquisition unit is specifically used to acquire the second response value of the color card to the first human eye color space based on the tristimulus value curve, the light source curve and the reflectance, wherein the first human eye color space is the response space corresponding to the human eye matching function. The fifth acquisition unit is specifically used to acquire the color correction matrix based on the first response value and the second response value, wherein the color correction matrix is ​​used to represent the correlation between the first response value and the second response value.

21. The device according to any one of claims 15 to 20, characterized in that, The processing unit is further configured to perform post-processing on the image after the white balance processing to obtain the first target image.

22. The device according to any one of claims 15 to 21, characterized in that, The device also includes: The display unit is used to display the first target image to the user.

23. The device according to any one of claims 15 to 22, characterized in that, The first acquisition unit is further configured to acquire a second image to be processed via the color camera; The second acquisition unit is further configured to acquire second environmental spectral information through the multispectral sensor, wherein the second environmental spectral information corresponds to the same shooting scene as the second image to be processed; The device also includes: A determining unit is configured to determine filtering parameters based on the similarity between the first environmental spectral information and the second environmental spectral information; A filtering unit is used to filter the first target image and the second image to be processed based on the filtering parameters to obtain correction parameters; The processing unit is further configured to adjust the second image to be processed based on the correction parameters to obtain the second target image.

24. The device according to claim 23, characterized in that, The determining unit is specifically used to generate a filter intensity function based on the similarity. The determining unit is specifically used to determine the filtering parameters based on the filtering intensity function.

25. An image processing device, characterized in that, The image processing device includes: The first acquisition unit is used to acquire a first image to be processed through the color camera; The second acquisition unit is used to acquire first environmental spectral information through the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The third acquisition unit is used to acquire multiple spectral response functions of the color camera; The third acquisition unit is further configured to acquire multiple compensation values ​​based on the first environmental spectral information and the multiple spectral response functions; The processing unit is configured to perform a first processing on the first image to be processed to obtain a first target image; The first process includes color shading based on the plurality of compensation values.

26. An image processing device, characterized in that, The image processing device includes: The first acquisition unit is used to acquire a first image to be processed through the color camera; The second acquisition unit is used to acquire first environmental spectral information through the multispectral sensor, wherein the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The third acquisition unit is used to acquire multiple spectral response functions of the color camera; The third acquisition unit is also used to acquire the tristimulus value curve and the reflectance of the color card; The third acquisition unit is further configured to acquire a color correction matrix based on the first environmental spectral information, the plurality of spectral response functions, the reflectance, and the tristimulus value curve; The processing unit is configured to perform a first processing on the first image to be processed to obtain a first target image; The first process includes color space conversion based on the color correction matrix.

27. The device according to claim 26, characterized in that, The third acquisition unit is specifically used to convert the first environmental spectral information into a light source curve. The third acquisition unit is specifically used to acquire the first response value of the color card to the color camera based on the plurality of spectral response functions, the light source curve, and the reflectance. The third acquisition unit is specifically used to acquire the second response value of the color card to the first human eye color space based on the tristimulus value curve, the light source curve, and the reflectance, wherein the first human eye color space is the response space corresponding to the human eye matching function. The third acquisition unit is specifically used to acquire the color correction matrix based on the first response value and the second response value, wherein the color correction matrix is ​​used to represent the conversion relationship between the first response value and the second response value.

28. The device according to claim 27, characterized in that, The processing unit is further configured to adjust the image after color space conversion based on the conversion relationship between the first human eye color space and the second human eye color space, wherein the second human eye color space is the response space corresponding to the color appearance model when performing color adaptation.

29. An image processing device, characterized in that, The image processing device includes a color camera, a multispectral sensor, and an image processor; The color camera is used to acquire the first image to be processed; The multispectral sensor is used to acquire first environmental spectral information, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The image processor is used to perform a first processing on the first image to be processed to obtain a first target image; The first processing includes white balance processing based on the white balance gain.

30. An image processing device, characterized in that, The image processing device includes a color camera, a multispectral sensor, and an image processor; The color camera is used to acquire the first image to be processed; The multispectral sensor is used to acquire first environmental spectral information, and the first environmental spectral information corresponds to the same shooting scene as the first image to be processed. The image processor is used to acquire multiple spectral response functions of the color camera; The image processor is further configured to obtain multiple compensation values ​​based on the first environmental spectral information and the multiple spectral response functions; The image processor is further configured to perform a first processing on the first image to be processed to obtain a first target image; The first process includes color shading processing based on the plurality of compensation values.

31. An image processing device, characterized in that, The image processing device includes a color camera, a multispectral sensor, and an image processor; The color camera is used to acquire the first image to be processed; The multispectral sensor is used to acquire first environmental spectral information corresponding to the first image to be processed; The image processor is used to acquire multiple spectral response functions of the color camera; The image processor is also used to obtain the tristimulus value curve and the reflectance of the color card; The image processor is further configured to obtain a color correction matrix based on the first environmental spectral information, the plurality of spectral response functions, the reflectance, and the tristimulus value curve; The image processor is further configured to perform a first processing on the first image to be processed to obtain a first target image; The first process includes color space conversion based on the color correction matrix.

32. An image processing device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the image processing device to perform the method as described in claims 1-14.

33. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 14.

34. A computer program product, characterized in that, When the computer program product is executed on a computer, it causes the computer to perform the method as described in any one of claims 1 to 14.

Images