Imaging method, system, and apparatus

Abstract

The present application provides an imaging method, a system, and an apparatus, for use in projecting high-dimensional color information in a multispectral image as a primary color into the space of each camera in a multi-camera scene, thereby improving the color consistency and accuracy of an image in the multi-camera scene. The method comprises: acquiring a plurality of images by means of a plurality of image sensors (301); acquiring a multispectral image by means of a multispectral sensor (302), wherein the plurality of images and the multispectral image comprise the same scene information; then acquiring a plurality of high-dimensional color mapping parameters on the basis of the multispectral image (303), wherein the plurality of high-dimensional color mapping parameters represent mapping relationships between the color of the multispectral image and colors of images, and each image sensor has a corresponding high-dimensional color mapping parameter; and performing color adjustment on first images on the basis of the plurality of high-dimensional color mapping parameters to obtain a plurality of color-adjusted images (304).

Description

An imaging method, system and apparatus

[0001] The present application claims priority to the Chinese Patent Application No. 202411280408.9, filed on September 11, 2024, entitled "An imaging method, system and apparatus", and the Chinese Patent Application No. 202510401161.X, filed on March 31, 2025, entitled "An imaging method, system and apparatus", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the field of image processing, and in particular, to an imaging method, system and apparatus. BACKGROUND

[0003] In order to improve the camera effect of a terminal, it is very common to arrange an array camera in the terminal, that is, to arrange multiple cameras. A common terminal usually needs to be equipped with a color image sensor. The imaging method of the color image sensor is usually to use a color filter array (CFA), also known as a Bayer filter, arranged above a photosensitive area. For example, a common CFA arrangement includes a 2x2 four-pixel point to form a cycle unit. This unit contains three primary colors: red (R), green (G), and blue (B), so the color image sensor is usually also called an RGB image sensor. However, the response of the RGB sensor is a sampling (or integration) of complete spectral information, which cannot completely capture spectral information and is prone to cause metamerism problems, resulting in limited color reproduction performance of the existing RGB camera in some difficult example scenes, such as scenes containing large-area pure color regions. Especially in a multi-camera scene, each camera can independently image, and may also be started at the same time when taking a picture. The color difference between different cameras in the same scene will seriously affect the user experience.

[0004] Therefore, how to improve the consistency of the imaging colors of each camera in a multi-camera scene has become a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide an imaging method, system and apparatus for projecting high-dimensional color information in a multi-spectral image as a baseline color into the space of each camera in a multi-camera, to improve the consistency and accuracy of image colors in a multi-camera scene.

[0006] Therefore, in a first aspect, embodiments of the present application provide an imaging method, which can be applied to an electronic device including a plurality of image sensors and a multispectral sensor, such as a first image sensor and a second image sensor. The method includes: acquiring a plurality of images by the plurality of image sensors, and acquiring a multispectral image by the multispectral sensor, the plurality of images including a first image acquired by the first image sensor and a second image acquired by the second image sensor, the plurality of images and the multispectral image including the same scene information; then obtaining a plurality of high-dimensional color mapping parameters according to the multispectral image, the plurality of high-dimensional color mapping parameters being used to represent a mapping relationship between colors of the multispectral image and colors of each image, each image sensor having a corresponding high-dimensional color mapping parameter, such as obtaining a first high-dimensional color mapping parameter and a second high-dimensional color mapping parameter according to the multispectral image, the first high-dimensional color mapping parameter being used to represent a mapping relationship between colors of the multispectral image and colors of the first image, and the second high-dimensional color mapping parameter being used to represent a mapping relationship between colors of the multispectral image and colors of the second image; color adjusting each image according to the plurality of high-dimensional color mapping parameters to obtain a plurality of color-adjusted images, such as color adjusting the first image according to the first high-dimensional color mapping parameter to obtain a third image, and color adjusting the second image according to the second high-dimensional color mapping parameter to obtain a fourth image.

[0007] The multispectral image is acquired by the multispectral sensor, which can acquire light signals of each spectral band, including visible light or invisible light, etc., so that the multispectral image includes richer and more accurate color information. In the embodiments of the present application, the color information in the multispectral image is used as a base color to adjust the color in each image, which can make the color in the output second image more accurate, and the color consistency between the plurality of color-adjusted images more optimal. Taking the first image acquired by the first image sensor and the second image acquired by the second image sensor as an example, the first high-dimensional color mapping parameter corresponding to the first image and the second high-dimensional color mapping image corresponding to the second image are determined based on the multispectral image, and the color of the first image is adjusted based on the first high-dimensional color mapping parameter to obtain a third image, and the color of the second image is adjusted based on the second high-dimensional color mapping matrix to obtain a fourth image, so that the third image and the fourth image both take the color of the multispectral image as a base color, and thus the colors of the third image and the fourth image have a certain degree of consistency.

[0008] In a possible implementation, the foregoing color adjusting the plurality of images according to the first high-dimensional color mapping parameter to obtain the plurality of color-adjusted images can include: projecting data of the multi-spectrum image to a space corresponding to each of the plurality of image sensors according to the plurality of high-dimensional color mapping parameters to obtain a plurality of multi-spectrum low-dimensional projection results, such as projecting data of the multi-spectrum image to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result; then determining a color adjustment parameter of each of the image sensors according to the plurality of multi-spectrum low-dimensional projection results, and color adjusting data of each of the images according to the color adjustment parameter of each of the image sensors to obtain the plurality of color-adjusted images, such as determining a first color adjustment parameter of the first image sensor according to the first multi-spectrum low-dimensional projection result, and color adjusting data of the first image according to the first color adjustment parameter to obtain a third image.

[0009] In the embodiments of the present application, the color information in the multi-spectrum image can be projected into the space corresponding to each of the image sensors based on the high-dimensional color mapping parameter corresponding to each of the image sensors, so as to project the high-dimensional color in the multi-spectrum image into the low-dimensional space corresponding to each of the image sensors, thereby adjusting the color of each of the images by taking the multi-spectrum color projected into the low-dimensional space as a base color, so as to improve the color accuracy of the output image and improve the color consistency in the multi-camera scene.

[0010] In a possible implementation, the foregoing projecting data of the multi-spectrum image into a space corresponding to each of the image sensors according to the plurality of high-dimensional color mapping parameters to obtain a plurality of multi-spectrum low-dimensional projection results can include: obtaining a high-dimensional white point in the multi-spectrum image; mapping the high-dimensional white point into a space corresponding to each of the images according to the plurality of high-dimensional color mapping parameters to obtain a plurality of low-dimensional white points, the multi-spectrum low-dimensional projection results including the plurality of low-dimensional white points; the foregoing determining a color adjustment parameter of each of the image sensors according to the multi-spectrum low-dimensional projection results, and color adjusting data of each of the images according to the color adjustment parameter of each of the image sensors to obtain the plurality of color-adjusted images can include: obtaining a white balance coefficient of each of the image sensors according to the plurality of low-dimensional white points, the color adjustment parameter including the white balance coefficient of each of the image sensors; and performing white balance processing on each of the images according to the white balance coefficient of each of the image sensors to obtain the plurality of color-adjusted images. For example, for the first image, the data of the multi-spectrum image is projected into a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result, the first color adjustment parameter of the first image sensor is determined according to the first multi-spectrum low-dimensional projection result, and the data of the first image is color adjusted according to the first color adjustment parameter to obtain a third image. In the implementation of the present application, the specific manner of color adjusting the first image can be white balance processing, so that the first image is white balanced based on the white point in the multi-spectrum image, and the color information of the multi-spectrum image is projected into each image through the white balance manner, so as to improve the color accuracy in the output image and the color consistency of the multi-camera image.

[0011] In a possible implementation, the foregoing projecting data of the multi-spectrum image into a space corresponding to each of the image sensors according to the plurality of high-dimensional color mapping parameters to obtain a plurality of multi-spectrum low-dimensional projection results can include: projecting the multi-spectrum image into a space corresponding to each of the images according to the plurality of high-dimensional color mapping parameters to obtain a plurality of low-dimensional color projection images, and the multi-spectrum low-dimensional projection result includes the plurality of low-dimensional color projection images, such as projecting the multi-spectrum image into a space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain a first low-dimensional color projection image, and the first multi-spectrum low-dimensional projection result includes the first low-dimensional color projection image; and the foregoing determining the color adjustment parameter of each image sensor according to the plurality of multi-spectrum low-dimensional projection results, and performing color adjustment on the data of each image according to the color adjustment parameter of each image sensor can include: determining the low-dimensional color mapping parameter of each image sensor according to the multi-spectrum low-dimensional color projection image, and the color adjustment parameter includes the low-dimensional color mapping parameter of each image sensor; and performing color mapping on the plurality of images according to the low-dimensional color mapping parameter to obtain the plurality of images after color adjustment, such as determining the first low-dimensional color mapping parameter of the first image sensor according to the multi-spectrum low-dimensional color projection image, and the first color adjustment parameter includes the first low-dimensional color mapping parameter; and performing color mapping on the first image according to the first low-dimensional color mapping parameter to obtain a third image. In the implementation of the present application, the color information in the multi-spectrum image can be projected into the low-dimensional space corresponding to each image sensor, so as to directly map the color in the multi-spectrum as a base color to the image collected by each image sensor, thereby improving the color accuracy of the output image and the color consistency of the multi-camera image.

[0012] In a possible implementation, the foregoing method can further include: obtaining a third high-dimensional color mapping parameter; projecting a color value of the multi-spectrum image into a standard space according to the third high-dimensional color mapping parameter to obtain a standard space projection image; and the foregoing determining the low-dimensional color mapping parameter of each image sensor according to the multi-spectrum low-dimensional color projection image includes: determining the low-dimensional color mapping parameter corresponding to each image sensor according to the multi-spectrum low-dimensional image and the standard space projection image, and specifically can include determining the first low-dimensional color mapping parameter corresponding to the first image according to the multi-spectrum low-dimensional image and the standard space projection image.

[0013] In the implementation of the present application, in order to improve the projection of the color in the multi-spectrum image into the low-dimensional space corresponding to the image sensor, the color in the multi-spectrum image can be projected into a color adapted to the color dimension in each space based on the standard color space, so as to improve the color accuracy of the output image and the color consistency of the multi-camera image.

[0014] In a possible implementation, the method further includes: obtaining a color mapping matrix by using the pre-trained color mapping network, the color mapping matrix being used to map color information in the multi-spectrum image to each of the images; and the color adjusting of the plurality of images according to the first high-dimensional color mapping parameter to obtain the plurality of color-adjusted images further includes: updating the color adjustment parameter according to the color mapping matrix to obtain an updated color adjustment parameter; and color adjusting each of the images according to the updated color adjustment parameter to obtain the plurality of color-adjusted images, such as updating the first color adjustment parameter according to the color mapping matrix to obtain an updated first color adjustment parameter; and color adjusting the first image according to the updated first color adjustment parameter to obtain a third image.

[0015] In the implementation, the neural network is also used to output the color mapping matrix, so that the color information in the multi-spectrum image is projected to each of the images as a base color by using the output capability of the neural network, thereby improving the color accuracy of the output image and the color consistency of the multi-camera images.

[0016] In a possible implementation, the method of obtaining a plurality of high-dimensional color mapping parameters from the multi-spectrum image includes: obtaining a plurality of initial first high-dimensional color mapping parameters from the multi-spectrum image; subsequently aligning the multi-spectrum image with each of the plurality of images to obtain a plurality of alignment results; and updating the plurality of initial first high-dimensional color mapping parameters according to each of the alignment results to obtain the plurality of high-dimensional color mapping parameters. That is, the process of obtaining the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter can include: obtaining an initial first high-dimensional color mapping parameter and an initial second high-dimensional color mapping parameter from the multi-spectrum image; aligning the multi-spectrum image with the initial first high-dimensional color mapping parameter to obtain a first alignment result, and aligning the multi-spectrum image with the initial second high-dimensional color mapping parameter to obtain a second alignment result; updating the initial first high-dimensional color mapping parameter according to the first alignment result to obtain the first high-dimensional color mapping parameter, and updating the initial second high-dimensional color mapping parameter according to the second alignment result to obtain the second high-dimensional color mapping parameter. Therefore, in the implementation, the initial first high-dimensional color mapping parameter can be adjusted based on the alignment result of the multi-spectrum image and each of the images, so that the obtained first high-dimensional color mapping parameter is more suitable for the space of each image sensor, thereby improving the accuracy of subsequent projection of the multi-spectrum image to the space of each image sensor.

[0017] In a possible implementation, the foregoing obtaining the plurality of high-dimensional color mapping parameters from the multi-spectrum image further includes: obtaining color information from the multi-spectrum image, which can specifically include at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum; and calculating the plurality of high-dimensional color mapping parameters according to the color information, such as calculating the plurality of high-dimensional color mapping parameters according to the color temperature or the light source spectrum in the multi-spectrum image. Correspondingly, the foregoing obtaining the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter from the multi-spectrum image can include: obtaining color information from the multi-spectrum image, the color information including at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum; and calculating the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter according to the color information.

[0018] In the implementation, the first high-dimensional color mapping parameter can be calculated by using the color information in the multi-spectrum image, so that the first high-dimensional color mapping parameter is more adaptive to the color information of the multi-spectrum image.

[0019] In a possible implementation, the foregoing obtaining the color information from the multi-spectrum image specifically includes: obtaining color information corresponding to N high-dimensional light sources from the multi-spectrum image, N being a positive integer, and N can be a preset value, for example, the types of light sources can be pre-divided. The foregoing calculating the plurality of high-dimensional color mapping parameters according to the color information includes: obtaining a first spectral curve of each image sensor and a second spectral curve corresponding to the multi-spectrum sensor; determining a first color card image corresponding to each sensor and a second color card image corresponding to the multi-spectrum sensor based on an imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve of each sensor, and the second spectral curve; and fitting the high-dimensional color mapping parameter according to the first color card image and the second color card image. Correspondingly, the first spectral curve corresponding to the multi-spectrum sensor, the second spectral curve corresponding to the first image sensor, and the third spectral curve corresponding to the third image sensor can be obtained; the first color card image corresponding to the multi-spectrum sensor, the second color card image corresponding to the first image sensor, and the third color card image corresponding to the second image sensor can be determined based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve, and the third spectral curve; and the first high-dimensional color mapping parameter is fitted according to the first color card image and the second color card image, and the second high-dimensional color mapping parameter is fitted according to the first color card image and the third color card image.

[0020] In the embodiments of the present application, the color information included in the multispectral image and the light source information when each image sensor acquires the first image can be used to fit a mapping relationship between the color in the multispectral image and the color of the image acquired by the image sensor based on an imaging model, so as to project the color in the multispectral image into the low-dimensional space corresponding to each image sensor, realize the conversion of the high-dimensional color in the multispectral image to the low-dimensional color, and thus improve the color accuracy in the low-dimensional first image and obtain a second image with higher color accuracy and better multi-camera color consistency.

[0021] In a possible implementation, the foregoing acquiring a plurality of images by a plurality of image sensors and acquiring a multispectral image by a multispectral sensor includes: acquiring the plurality of images by the plurality of image sensors and acquiring the multispectral image by the multispectral sensor at the same time. In the embodiments of the present application, the multispectral image and the plurality of images can be acquired at the same time, so that the multispectral image and the plurality of images contain information of the same scene, that is, the first image is acquired by the first image sensor, the second image is acquired by the second image sensor, and the multispectral image is acquired by the multispectral sensor at the same time.

[0022] In a possible implementation, the foregoing electronic device further includes a third camera, or the electronic device further includes a third image sensor and a fourth image sensor, and the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor correspond to at least two focal lengths.

[0023] In a possible implementation, the foregoing first image sensor or second image sensor includes at least one of the following sensors: a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera, or an ultra-long-focus camera.

[0024] In a second aspect, the embodiments of the present application provide an imaging system, including: a plurality of image sensors, a multispectral sensor, and a processing unit, the plurality of image sensors including a first image sensor and a second image sensor;

[0025] The plurality of image sensors are configured to acquire a plurality of images, wherein the first image sensor is configured to acquire a first image, and the second image sensor is configured to acquire a second image;

[0026] The multispectral sensor is configured to acquire a multispectral image;

[0027] The processing unit is configured to acquire high-dimensional color mapping parameters according to the multi-spectrum image, the high-dimensional color mapping parameters representing a mapping relationship between colors of the multi-spectrum image and colors of the multiple images. Specifically, the first high-dimensional color mapping parameters and the second high-dimensional color mapping parameters are acquired according to the multi-spectrum image. The first high-dimensional color mapping parameters are used to represent a mapping relationship between colors of the multi-spectrum image and colors of the first image. The second high-dimensional color mapping parameters are used to represent a mapping relationship between colors of the multi-spectrum image and colors of the second image.

[0028] The processing unit is further configured to perform color adjustment on the multiple images according to the first high-dimensional color mapping parameters to obtain the multiple images after color adjustment. Specifically, the first image is color adjusted according to the first high-dimensional color mapping parameters to obtain the third image. The second image is color adjusted according to the second high-dimensional color mapping parameters to obtain the fourth image.

[0029] Effects of the second aspect and any optional implementation manner of the second aspect can be referred to the description of the first aspect or any optional implementation manner of the first aspect, which will not be described here.

[0030] In a possible implementation manner, the processing unit is specifically configured to: project data in the multi-spectrum image to a space corresponding to each image sensor of the multiple image sensors according to the first high-dimensional color mapping parameters to obtain a low-dimensional projection result; determine a color adjustment parameter of each image sensor according to the low-dimensional projection result, and adjust colors of each image according to the color adjustment parameter of each image sensor to obtain the multiple images after color adjustment. Specifically, the data in the multi-spectrum image is projected to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameters to obtain a first multi-spectrum low-dimensional projection result. A first color adjustment parameter of the first image sensor is determined according to the low-dimensional projection result, and colors of the first image are adjusted according to the first color adjustment parameter to obtain the third image.

[0031] In a possible implementation, the processing unit is specifically configured to: obtain a high-dimensional white point in the multi-spectrum image; map the high-dimensional white point to a space corresponding to each of the images according to a plurality of high-dimensional color mapping parameters, to obtain a plurality of low-dimensional white points, the multi-spectrum low-dimensional projection result including the plurality of low-dimensional white points; obtain a white balance coefficient of each image sensor according to the plurality of low-dimensional white points, the color adjustment parameter including the white balance coefficient of each image sensor; and perform white balance processing on each image according to the white balance coefficient of each image sensor, to obtain the plurality of images after color adjustment. Specifically, the high-dimensional white point in the multi-spectrum image is obtained, the high-dimensional white point is mapped to a space corresponding to the first image according to a first high-dimensional color mapping parameter, to obtain a first low-dimensional white point, the multi-spectrum low-dimensional projection result including the first low-dimensional white point, a first white balance coefficient of the first image sensor is obtained according to the first low-dimensional white point, the first color adjustment parameter including the first white balance coefficient, and the first image is processed according to the first white balance coefficient to obtain the third image.

[0032] In a possible implementation, the processing unit is specifically configured to: project the multi-spectrum image to a space corresponding to each of the images according to a plurality of high-dimensional color mapping parameters, to obtain a plurality of low-dimensional color projection images, the multi-spectrum low-dimensional projection result including the plurality of low-dimensional color projection images; determine a low-dimensional color mapping parameter of each image sensor according to the multi-spectrum low-dimensional color projection image, the color adjustment parameter including the low-dimensional color mapping parameter of each image sensor; and perform color mapping on the plurality of images according to the low-dimensional color mapping parameter, to obtain the plurality of images after color adjustment. Specifically, the multi-spectrum image is projected to a space corresponding to the first image according to a first high-dimensional color mapping parameter, to obtain a first low-dimensional color projection image, the first multi-spectrum low-dimensional projection result including the first low-dimensional color projection image, a low-dimensional color mapping parameter of the first image sensor is determined according to the multi-spectrum low-dimensional color projection image, the color adjustment parameter including the first low-dimensional color mapping parameter, and the first image is processed according to the first low-dimensional color mapping parameter, to obtain the third image.

[0033] In a possible implementation, the processing unit is further configured to: obtain a third high-dimensional color mapping parameter; project a color value of the multi-spectrum image to a standard space according to the third high-dimensional color mapping parameter, to obtain a standard space projection graph; and determine a low-dimensional color mapping parameter corresponding to each image sensor according to the multi-spectrum low-dimensional image and the standard space projection graph. Specifically, the third high-dimensional color mapping parameter is obtained; a color value of the multi-spectrum image is projected to a standard space according to the third high-dimensional color mapping parameter, to obtain a standard space projection graph; and the first low-dimensional color mapping parameter is determined according to the multi-spectrum low-dimensional image and the standard space projection graph.

[0034] In a possible implementation, the processing unit is further configured to: obtain a color mapping matrix through the color mapping network; update the color adjustment parameter according to the color mapping matrix to obtain an updated color adjustment parameter; and perform color adjustment on each image according to the updated color adjustment parameter to obtain the plurality of color-adjusted images. Specifically, the processing unit can obtain a first color mapping matrix through the color mapping network; update the first color adjustment parameter according to the first color mapping matrix to obtain an updated first color adjustment parameter; and perform color adjustment on the first image according to the updated first color adjustment parameter to obtain a third image.

[0035] In a possible implementation, the processing unit is specifically configured to: obtain a plurality of initial high-dimensional color mapping parameters according to the multi-spectrum image; perform alignment on the multi-spectrum image and each image in the plurality of images to obtain a plurality of alignment results; and update the plurality of initial high-dimensional color mapping parameters according to each alignment result to obtain the plurality of high-dimensional color mapping parameters. Specifically, the processing unit can obtain an initial first high-dimensional color mapping parameter and an initial second high-dimensional color mapping parameter according to the multi-spectrum image; perform alignment on the multi-spectrum image and the initial first high-dimensional color mapping parameter to obtain a first alignment result; perform alignment on the multi-spectrum image and the initial second high-dimensional color mapping parameter to obtain a second alignment result; update the initial first high-dimensional color mapping parameter according to the first alignment result to obtain a first high-dimensional color mapping parameter; and update the initial second high-dimensional color mapping parameter according to the second alignment result to obtain a second high-dimensional color mapping parameter.

[0036] In a possible implementation, the processing unit is specifically configured to: obtain color information from the multi-spectrum image, the color information including at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum; and calculate the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter according to the color information.

[0037] In a possible implementation, the processing unit is specifically configured to: obtain color information corresponding to N high-dimensional light sources from the multi-spectrum image; obtain a first spectral curve of each of the plurality of image sensors and a second spectral curve corresponding to the multi-spectrum sensor; determine a first color card image corresponding to each of the sensors and a second color card image corresponding to the multi-spectrum sensor based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve of each of the sensors, and the second spectral curve; and obtain the first high-dimensional color mapping parameter by fitting the first color card image and the second color card image. Specifically, the processing unit can obtain the color information corresponding to the N high-dimensional light sources from the multi-spectrum image, obtain a first spectral curve corresponding to the multi-spectrum sensor and a second spectral curve corresponding to the first image sensor and a third spectral curve corresponding to the third image sensor, determine a first color card image corresponding to the multi-spectrum sensor, a second color card image corresponding to the first image sensor, and a third color card image corresponding to the second image sensor based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve, and the third spectral curve, and obtain the first high-dimensional color mapping parameter by fitting the first color card image and the second color card image and obtain a second high-dimensional color mapping parameter by fitting the first color card image and the third color card image.

[0038] In a possible implementation, the plurality of image sensors capture a plurality of images and the multi-spectrum sensor captures a multi-spectrum image at the same time. Specifically, the first image sensor captures a first image, the second image sensor captures a second image, and the multi-spectrum sensor captures a multi-spectrum image at the same time.

[0039] In a possible implementation, the electronic device further includes a third camera, or the electronic device further includes a third image sensor and a fourth image sensor, and the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor correspond to at least two focal lengths.

[0040] In a possible implementation, the first image sensor or the second image sensor includes at least one of the following sensors: a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera, or an ultra-long-focus camera.

[0041] In a third aspect, an embodiment of the present application provides an imaging system, including a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor invokes program codes in the memory to perform functions related to processing in the imaging method shown in any of the first aspect. Optionally, the imaging system can include a chip.

[0042] In a fourth aspect, an electronic device is provided, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is configured to perform the processing-related functions in the first aspect or any of the optional implementation forms of the first aspect.

[0043] In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions. When the instructions are executed on a computer, the computer is caused to perform the method in the first aspect or any of the optional implementation forms of the first aspect.

[0044] In a sixth aspect, a computer program product is provided. The computer program product includes instructions. When the instructions are executed on a computer, the computer is caused to perform the method in the first aspect or any of the optional implementation forms of the first aspect.

[0045] In a seventh aspect, a chip is provided. The chip includes at least one processor and an interface. The at least one processor obtains program instructions or data through the interface. The at least one processor is configured to execute the program instructions to implement the method in the first aspect or any of the implementation forms of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0046] FIG. 1 is a structural schematic diagram of an electronic device provided by the present application;

[0047] FIG. 2 is a schematic diagram of a system architecture provided by the present application;

[0048] FIG. 3 is a flow schematic diagram of an imaging method provided by the present application;

[0049] FIG. 4 is a flow schematic diagram of another imaging method provided by the present application;

[0050] FIG. 5 is a schematic diagram of an imaging system architecture provided by the present application;

[0051] FIG. 6 is a flow schematic diagram of another imaging method provided by the present application;

[0052] FIG. 7 is a flow schematic diagram of another imaging method provided by the present application;

[0053] FIG. 8 is a flow schematic diagram of another imaging method provided by the present application;

[0054] FIG. 9 is a flow schematic diagram of another imaging method provided by the present application;

[0055] FIG. 10 is a flow schematic diagram of another imaging method provided by the present application;

[0056] FIG. 11 is a flow schematic diagram of another imaging method provided by the present application;

[0057] FIG. 12 is a schematic diagram of an architecture of another imaging system provided by the present application;

[0058] FIG. 13 is a schematic diagram of a structure of another electronic device provided by the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0060] For ease of understanding, some terms or concepts related to the method provided by the present application are explained below.

[0061] (1) Multi-camera consistency

[0062] Array camera is a common camera deployment method of electronic devices. The main difference between these cameras is the focal length, mainly divided into ultra-wide angle, wide angle (main camera) and telephoto. Different mobile phone configurations may be different, but usually at least two are set. These cameras can all independently image, and all have the need for photographic records, and are directly presented to the user for browsing, so the user has a high requirement for the color consistency of each camera of the array camera.

[0063] (2) Multi-Spectral Imaging / Image (MSI)

[0064] It refers to a spectral detection technology that can simultaneously obtain multiple optical spectral bands (usually more than 3), and can expand in two directions of infrared light and ultraviolet light on the basis of visible light. Compared with RGB images, MSI can contain more channel information, such as ultraviolet, near-infrared or short-wave infrared, etc., such as information of channels including brightness (Y), purple (C), magenta (M), green (P) or visible light (V), etc., so MSI images can contain more color information.

[0065] In addition, for MSI images containing a large number of channels, a hyperspectral image (HSI) can also be defined. Compared with the commonly referred to MSI, the number of spectral bands covered by the HSI far exceeds the conventional MSI, and the HSI can distinguish thousands of individual spectral bands, providing more abundant and detailed spectral information. And the HSI can capture many very narrow bands, the width of which is usually between 10-20 nanometers. This high resolution enables the HSI to more finely analyze the composition and characteristics of a substance.

[0066] (3) White Balance (WB)

[0067] The basic concept is "no matter under any light source, white objects can be restored to white", and the color cast phenomenon that occurs when shooting under a specific light source is compensated by strengthening the corresponding complementary color. White balance is usually a measure of the accuracy of white color generated by mixing red, green and blue three primary colors, which can solve a series of problems of color restoration and tone processing, and ensure that the camera image can accurately reflect the color condition of the object being photographed.

[0068] (4) Image Signal Processor (ISP)

[0069] It is used to process the image signal output by the image signal sensor. It plays a core and dominant role in the camera system and is an important device to constitute the camera. Its main functional characteristics include: demosaicing, auto exposure, auto white balance, lens shading correction, gamma correction, color space conversion, dynamic range correction, image cropping, etc.

[0070] (5) Spatial / Spectral Resolution

[0071] H x W x C is used to represent an image, where H x W represents the spatial resolution of the image, and C represents the spectral resolution of the image. For example, a common 3-channel color image has a spectral resolution of 3, while for a multispectral image, its spectral resolution C > 3.

[0072] (6) Raw data: Raw data records the original information of the camera sensor, which is in an unprocessed and uncompressed format. RAW can be conceptualized as "original image encoding data" or more vividly referred to as "digital negative".

[0073] (7) Color Difference

[0074] Also known as color distance, is a concern in color science. It quantifies a concept. Color difference can be calculated simply by Euclidean distance in color space (such as angular error (AE)), or using the more complex, uniform human perception formula of the International Commission on Illumination (such as Delta E (dE)).

[0075] (8) Color Constancy

[0076] It refers to the perceptual characteristic that when the color light illuminating the surface of an object changes, people's perception of the color of the object surface remains unchanged. In the field of image processing science, based on the cognitive characteristics of the human eye to the scene, separating the background elements and lighting elements in the image is the key to solving the problem.

[0077] From the deployment manner of the method provided in the application, the method provided in the application can be divided into various deployment manners. For example, the method provided in the application can be deployed in an electronic device, and a user can directly use the electronic device for image processing, or it can also be deployed in the cloud to provide image processing services for user terminals. Different deployment manners will be introduced below.

[0078] Deployment manner one, deployed in an electronic device

[0079] The electronic device provided in the embodiment of the application can specifically include a handheld device, a vehicle-mounted device, a data processing device, a computing device, and an electronic device including or connected with an image sensor. It can also include a digital camera, a cellular phone, a camera, a smart phone, a personal digital assistant (PDA) computer, a tablet computer, a laptop computer, a machine type communication (MTC) terminal, a point of sales (POS), a vehicle-mounted computer, a head-mounted device, a data processing device (such as a bracelet, a smart watch, etc.), a security device, a virtual reality (VR) device, an augmented reality (AR) device, and other electronic devices with imaging functions.

[0080] Taking a digital camera as an example, a digital camera is a digital camera, which is a kind of camera that converts optical image into digital signal by using photoelectric sensor. Unlike traditional cameras that rely on the change of photosensitive chemical substances on film to record images, the sensor of a digital camera is a light-sensitive charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS). Compared with traditional cameras, digital cameras have the advantages of convenience, speed, repeatability, timeliness and other advantages due to the direct use of photoelectric conversion image sensor. With the development of CMOS processing technology, the function of digital camera is becoming more and more powerful, and it has almost completely replaced traditional film camera, and has extremely wide application in consumer electronics, human-computer interaction, computer vision, automatic driving and other fields.

[0081] Exemplarily, FIG. 1 shows a schematic diagram of an electronic device provided by the present application. As shown in the figure, the electronic device can include a lens group 110, an image sensor 120 and an electrical signal processor 130. The electrical signal processor 130 can include an analog-to-digital (A / D) converter 131 and a digital signal processor 132. The analog-to-digital converter 131 is an analog signal to digital signal converter, which is used to convert analog electrical signal to digital electrical signal.

[0082] It should be understood that the electronic device shown in FIG. 1 is not limited to including the above devices, and can also include more or less other devices, such as battery, flash, key, sensor, etc. The embodiments of the present application only take the electronic device with image sensor 120 as an example for description, but the elements installed on the electronic device are not limited thereto.

[0083] In the embodiments of the present application, the aforementioned image sensor 120 can specifically include an image sensor, a multispectral image (MSI) sensor, etc.

[0084] The light signal reflected by the photographed object is converged through the lens group 110 and imaged on the image sensor 120. The image sensor 120 converts the light signal into an analog electrical signal. The analog electrical signal is converted into a digital electrical signal by the analog-to-digital (A / D) converter 131 in the electrical signal processor 130, and the digital electrical signal is processed by the digital signal processor 132, for example, the data electrical signal is optimized by a series of complex mathematical algorithm operation, and finally the image is output. The electrical signal processor 130 can also include an analog signal preprocessor 133, which is used to pre-process the analog electrical signal transmitted by the image sensor and output to the analog-to-digital converter 131.

[0085] The performance of the image sensor affects the quality of the final output image. The image sensor, also known as a photosensitive chip, a photosensitive element, etc., contains hundreds of thousands to millions of photoelectric conversion elements that generate electric charges when exposed to light and are converted into digital signals by an analog-to-digital converter chip. The image sensor includes a plurality of photosensitive elements that achieve imaging through photoelectric response.

[0086] The MSI sensor can simultaneously collect image signals of multiple spectral bands, and the multi-spectral bands contain more frequency bands, and the MSI image contains more color information, which has the potential to improve the effect of RGB images or videos. Therefore, the method provided in the present application can be deployed in the electrical signal processor 130 of the electronic device, such as the digital signal processor 132, or other processors of the electronic device. The method provided in the present application can use multi-spectral images to adjust the color of RGB images in a multi-camera scene, so as to use the richer color information contained in the multi-spectral images to adjust the color of the RGB images, so that the image color of the multi-camera RGB image is more accurate and more consistent.

[0087] Further, the digital signal processor can specifically include a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU), or an application specific integrated circuit (ASIC), etc.

[0088] Deployment mode two, deployed in the cloud

[0089] The present application also provides a cloud platform, and one or more terminals access the platform. The method provided in the present application can be deployed in the cloud to provide image denoising or enhancement services for the terminal.

[0090] For example, FIG. 2 is an application scenario of the method provided in the present application, which can include a cloud platform 11 and a terminal 12. The cloud platform 11 and the terminal 12 can be connected through wired or wireless connection.

[0091] The cloud platform 11 can specifically include a server cluster with storage and processing functions. The method provided in the present application can be deployed in the cloud platform 11, which can specifically receive multiple frames of images from the terminal 12 and perform image processing based on the multiple frames of images, such as adjusting the color of the RGB image in combination with the multi-spectral image, and feeding back the processed image to the terminal 12.

[0092] The terminal 12 can implement image processing by interacting with the cloud 11. The terminal can specifically include, but is not limited to, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like. The terminal 12 can transmit an image to the cloud 11, which can be an image captured by the terminal itself, an image including user input, or an image stored locally by the terminal, and the like. For example, the cloud can provide services for users through a client deployed in the terminal or a web page in the terminal, and taking the deployment of the client in the terminal as an example, the user can send the image collected by the terminal to the cloud through the client deployed in the terminal, such as transmitting an RGB image and an MSI image, and the cloud 11 adjusts the color of the RGB image through the method provided in the application, such as performing white balance or color mapping based on the MSI image, and outputs a high-definition RGB image and feeds back the terminal 12.

[0093] In a possible scenario, it can also be applied to a scenario of multiple terminals, for example, the user can use other terminals different from the terminal 12 to capture the RGB image and the MSI image, and transmit the RGB image and the MSI image to the terminal 12, and the terminal 12 uploads the RGB image and the MSI image to the cloud 11 for image denoising or enhancement processing, and then feeds back the enhanced image to the terminal 12, and the terminal 12 feeds back the enhanced image to the terminal that captures the RGB image and the MSI image.

[0094] In electronic devices, such as cameras and smart phones, a color image sensor is usually mounted to capture an RGB image. A common color imaging system uses a color filter array (CFA), also known as a Bayer filter, arranged above the light-sensitive area. For example, a common arrangement of the CFA consists of a 2x2 four-pixel point to form a cycle unit. This unit contains three primary colors: red (R), green (G), and blue (B), so the color image sensor is also commonly referred to as an RGB image sensor.

[0095] RGB sensors aim to directly mimic the responses of the three types of cone cells (LMS) of the human eye to the light spectrum, but due to the deviation of the sensor response curve from the LMS cell response curve, the original sensor response (i.e. RAW image signal) usually needs to be color corrected to ensure that the colors of the image can be more accurately reproduced to what the human eye sees in the shooting scene. In addition, due to the color constancy of the human eye, the RAW image response usually also needs to be corrected through white balance and the like. However, the response of the RGB sensor is a sampling (or integration) of the complete spectral information, which cannot completely capture the spectral information and is prone to metamerism problems, resulting in limited color reproduction performance of existing RGB cameras in some difficult example scenes, such as scenes containing large-area pure color regions.

[0096] Especially in multi-camera scenes, conventional multi-camera is to use array cameras, the main difference of these cameras is the focal length, mainly divided into ultra-wide angle, wide angle (main camera) and long focal length. These cameras can all independently image, and all have the need for photographic records, and are directly presented to the user for browsing, so the user has high requirements for the color consistency of each camera of the array camera. The color difference of different cameras in the same scene will seriously affect the user experience. The existing consistency color processing scheme in multi-camera scenes usually first obtains the main camera color processing parameters. The "low-dimensional" mapping relationship between the main camera and the wide angle / long focal length camera is calculated, and the color parameters of the main camera are mapped to the wide angle / long focal length. That is, based on the main camera color, the color of the main camera is migrated to the wide angle / long focal length camera. In some scenes, the colors of the main camera image and the long focal length image may not be consistent, such as the main camera imaging image is basically normal, but the long focal length imaging image may be purple red, causing multi-camera imaging inconsistency, resulting in poor user experience.

[0097] And for the problem of multi-camera color consistency, in some solutions, first, the main camera raw is used to obtain the main camera color processing parameters; the mapping relationship in the low-dimensional space between the main camera and the wide angle / long focal length camera is calculated through the main camera / long focal length raw, and the color parameters of the main camera are mapped to the wide angle / long focal length; finally, the wide angle / long focal length RGB imaging image is produced through the ISP processing unit through the color processing parameters of the wide angle / long focal length. In this scheme, the baseline color is obtained through the main camera, and due to the phenomenon of metamerism, there may also be a problem of multi-camera color inconsistency. Moreover, the mapping relationship is the 3D space of the main camera RGB to the 3D space of the wide angle / long focal length RGB, which has a low dimension, and the improvement of the processing effect of multi-camera consistency is not obvious.

[0098] For example, in some solutions, the color parameters of the main camera can be calculated by an automatic white balance algorithm, and then the color auxiliary information can be extracted by the multispectral camera to determine the correspondence between the main camera and the wide-angle / telephoto camera, and the color of the wide-angle / telephoto camera is adjusted based on the correspondence. Although the multispectral image is combined in this solution, the multispectral is used to calculate the auxiliary imaging information, and the mapping relationship between the main camera and the wide-angle / telephoto camera is obtained through the auxiliary imaging information. The mapping matrix is still a mapping in a low-dimensional space, and the improvement of the processing effect of the multi-camera consistency is not obvious.

[0099] Therefore, the embodiments of the present application provide an imaging method for performing consistent color adjustment on multi-cameras by using a multispectral image, so that the color consistency effect under the multi-camera scene is better.

[0100] The method flow provided by the embodiments of the present application is introduced below.

[0101] Referring to FIG. 3, a flowchart of an imaging method provided by the embodiments of the present application is as follows.

[0102] 301, a plurality of images are collected by a plurality of image sensors.

[0103] Each image sensor can collect a first image, and the plurality of images are images taken under the same scene. In combination with the application scenarios described above, the method provided by the embodiments of the present application can be applied to an electronic device, which can include a plurality of image sensors. The plurality of image sensors can be used to collect a plurality of images in the same short time period (such as a time period in which the scene change or the position change of the electronic device is less than a certain value, such as within 0.1 m or within 0.2 m, etc.), or at the same time.

[0104] For example, the plurality of image sensors include a first image sensor and a second image sensor. The first image is collected by the first image sensor, and the second image is collected by the second image sensor. In the subsequent processing flow of the embodiments of the present application, the processing flow of the first image and the second image is similar. To reduce redundancy, some steps are described below by taking the processing flow of the first image as an example. The processing flow of the image collected by other image sensors can be referred to the processing flow of the first image.

[0105] In addition, the electronic device can further include more image sensors, such as a third image sensor, or a third image sensor and a fourth image sensor, etc. The number of image sensors included in the electronic device can be determined according to the actual application scenario.

[0106] The plurality of image sensors can specifically include, but are not limited to, a main camera (which can be referred to as a main camera for short) in an electronic device, a wide-angle camera, an ultra-wide-angle camera, an ultra-long-focus camera, or a long-focus camera, and the like, which can be used to capture RGB images. The types of cameras can generally be divided according to the focal length, and are generally divided into ultra-wide-angle, wide-angle, standard, medium-focus, medium-long-focus, long-focus, and ultra-long-focus focal lengths. Cameras that capture different focal lengths can be named by focal length, such as ultra-wide-angle camera, wide-angle camera, standard camera, medium-focus camera, medium-long-focus camera, long-focus camera, and ultra-long-focus, and the focal length of the lens is generally converted from the focal length of a 35mm camera (i.e., a 135 camera).

[0107] Generally, for the images captured by the plurality of cameras, the images can be respectively displayed in the display interface of the electronic device, or the captured images can be fused and displayed for the user in the display interface. Therefore, improving the color consistency of the plurality of images is very important for user experience.

[0108] Further, the plurality of images can specifically be raw data. For example, if the method provided by the embodiments of the present application is deployed in the aforementioned digital signal processor 132, the plurality of images can be digital signals output by the analog-to-digital converter 131, that is, raw data.

[0109] 302, capture a multi-spectral image by a multi-spectral sensor.

[0110] The multi-spectral image can be captured by an MSI sensor, and in some scenarios, the MSI mentioned in the embodiments of the present application can also include an HSI sensor that can be used to capture a multi-spectral image. Generally, the number of channels of the multi-spectral image is greater than the number of channels of the first image captured by the image sensor.

[0111] The multi-spectral image can be captured at the same time as the plurality of images, or can be captured in the same short time period, such as a time period in which the scene changes or the position of the electronic device changes by less than a certain value. That is, at the same time, the first image is captured by the first image sensor, the second image is captured by the second image sensor, and the multi-spectral image is captured by the multi-spectral sensor. For example, within 0.1 m or 0.2 m, etc. That is, the multi-spectral image and the plurality of images can be obtained by photographing the same photographing scene.

[0112] Similarly, the multi-spectral image can also be raw data. For example, if the method provided by the embodiments of the present application is deployed in the aforementioned digital signal processor 132, the multi-spectral image can be a digital signal output by the analog-to-digital converter 131, that is, raw data.

[0113] 303, obtain a plurality of high-dimensional color mapping parameters according to the multi-spectral image.

[0114] The plurality of high-dimensional color mapping parameters include mapping relationships between colors of the multispectral image and colors of each image. That is, each image sensor corresponds to a high-dimensional color mapping parameter.

[0115] For example, according to the multispectral image, a first high-dimensional color mapping parameter and a second high-dimensional color mapping parameter are obtained. The first high-dimensional color mapping parameter includes a mapping relationship between colors of the multispectral image and colors of the first image. The second high-dimensional color mapping parameter includes a mapping relationship between colors of the multispectral image and colors of the second image.

[0116] Specifically, when calculating the high-dimensional color mapping parameter, the color temperature of the multispectral image can be used for calculation, or the high-dimensional white point, the light source reflection spectrum, or the light source spectrum can be used for calculation. For example, color information can be obtained from the multispectral image. The color information includes at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum. Then, the high-dimensional color mapping parameter is calculated according to the color information. For the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter, color information can be obtained from the multispectral image. The color information includes at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum. The first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter are calculated according to the color information.

[0117] Therefore, in the embodiments of the present application, when calculating the high-dimensional color mapping parameter corresponding to each image sensor, the color temperature, the white point, the light source reflection spectrum, or the light source spectrum can be used for calculation to obtain an accurate high-dimensional color mapping parameter.

[0118] In a possible implementation, color information corresponding to N high-dimensional light sources can be obtained from the multi-spectrum image; a first spectral curve of each of the plurality of sensors and a second spectral curve corresponding to the multi-spectrum sensor are obtained; then based on an imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve of each image sensor and the second spectral curve, a first color card image corresponding to each image sensor and a second color card image corresponding to the multi-spectrum sensor are determined, wherein the first image sensor corresponds to the second color card image and the second image sensor corresponds to a third color card image, the imaging model is a model constructed in advance using light sources, colors or spaces; then the high-dimensional color mapping parameters are obtained by fitting the first color card image and the second color card image. Correspondingly, the first spectral curve corresponding to the multi-spectrum sensor, the second spectral curve corresponding to the first image sensor and the third spectral curve corresponding to the third image sensor can be obtained; based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve and the third spectral curve, the first color card image corresponding to the multi-spectrum sensor, the second color card image corresponding to the first image sensor and the third color card image corresponding to the second image sensor are determined; the first high-dimensional color mapping parameters are obtained by fitting the first color card image and the second color card image, and the second high-dimensional color mapping parameters are obtained by fitting the first color card image and the third color card image.

[0119] In the embodiments of the present application, the imaging model and the color information carried in the multi-spectrum image are used to determine the parameters used when projecting the multi-spectrum image to the low-dimensional space corresponding to the image sensor, so that the richer color information carried in the multi-spectrum image can be projected into the low-dimensional space aligned with each image in the subsequent process, so as to adjust the color value of each image based on the projection result in the low-dimensional space, improve the color accuracy of each image, and improve the color consistency in the multi-camera scene.

[0120] In a possible implementation, to improve the accuracy of adjusting the colors of the multi-camera images using the colors of the multi-spectrum image as the base colors, the multi-spectrum image can also be aligned with each image, and more accurate high-dimensional color mapping parameters can be obtained based on the alignment results of each image. For example, the initial high-dimensional color mapping parameters of each image can be obtained according to the multi-spectrum image, and the multi-spectrum image can be aligned with each of the plurality of images respectively to obtain a plurality of alignment results, each image sensor corresponding to an alignment result, and then the initial high-dimensional color mapping parameters corresponding to each alignment result can be updated to obtain a plurality of high-dimensional color mapping parameters respectively, for example, the coordinates or parameter values of each element in the initial high-dimensional color mapping parameters corresponding to an alignment result are corrected, so that the high-dimensional color mapping parameters of each image sensor are closer to the space of each image sensor, thereby improving the color accuracy obtained when the images are color adjusted based on the high-dimensional color mapping parameters. The foregoing process of obtaining the first high-dimensional color mapping parameters and the second high-dimensional color mapping parameters can include obtaining initial first high-dimensional color mapping parameters and initial second high-dimensional color mapping parameters according to the multi-spectrum image; aligning the multi-spectrum image with the initial first high-dimensional color mapping parameters to obtain a first alignment result, and aligning the multi-spectrum image with the initial second high-dimensional color mapping parameters to obtain a second alignment result; updating the initial first high-dimensional color mapping parameters according to the first alignment result to obtain the first high-dimensional color mapping parameters, and updating the initial second high-dimensional color mapping parameters according to the second alignment result to obtain the second high-dimensional color mapping parameters.

[0121] Optionally, after the high-dimensional color mapping parameters corresponding to each image sensor are calculated, the high-dimensional color mapping parameters corresponding to each image sensor can be saved in a register, which can be used to continue to adjust the colors of the RGB images captured by each image sensor using the colors in the multi-spectrum image as the base colors when a multi-spectrum image and RGB images captured by each image sensor are subsequently acquired. For example, after the high-dimensional color mapping parameters corresponding to each image sensor are calculated based on the multi-spectrum image and the plurality of RGB images acquired at time t, the high-dimensional color mapping parameters can be saved in an ISP register, and the multi-spectrum image and the multi-camera RGB images acquired at time t+n, n being an integer greater than 0, can also be adjusted in color based on the high-dimensional color mapping parameters saved in the ISP register using the colors in the multi-spectrum image as the base colors. Therefore, in the embodiments of the present application, the calculated high-dimensional color mapping parameters can be reused, and the imaging efficiency of the imaging system can be improved.

[0122] 304. Color adjust each image according to the plurality of high-dimensional color mapping parameters to obtain a plurality of color-adjusted images.

[0123] Specifically, the first image can be color adjusted according to the high-dimensional color mapping parameter corresponding to each image sensor, so as to adjust the color in the first image by using the color information included in the multi-spectrum image, and obtain the color-adjusted image corresponding to each image. For example, the first image can be color adjusted according to the first high-dimensional color mapping parameter to obtain the third image, and the second image can be color adjusted according to the second high-dimensional color mapping parameter to obtain the fourth image.

[0124] Therefore, in the embodiments of the present application, the color in the multi-spectrum image can be used as the base color to adjust the images collected by the image sensors, so as to adjust the colors of the images collected by the multi-camera based on the colors in the multi-spectrum image, make the colors of the multiple images collected by the multi-camera more consistent, and achieve better color consistency effect. Taking the first image collected by the first image sensor and the second image collected by the second image sensor as an example, the first high-dimensional color mapping parameter corresponding to the first image and the second high-dimensional color mapping parameter corresponding to the second image are determined based on the multi-spectrum image, the color of the first image is adjusted based on the first high-dimensional color mapping parameter to obtain the third image, and the color of the second image is adjusted based on the second high-dimensional color mapping matrix to obtain the fourth image, so that the third image and the fourth image both use the color of the multi-spectrum image as the base color, and therefore the colors of the third image and the fourth image have a certain degree of consistency.

[0125] Specifically, the data of the multi-spectrum image can be projected into the space corresponding to each image sensor according to the multiple high-dimensional color mapping parameters to obtain multiple multi-spectrum low-dimensional projection results; the color adjustment parameter of each image sensor is determined according to the multiple multi-spectrum low-dimensional projection results, and the data of each image is color adjusted, such as white balance or color mapping, according to the color adjustment parameter of each image sensor to obtain the multiple color-adjusted images. Taking the processing process of the first image and the second image as an example, the data of the multi-spectrum image is projected into the space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain the first multi-spectrum low-dimensional projection result, the first color adjustment parameter of the first image sensor is determined according to the first multi-spectrum low-dimensional projection result, and the data of the first image is color adjusted according to the first color adjustment parameter to obtain the third image.

[0126] Therefore, in the embodiments of the present application, in order to facilitate the projection of the color information in the multispectral image into the image captured by the image sensor, the information in the multispectral image can be first projected into the space corresponding to each image sensor, and then the projection result obtained by projecting the multispectral image into the space corresponding to each image sensor is used to adjust the image captured by each image sensor, so that the color in the multispectral image can be used as a base color to adjust the image captured by the image sensor, the color accuracy in the first image is improved, and the consistency of the images of the multiple image sensors in the multi-camera scene is more optimal.

[0127] In a possible implementation, the high-dimensional white point in the multispectral image can be projected into the space of each image sensor, so that the white balance processing is performed on each image based on the projection result, and the image with more accurate color after the white balance processing is obtained. Specifically, the high-dimensional white point in the multispectral image can be obtained; then the high-dimensional white point is mapped to the space corresponding to each image sensor according to the high-dimensional color mapping parameter corresponding to each image sensor, and a plurality of low-dimensional white points are obtained, and the multispectral low-dimensional projection result includes the plurality of low-dimensional white points; then the white balance coefficient of each image sensor is obtained according to the plurality of low-dimensional white points, and the color adjustment parameter can include the white balance coefficient of each image sensor; and the white balance processing is performed on each image according to the white balance coefficient of each image sensor, and the plurality of images after color adjustment are obtained. For the first image, for example, the data of the multispectral image is projected into the space corresponding to the first image sensor according to the first high-dimensional color mapping parameter, and the first multispectral low-dimensional projection result is obtained, the first color adjustment parameter of the first image sensor is determined according to the first multispectral low-dimensional projection result, and the data of the first image is color adjusted according to the first color adjustment parameter, and the third image is obtained.

[0128] Therefore, in the embodiments of the present application, the high-dimensional white point in the multispectral image can be projected into the low-dimensional space of each image sensor, and the parameter for adjusting each image is determined based on the low-dimensional white point after the projection, so that the white balance processing is performed on each image with the high-dimensional white point in the multispectral image as a base color, the color accuracy of the image of each image sensor is improved, and the consistency of the images of the multiple image sensors in the multi-camera scene is more optimal.

[0129] In a possible implementation, the colors in the multi-spectrum image can also be projected into the space of each image sensor to obtain a low-dimensional color projection image corresponding to each image sensor, and the colors in the low-dimensional color projection image can be mapped into the corresponding first image to map the colors in the multi-spectrum image into the images of each image sensor, so that the color accuracy is improved and the color consistency between the multi-camera images is better. Specifically, the multi-spectrum image can be projected into the space corresponding to each image according to the high-dimensional color mapping parameter to obtain a plurality of low-dimensional color projection images, and the multi-spectrum low-dimensional projection result includes the plurality of low-dimensional color projection images; the low-dimensional color mapping parameter corresponding to each image sensor can be determined according to the multi-spectrum low-dimensional color projection image, and the color adjustment parameter includes the low-dimensional color mapping parameter corresponding to each image sensor; then the colors in the low-dimensional color projection image can be mapped into the first and image according to the low-dimensional color mapping parameter to obtain the plurality of images after color adjustment. For example, the multi-spectrum image can be projected into the space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain the first low-dimensional color projection image, the first multi-spectrum low-dimensional projection result includes the first low-dimensional color projection image; the first low-dimensional color mapping parameter corresponding to the first image sensor can be determined according to the multi-spectrum low-dimensional color projection image, and the first color adjustment parameter includes the first low-dimensional color mapping parameter; the first image can be color-mapped according to the first low-dimensional color mapping parameter to obtain the third image.

[0130] Therefore, in the embodiments of the present application, the colors in the multi-spectrum image can be mapped into the images captured by the image sensors, so that the color accuracy of each image sensor is improved, and after the colors in the multi-spectrum image are mapped into the images of the plurality of image sensors, the color consistency of each image sensor image is better.

[0131] In addition, in a possible implementation, the colors in the multi-spectrum image can also be projected into a standard space; when the low-dimensional color mapping parameter is determined, the low-dimensional color mapping parameter corresponding to each image sensor can be determined based on the low-dimensional color projection image in the space of each image sensor and the standard space projection image. Specifically, the third high-dimensional color mapping parameter can be obtained; the color value of the multi-spectrum image can be projected into the standard space according to the third high-dimensional color mapping parameter to obtain the standard space projection image; then the low-dimensional color mapping parameter corresponding to each image sensor can be determined according to the multi-spectrum low-dimensional image and the standard space projection image, for example, the first low-dimensional color mapping parameter corresponding to the first image can be determined according to the multi-spectrum low-dimensional image and the standard space projection image. Therefore, the accuracy of mapping the colors in the multi-spectrum image into the images captured by each image sensor can be improved by combining the standard space projection image, and the image color consistency in the multi-camera scene is further improved.

[0132] In a possible implementation, in order to further obtain a multi-camera image with better consistency, a neural network can also be used to output a parameter for adjusting the multi-camera image with the color of the multispectral image as a base color. For example, a pre-trained color mapping network can be set, and a color mapping matrix is obtained through the color mapping network, that is, a parameter for adjusting the multi-camera image with the color of the multispectral image as a base color; then the color adjustment parameter is updated according to the color mapping matrix to obtain an updated color adjustment parameter, and each image is color adjusted according to the updated color adjustment parameter to obtain a plurality of color-adjusted images. For example, the first color adjustment parameter is updated according to the color mapping matrix to obtain an updated first color adjustment parameter; and the first image is color adjusted according to the updated first color adjustment parameter to obtain a third image. Therefore, in the implementation of the present application, the mapping capability of the neural network can be combined to adjust the multi-camera image with the color of the multispectral image as a base color, and the color accuracy and consistency of the image in the multi-camera scene are further guaranteed.

[0133] The foregoing introduces the flow of the method provided by the embodiments of the present application. In the following, the method flow provided by the embodiments of the present application is introduced in more detail in combination with a specific application scenario.

[0134] The flow of the method provided by the embodiments of the present application can be divided into multiple modules, or the imaging system provided by the embodiments of the present application can be divided into multiple modules, for example, as shown in FIG. 4, which can be divided into an input stage 401, a consistency processing stage 402, a color adjustment stage 403, and the like. In combination with the imaging system provided by the embodiments of the present application shown in FIG. 5, in the input stage, the raw data of the RGB image and the multispectral image are collected by using multiple image sensors and MSI respectively, in the consistency processing stage, the high-dimensional color mapping parameter is calculated by using the high-dimensional color information in the multispectral image, such as color temperature, high-dimensional white point, light source reflection spectrum or light source spectrum, and the high-dimensional color in the multispectral image is projected into a low-dimensional space by using the parameter, such as projecting the high-dimensional white point or the high-dimensional color, and the color adjustment parameter of the RGB image is calculated based on the projection result; in the color adjustment stage, the color of each RGB can be adjusted based on the color adjustment parameter calculated in the consistency processing stage, so as to adjust the color of the RGB image with the color of the multispectral image as a baseline color, and improve the color accuracy and consistency of each RGB image.

[0135] In the embodiments of the present application, multiple processing methods can be set in each stage, for example, in the consistency processing stage, the high-dimensional white point, the high-dimensional color or the combination thereof in the multispectral image can be projected, and a neural network can also be used to output a possible color adjustment parameter, and the like, which will be introduced in the following.

[0136] Implementation one, white balance processing

[0137] Referring to FIG. 6, a flowchart of another imaging method provided by the embodiments of the present application is shown.

[0138] 601, input.

[0139] The input data includes multiple types, which are classified into multispectral raw data and image sensor raw data (i.e., the first image mentioned above) in the embodiments of the present application. The multiple image sensors mentioned above can be one or more of a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera, or an ultra-long-focus camera. For example, the embodiments of the present application are described by taking the image sensors including a main camera, a wide-angle camera, and a long-focus camera as an example. The main camera, the wide-angle camera, and the long-focus camera mentioned in the embodiments of the present application can also be replaced by other combinations of multiple types of sensors, which can be determined according to actual application scenarios, and the present application does not limit this.

[0140] Optionally, the RGB image raw data can be preprocessed, such as filtering noise or brightness correction, to obtain preprocessed RGB image raw data.

[0141] The multispectral image raw data can also be processed by a multispectral processing unit (MSPU) to extract high-dimensional color information, including light source color temperature, high-dimensional white point, light source reflection spectrum, or light source spectrum, etc. For example, the multispectral sensor can capture information on different spectral bands, including different frequency bands of visible light, infrared light, or ultraviolet light, etc. After processing by the MSPU, each spectral band is taken as a dimension, and the color information in multiple dimensions, i.e., high-dimensional color information, is extracted.

[0142] Optionally, the multispectral image raw data can be preprocessed, such as filtering noise or brightness correction, to obtain preprocessed multispectral raw data.

[0143] 602, consistency processing.

[0144] In the consistency processing stage, the high-dimensional color information extracted from the multispectral image is first used to calculate high-dimensional color mapping parameters in combination with the multispectral image, and then the high-dimensional color information of the multispectral image is projected into the space of each image sensor based on the high-dimensional color mapping parameters to obtain multispectral projection results; and then the color adjustment parameters are calculated by using the projection results. That is, the dynamic color mapping parameters are obtained by the high-dimensional color information (color temperature information), the high-dimensional color mapping parameters are optimized in real time by the multispectral image and the main camera, the wide-angle camera, and the long-focus camera, and the high-dimensional color information is projected into the low-dimensional color information of the main camera, the wide-angle camera, and the long-focus camera by the dynamic color mapping parameters.

[0145] Exemplarily, in the embodiments of the present application, the high-dimensional white point in the multi-spectrum image is projected to the low-dimensional space of each image sensor. Under N high-dimensional light sources, a mapping parameter table is calibrated respectively: the light source spectrum under different high-dimensional light sources, the reflectance spectrum of the 24-color card, the spectral corresponding curves of the multi-spectrum camera and the main camera, the wide-angle camera and the long-focus camera are obtained; according to the imaging model, the virtual color card images of the multi-spectrum camera and the main camera / wide-angle camera / long-focus camera are calculated respectively from the spectral corresponding curves, the light source spectrum and the virtual color card reflectance spectrum; the high-dimensional color mapping matrix T of the multi-spectrum camera and the main camera / wide-angle camera / long-focus camera is fitted respectively by the least square method.

[0146] For example, N high-dimensional color information (high-dimensional color temperature or high-dimensional white point, etc.) is used to obtain the corresponding N high-dimensional color mapping parameters. The light source under different scenes may be different, such as can be divided into shop window spotlight (A), sunlight (D50), blue sky daylight (D65) and the like, and the light reflected by the object under different light sources may be different, so different light sources can be distinguished, color correction is performed under each light source, and more accurate color can be obtained.

[0147] The high-dimensional color mapping relationship, that is, the high-dimensional color mapping parameter can include a linear mapping matrix, a nonlinear mapping matrix or a neural network, etc. The low-dimensional image of the embodiments of the present application takes RGB as an example, and the high-dimensional multi-spectrum image takes RGBYPVCM as an example.

[0148] For example, the linear transformation matrix T can be represented as:

[0149] The nonlinear matrix T can be represented as:

[0150] Under N high-dimensional light sources, the high-dimensional color mapping matrix is calibrated respectively. Specifically, the light source spectrum under different high-dimensional light sources, the reflectance spectrum of the 24-color card, or the color temperature, etc. can be obtained offline. The spectral curves of the multi-spectrum camera and the main camera, the wide-angle camera or the long-focus camera are measured using a measuring instrument, including the first spectral curve, the second spectral curve and the third spectral curve mentioned above; the virtual color card images of the multi-spectrum sensor and the main camera, the wide-angle camera or the long-focus camera image sensor are obtained according to the imaging model from the spectral corresponding curves, the light source spectrum and the virtual color card reflectance spectrum; then the multi-spectrum virtual color card image and the main camera, the wide-angle camera or the long-focus camera virtual color card image are fitted, such as fitting by the least square method or other fitting algorithms, to obtain the high-dimensional color mapping matrix of the main camera the high-dimensional color mapping matrix of the wide-angle camera the high-dimensional color mapping matrix of the long-focus camera The high-dimensional color mapping matrix of each camera is represented as That is, the high-dimensional color mapping matrix corresponding to the main camera, the wide-angle camera or the long-focus camera respectively.

[0151] Subsequently, the high-dimensional color mapping parameters The high-latitude white points are projected into the low-dimensional space of the main camera, wide-angle camera, or long-focus camera, respectively, to obtain the white points in the low-dimensional space corresponding to the main camera, wide-angle camera, or long-focus camera of the multi-spectral image, which can be referred to as low-dimensional white points. The white balance coefficients Gain m , Gain w , or Gain l of the main camera, wide-angle camera, or long-focus camera are obtained, respectively. For ease of understanding, the white balance coefficients of the main camera, wide-angle camera, or long-focus camera are collectively referred to as the white balance coefficient Gain m / w / l , or when the embodiments of the present application are deployed in an ISP chip, referred to as an ISP register parameter. The ISP register parameter can be saved in an ISP register. When a multi-spectral image and a multi-camera RGB image captured at a time t are used to calculate the ISP register parameter, the color in the multi-spectral image can be projected into the multi-camera RGB space based on the parameter saved in the ISP register when a multi-spectral image and a multi-camera RGB image are captured at a subsequent time t+n. That is, the ISP register parameter obtained can be reused, and there is no need to calculate the ISP register parameter for each time, which can improve the imaging efficiency of the imaging system.

[0152] The white balance coefficients corresponding to each sensor, that is, the first high-dimensional color parameter and the second color mapping parameter mentioned above.

[0153] It can be understood that, in order to restore the human eye's perception of scene color, due to the deviation between the sensor response curve and the LMS cell response curve, the sensor original response (i.e., the RAW image signal) usually needs to be color corrected to ensure that the color of the image can be more accurately reproduced to the situation seen by the human eye in the shooting scene. The sampling (or integration) of complete spectral information by the RGB image sensor cannot completely capture the spectral information, which is prone to cause the metamerism problem, resulting in limited color restoration performance of the existing RGB camera in some difficult example scenes.

[0154] Compared with the RGB color sensor, the multi-spectral sensor aims to more completely capture the original spectral signal. For example, above the gray light-sensitive area, a cycle unit is composed of 4x4 sixteen pixel points, and each pixel point corresponds to a color filter (or color filter coating). Through this design, the resolution of the sensor to the spectrum is improved from 3 dimensions to 16 dimensions, which can more completely capture the spectral signal, thereby improving the image color restoration performance in difficult example scenes. In the embodiments of the present application, the baseline color information is derived from the multi-spectral camera high-dimensional information through the multi-spectral camera "high-dimensional" color information, which has higher accuracy than the color of the RGB sensor as the baseline color in the existing scheme.

[0155] Therefore, in the embodiments of the present application, the multispectral sensor is used to collect more spectral signals of frequency bands, i.e., high-dimensional color information, and an imaging model is used to determine the mapping relationship of the high-dimensional color information in the low-dimensional space of the RGB sensor, i.e., the high-dimensional color mapping relationship, which is the aforementioned high-dimensional color mapping parameter. Based on the high-dimensional color mapping relationship, the color information in the multispectral image, such as high-dimensional white points or colors, is projected into the low-dimensional space to obtain a projection result. The projection result forms a projection map of high-dimensional color to low-dimensional color, or a projection value of high-dimensional white point to low-dimensional white point, and the like. Here, the high-dimensional white point is projected into the low-dimensional white point in the low-dimensional space as an example, so that the white point distribution of the white point collected by the multispectral sensor after being projected into the low-dimensional space is determined based on the projection result, so as to output the white balance coefficient based on the white point distribution.

[0156] 603, color adjustment.

[0157] After the white balance coefficient is calculated, the white balance processing can be performed on each RGB image based on the white balance coefficient.

[0158] Specifically, the white balance can be used to adjust the color balance in the image, and can be used to adjust the proportion of red, green and blue three primary colors in the RGB image, so as to ensure that the color of the object under different light sources can be accurately restored, the white object is restored to white, and the color deviation is avoided.

[0159] Generally, the white balance is closely related to the color temperature, and the multispectral image usually contains more color information. Therefore, the high-dimensional color information in the multispectral image is used to calculate the white balance coefficient, which is equivalent to balancing the high-dimensional color information in the multispectral image as the base color, so that the color in the RGB image can be more accurately balanced, the color in the RGB image can be more accurately restored, and the color consistency in the multi-camera scene is improved.

[0160] Embodiment two, aligning the RGB image and the multispectral image, and white balance processing

[0161] Referring to FIG. 7, the flowchart of another imaging method provided by the embodiments of the present application is shown.

[0162] 701, input.

[0163] 702, consistency processing.

[0164] The steps 701 and 702 are similar to the aforementioned steps 601 and 602, and the similar parts will not be described here. Some differences will be introduced below.

[0165] Specifically, the multispectral image RAW data is processed by the MSU to obtain scene color temperature, multispectral high-dimensional white point and the like, and the initial high-dimensional color mapping parameters of the main camera, wide-angle camera or long-focus camera are obtained through the scene color temperature

[0166] The multispectral image and the RAW of the main camera, wide-angle camera or long-focus camera are aligned to obtain an aligned image, and the initial high-dimensional color mapping parameters are updated through an optimization algorithm to obtain updated high-dimensional color mapping parameters Subsequently, the high-dimensional color mapping parameters can be used to project the high-latitude white point to the low-dimensional space of the main camera, wide-angle camera or long-focus camera to obtain a low-dimensional white point in the main camera, wide-angle camera or long-focus camera space of the multispectral image, and the white balance coefficients Gain of the main camera, wide-angle camera or long-focus camera are calculated respectively m / w / l .

[0167] The optimization algorithm can specifically include adjusting the positions or coordinates of the elements in the matrix through the alignment result, so that the color mapping parameters are more adapted to the actual space of the RGB image, and the color accuracy of the obtained RGB image is improved.

[0168] 703、color adjustment.

[0169] The step 703 is similar to the step 603, and details can be referred to the description of the step 603.

[0170] Therefore, in the embodiments of the present application, the multispectral image and the RGB image can be aligned, and the high-dimensional color mapping parameters are updated based on the alignment result, so that the elements in the high-dimensional color mapping parameters are more adapted to the space of the RGB image, the adaptability of the obtained high-dimensional color mapping parameters to the RGB space is improved, and the color accuracy of the subsequently output RGB image is improved.

[0171] Embodiment three, color mapping

[0172] Referring to FIG. 8, the present application provides another imaging method flow chart.

[0173] 801、input.

[0174] 802、consistency processing.

[0175] The steps 801 and 802 are similar to the steps 601 and 602, and details are not repeated here. Some differences are introduced below.

[0176] As shown in FIG. 8, the multispectral image RAW data is processed by the MSU to obtain scene color temperature, multispectral high-dimensional white point and the like, and the initial high-dimensional color mapping parameters of the main camera, wide-angle camera or long-focus camera are obtained through the scene color temperature​ i.e. the first high-dimensional color mapping parameter or the second high-dimensional color mapping parameter mentioned above. And the high-dimensional color mapping parameter of projecting the multi-spectrum image into the standard color space P3 / sRGB i.e. the third high-dimensional color mapping parameter mentioned above.

[0177] by the high-dimensional color mapping parameter projecting the multi-spectrum image into the low-dimensional space of the main camera, wide-angle or telephoto, to obtain the low-dimensional projection image of the multi-spectrum image in the space of the main camera, wide-angle or telephoto, collectively referred to as MSC2RGB m / w / l by the high-dimensional color mapping parameter projecting the multi-spectrum image into the standard color space P3 / sRGB to obtain the standard space projection image MSC2RGB p3 / sRGB , and then fitting the low-dimensional projection MSC2RGB m / w / l of the main camera, wide-angle or telephoto and the standard space projection image MSC2RGB p3 / sRGB , such as fitting by least squares, to obtain the color mapping matrix corresponding to the main camera, wide-angle or telephoto, collectively referred to as Each element in the matrix can include the weight of the RGB channel respectively, which is equivalent to projecting the color value of each spectrum in the multi-spectrum image into the RGB channel.

[0178] The difference from the foregoing step 602 is that in the foregoing step 602, only the white balance coefficient needs to be calculated, and there is no need to directly map the color in the multi-spectrum image to the RGB image, only the proportion of each color value needs to be calculated. In step 802, the color in the multi-spectrum image needs to be converted into the color representation in the standard RGB space, so as to map the color in the multi-spectrum image to the multi-camera RGB image in the dimension of the RGB space, realizing the conversion of multi-spectrum color to RGB color.

[0179] 803, color adjustment.

[0180] Compared with the foregoing step 603, in the embodiment of the application, the RGB image is color mapped based on the color mapping matrix . Specifically, the color correction matrix (CCM) corresponding to the RGB image can be adjusted based on the matrix Each element in the matrix can represent the weight of a color channel, such as directly taking as the CCM of the main camera, wide-angle or telephoto camera, or weighting and fusing with the initial CCM of the main camera, wide-angle or telephoto camera, where Set a larger weight, so that the color value in the multispectral image as the RGB image as the base color.

[0181] Therefore, in the embodiments of the present application, the colors of the multispectral image can be projected into the low-dimensional space of each main camera, wide-angle camera or long-focus camera, and the standard color space, so as to fit the colors in the multispectral image into the color values in the RGB image, adjust the colors of the RGB image with more rich and accurate colors in the multispectral image as the base color, improve the color accuracy of the output RGB image, and improve the image color consistency in the multi-camera scene.

[0182] Embodiment four, aligning the RGB image and the multispectral image, and color mapping

[0183] Referring to FIG. 9, the present application provides another flowchart of the imaging method.

[0184] 901, input.

[0185] 902, consistency processing.

[0186] Among them, step 901 and step 902 are similar to the aforementioned step 801 and step 802, for the similar places, this place will not be repeated, and the following will be introduced.

[0187] Similarly to the aforementioned embodiment two, before projecting the colors of the multispectral image into the space of the main camera, wide-angle camera or long-focus camera, the calculated high-dimensional color mapping parameter is taken as the initial high-dimensional color mapping parameter At the same time, the multispectral image and the RGB image collected by the main camera, wide-angle camera and long-focus camera can be aligned respectively, and the initial high-dimensional color mapping parameter is updated through the optimization algorithm based on the alignment result, to obtain the updated high-dimensional color mapping parameter Subsequently, the high-dimensional white point can be projected into the low-dimensional space of the main camera, wide-angle camera or long-focus camera through the high-dimensional color mapping parameter, to obtain the low-dimensional white point of the multispectral image in the space of the main camera, wide-angle camera or long-focus camera, and the white balance coefficient Gain of the main camera, wide-angle camera or long-focus camera is calculated respectively m / w / l .

[0188] Among them, the optimization algorithm can specifically include adjusting the position or coordinate of each element in the matrix, so that the color mapping parameter is more adapted to the actual space of the RGB image, and the color accuracy of the final result is improved.

[0189] 903, color adjustment.

[0190] Among them, step 903 is similar to step 803, and can refer to the description of the aforementioned step 803, which will not be repeated here.

[0191] In this embodiment, before projecting the colors of the multispectral image to a low-dimensional space, the RGB image and the multispectral image can be aligned to obtain high-dimensional color mapping parameters that are more compatible with the spaces of each camera, thereby improving the accuracy of projecting the colors in the multispectral image to the spaces corresponding to each camera.

[0192] Implementation Method 5: Calculate color adjustment parameters using a color mapping network.

[0193] Referring to Figure 10, this is a schematic flowchart of another imaging method provided in an embodiment of this application.

[0194] 1001, Input.

[0195] 1002. Consistency processing.

[0196] Step 1002 is similar to steps 602 and 802 above, where the scene color temperature in the multispectral image output by the MSPU is used to obtain the high-dimensional color mapping parameters corresponding to each camera. And high-dimensional color mapping parameters for projecting multispectral images onto the standard color space P3 / sRGB.

[0197] Through high-dimensional color mapping parameters Multispectral images are projected onto the low-dimensional spaces of the main camera, wide-angle lens, or telephoto lens to obtain low-dimensional projection images of the multispectral images in the spaces of the main camera, wide-angle lens, or telephoto lens, respectively. These are collectively referred to as MSC2RGB. m / w / l Through high-dimensional color mapping parameters The multispectral image is projected onto the standard color space P3 / sRGB to obtain the standard spatial projection map MSC2RGB. p3 / sRGB Subsequently, the MSC2RGB low-dimensional projection of the main camera, wide-angle, or telephoto lens was applied. m / w / l and standard spatial projection diagram MSC2RGB p3 / sRGB To perform fitting, for example, using the least squares method, the color mapping matrices corresponding to the main camera, wide-angle, or telephoto lens are obtained respectively; these are collectively referred to as...

[0198] Furthermore, in this embodiment, a color mapping network is added. The multispectral image raw data and the raw data of RGB images captured by each camera are used as inputs to the pre-trained color mapping network, outputting a dynamic color mapping matrix. Subsequently and To achieve fusion, for example, matrix multiplication can be used for the operation, and the result can be... As the final ISP register parameter.

[0199] 1003. Color adjustment.

[0200] In the step 1003, the color adjustment is similar to the step 903, and details can be referred to the description of the step 803.

[0201] Therefore, in the embodiments of the present application, a pre-trained neural network is introduced to project the color information in the multispectral image as a base color into the space corresponding to each camera, and the color adjustment parameter is calculated after the projection combined with the high-dimensional color mapping parameter determined based on the color information, which can further improve the accuracy of the final color adjustment parameter. The color in the multispectral image can be more accurately mapped to the RGB image of each camera, the color accuracy of each RGB image is improved, and the color consistency of the RGB images of multiple cameras is improved.

[0202] In combination with the embodiments provided in the foregoing, it can be understood that, in the method provided by the embodiments of the present application, as shown in FIG. 11, high-dimensional color information can be extracted from the multispectral image and projected into the space of the main camera and the long-focus camera to realize high-dimensional projection, and the high-dimensional color information is used as a base color to adjust the color of the images captured by the main camera and the long-focus camera, so as to improve the color consistency and accuracy of the images captured by the main camera and the long-focus camera.

[0203] The foregoing introduces the method provided by the embodiments of the present application, and the following introduces the device structure for executing the foregoing method.

[0204] Referring to FIG. 12, another architecture of an imaging system provided by the embodiments of the present application is shown, which can be used to execute the method steps shown in FIGS. 3 to 11. The imaging system includes a plurality of image sensors 1201, a multispectral sensor 1202, and a processing unit 1203, wherein the plurality of image sensors includes a first image sensor and a second image sensor.

[0205] The plurality of image sensors 1201 is configured to capture a plurality of images, wherein the first image sensor is configured to capture a first image, and the second image sensor is configured to capture a second image.

[0206] The multispectral sensor 1202 is configured to capture a multispectral image.

[0207] The processing unit 1203 is configured to acquire a high-dimensional color mapping parameter according to the multi-spectrum image, the high-dimensional color mapping parameter representing a mapping relationship between colors of the multi-spectrum image and colors of the plurality of images. Specifically, the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter are acquired according to the multi-spectrum image. The first high-dimensional color mapping parameter includes a mapping relationship between colors of the multi-spectrum image and colors of the first image. The second high-dimensional color mapping parameter includes a mapping relationship between colors of the multi-spectrum image and colors of the second image.

[0208] The processing unit 1203 is further configured to perform color adjustment on the plurality of images according to the high-dimensional color mapping parameter to obtain the plurality of images after color adjustment. Specifically, the first image is color adjusted according to the first high-dimensional color mapping parameter to obtain the third image, and the second image is color adjusted according to the second high-dimensional color mapping parameter to obtain the fourth image.

[0209] In a possible implementation, the processing unit 1203 is specifically configured to: project data in the multi-spectrum image to a space corresponding to each image sensor of the plurality of image sensors according to the high-dimensional color mapping parameter to obtain a low-dimensional projection result; and determine a color adjustment parameter of each image sensor according to the low-dimensional projection result, and adjust the color of each image according to the color adjustment parameter of each image sensor to obtain the plurality of images after color adjustment. Specifically, the data in the multi-spectrum image is projected to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result. The first color adjustment parameter of the first image sensor is determined according to the low-dimensional projection result, and the color of the first image is adjusted according to the first color adjustment parameter to obtain the third image.

[0210] In a possible implementation, the processing unit 1203 is specifically configured to: acquire a high-dimensional white point in the multi-spectrum image; map the high-dimensional white point to a space corresponding to each image of the plurality of images according to the plurality of high-dimensional color mapping parameters to obtain a plurality of low-dimensional white points, the multi-spectrum low-dimensional projection result including the plurality of low-dimensional white points; and acquire a white balance coefficient of each image sensor according to the plurality of low-dimensional white points, the color adjustment parameter including the white balance coefficient of each image sensor; and perform white balance processing on each image according to the white balance coefficient of each image sensor to obtain the plurality of images after color adjustment. Specifically, the high-dimensional white point in the multi-spectrum image is acquired, the high-dimensional white point is mapped to a space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain a first low-dimensional white point, the multi-spectrum low-dimensional projection result includes the first low-dimensional white point, the first white balance coefficient of the first image sensor is acquired according to the first low-dimensional white point, the first color adjustment parameter includes the first white balance coefficient, and the first image is processed according to the first white balance coefficient to obtain the third image.

[0211] In a possible implementation, the processing unit 1203 is specifically configured to: project the multi-spectrum image into a space corresponding to each of the plurality of images according to the high-dimensional color mapping parameter, to obtain a plurality of low-dimensional color projection images, the multi-spectrum low-dimensional projection result including the plurality of low-dimensional color projection images; determine the low-dimensional color mapping parameter of each image sensor according to the multi-spectrum low-dimensional color projection image, the color adjustment parameter including the low-dimensional color mapping parameter of each image sensor; and perform color mapping on the plurality of images according to the low-dimensional color mapping parameter, to obtain the plurality of images after color adjustment, specifically, projecting the multi-spectrum image into a space corresponding to the first image according to the first high-dimensional color mapping parameter, to obtain a first low-dimensional color projection image, the first multi-spectrum low-dimensional projection result including the first low-dimensional color projection image, determining the low-dimensional color mapping parameter of the first image sensor according to the multi-spectrum low-dimensional color projection image, the color adjustment parameter including the first low-dimensional color mapping parameter, and performing color mapping on the first image according to the first low-dimensional color mapping parameter, to obtain a third image.

[0212] In a possible implementation, the processing unit 1203 is further configured to: obtain a third high-dimensional color mapping parameter; project a color value of the multi-spectrum image into a standard space according to the third high-dimensional color mapping parameter, to obtain a standard space projection image; and determine the low-dimensional color mapping parameter corresponding to each image sensor according to the multi-spectrum low-dimensional image and the standard space projection image, specifically, obtaining the third high-dimensional color mapping parameter; projecting the color value of the multi-spectrum image into the standard space according to the third high-dimensional color mapping parameter, to obtain the standard space projection image; and determining the first low-dimensional color mapping parameter according to the multi-spectrum low-dimensional image and the standard space projection image.

[0213] In a possible implementation, the processing unit 1203 is further configured to: obtain a color mapping matrix through the color mapping network; update the color adjustment parameter according to the color mapping matrix, to obtain an updated color adjustment parameter; and perform color adjustment on each image according to the updated color adjustment parameter, to obtain the plurality of images after color adjustment, specifically, obtaining a first color mapping matrix through the color mapping network; updating the first color adjustment parameter according to the first color mapping matrix, to obtain an updated first color adjustment parameter; and performing color adjustment on the first image according to the updated first color adjustment parameter, to obtain a third image.

[0214] In a possible implementation, the processing unit 1203 is specifically configured to: obtain a plurality of initial high-dimensional color mapping parameters according to the multi-spectrum image; align the multi-spectrum image with each of the plurality of images to obtain a plurality of alignment results; and update the plurality of initial high-dimensional color mapping parameters according to each of the alignment results to obtain a plurality of high-dimensional color mapping parameters. Specifically, the initial first high-dimensional color mapping parameter and the initial second high-dimensional color mapping parameter can be obtained according to the multi-spectrum image, the multi-spectrum image is aligned with the initial first high-dimensional color mapping parameter to obtain a first alignment result, the multi-spectrum image is aligned with the initial second high-dimensional color mapping parameter to obtain a second alignment result, the initial first high-dimensional color mapping parameter is updated according to the first alignment result to obtain the first high-dimensional color mapping parameter, and the initial second high-dimensional color mapping parameter is updated according to the second alignment result to obtain the second high-dimensional color mapping parameter.

[0215] In a possible implementation, the processing unit 1203 is specifically configured to: obtain color information from the multi-spectrum image, the color information including at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum, or a light source spectrum; and calculate the first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter according to the color information.

[0216] In a possible implementation, the processing unit 1203 is specifically configured to: obtain color information corresponding to N high-dimensional light sources from the multi-spectrum image; obtain a first spectral curve of each of the plurality of image sensors and a second spectral curve corresponding to the multi-spectrum sensor; determine a first color card image corresponding to the multi-spectrum sensor and a color card image corresponding to each of the sensors based on an imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve of each of the sensors, and the second spectral curve, wherein a second color card image corresponding to the first image sensor and a third color card image corresponding to the second image sensor; and fit the first high-dimensional color mapping parameter according to the first color card image and the second color card image. Specifically, the first spectral curve corresponding to the multi-spectrum sensor, the second spectral curve corresponding to the first image sensor, and the third spectral curve corresponding to the third image sensor can be obtained from the color information corresponding to the N high-dimensional light sources from the multi-spectrum image, the first color card image corresponding to the multi-spectrum sensor, the second color card image corresponding to the first image sensor, and the third color card image corresponding to the second image sensor can be determined based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve, and the third spectral curve, the first high-dimensional color mapping parameter can be fitted according to the first color card image and the second color card image, and the second high-dimensional color mapping parameter can be fitted according to the first color card image and the third color card image.

[0217] In a possible implementation, the plurality of image sensors capture a plurality of images and the multispectral sensor captures a multispectral image at the same time, specifically, a first image is captured by the first image sensor, a second image is captured by the second image sensor, and a multispectral image is captured by the multispectral sensor at the same time.

[0218] In a possible implementation, the foregoing electronic device further includes a third camera, or the electronic device further includes a third image sensor and a fourth image sensor, and the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor correspond to at least two focal lengths.

[0219] In a possible implementation, the foregoing first image sensor or second image sensor includes at least one of the following sensors: a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera, or an ultra-long-focus camera.

[0220] As shown in FIG. 13, it is a hardware structure schematic diagram of an imaging system or an electronic device 130 provided by an embodiment of the present application. The imaging system or the electronic device 130 can be used to implement the steps of the methods in the foregoing FIGS. 3 to 11.

[0221] The imaging system or the electronic device 130 shown in FIG. 13 can include a processor 1301, a memory 1302, a communication interface 1303, and a bus 1304. The processor 1301, the memory 1302, and the communication interface 1303 can be connected through the bus 1304.

[0222] The processor 1301 is the control center of the imaging system or the electronic device 130, and can be a general central processing unit (CPU), or other general-purpose processors, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., which specifically can include a GPU or an NPU, etc., and can be adaptively set according to actual application scenarios.

[0223] As an example, the processor 1301 can include one or more CPUs, and can also include other processors, such as the CPU, NPU, or GPU shown in FIG. 13, etc.

[0224] The memory 1302 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0225] In a possible implementation, the memory 1302 can exist independently of the processor 1301. The memory 1302 can be connected to the processor 1301 through the bus 1304, for storing data, instructions, or program codes. When the processor 1301 invokes and executes the instructions or program codes stored in the memory 1302, the method provided by the embodiments of the present application can be implemented, for example, the method shown in FIGS. 3 to 11.

[0226] In another possible implementation, the memory 1302 can also be integrated with the processor 1301.

[0227] The communication interface 1303 is configured to connect the imaging system or the electronic device 130 to other devices through a communication network, which can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), or the like. The communication interface 1303 can include a receiving unit configured to receive data, and a sending unit configured to send data.

[0228] The bus 1304 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in FIG. 13, but it does not mean that there is only one bus or only one type of bus.

[0229] It should be noted that the structure shown in FIG. 13 does not constitute a limitation on the imaging system or the electronic device 130, and the imaging system or the electronic device 130 can include more or fewer components than shown in FIG. 13, or combine certain components, or arrange different components.

[0230] Those skilled in the art can clearly understand the application by the description of the above embodiments. The application can be implemented by means of software and necessary general hardware, of course, it can also be implemented by special hardware including special integrated circuit, special CPU, special memory, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same function can be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the application can be embodied in the form of software product, which is stored in a readable storage medium, such as computer floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for making a device (which can be a personal computer, server or network equipment, etc.) execute the method described in various embodiments of the application.

[0231] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of computer program product in whole or in part.

[0232] The computer readable storage medium in the embodiments of the application stores a program for training a model or executing an inference task, which, when running on a computer, causes the computer to execute all or part of the steps of the method described in the foregoing embodiments of FIG. 3 to FIG. 8.

[0233] The embodiments of the present application also provide a digital processing chip. The digital processing chip integrates a circuit for implementing the processor or the function of the processor and one or more interfaces. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the preceding embodiments. When the digital processing chip does not integrate a memory, the digital processing chip can be connected with an external memory through a communication interface. The digital processing chip implements the method steps of any one or more of the preceding embodiments according to program codes stored in the external memory. For example, the chip can be an ISP, and refer to the digital signal processor 132 shown in FIG. 1.

[0234] The embodiments of the present application also provide a computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0235] The data comparison device provided by the embodiments of the present application can be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin, a circuit or the like. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the server executes the method described in the embodiments shown in FIGS. 3 to 11. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache or the like. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) and the like.

[0236] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like. The general-purpose processor can be a microprocessor or any conventional processor, and the like.

[0237] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0238] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. Generally, any function completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special-purpose circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0239] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0240] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0241] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or inverse order, depending upon the circumstances. The term "and / or" in the present application is merely used to represent an association between associated objects, and it is possible that three relationships exist, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules as an example is not necessarily limited to those steps or modules that are clearly listed, but can include other steps or modules that are not clearly listed or inherent to such process, method, product or device. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can be executed in a different order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some ports, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. Furthermore, the modules or sub-modules described as separate components can or can not be physically separated, and can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present application.

Claims

1. An imaging method characterized by, The method is applied to an electronic device including a first image sensor, a second image sensor and a multi-spectrum sensor, and includes: acquiring a first image by the first image sensor, a second image by the second image sensor, and a multi-spectrum image by the multi-spectrum sensor; acquiring a first high-dimensional color mapping parameter and a second high-dimensional color mapping parameter according to the multi-spectrum image, the first high-dimensional color mapping parameter representing a mapping relationship between colors of the multi-spectrum image and colors of the first image, and the second high-dimensional color mapping parameter representing a mapping relationship between colors of the multi-spectrum image and colors of the second image; color adjusting the first image according to the first high-dimensional color mapping parameter to obtain a third image; color adjusting the second image according to the second high-dimensional color mapping parameter to obtain a fourth image.

2. The method of claim 1, wherein, The color adjusting the first image according to the first high-dimensional color mapping parameter to obtain a third image includes: projecting data of the multi-spectrum image to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result; determining a first color adjustment parameter of the first image sensor according to the first multi-spectrum low-dimensional projection result, and color adjusting data of the first image according to the first color adjustment parameter to obtain the third image.

3. The method of claim 2, wherein, The projecting data of the multi-spectrum image to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result includes: acquiring a high-dimensional white point in the multi-spectrum image; mapping the high-dimensional white point to a space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain a first low-dimensional white point, the first multi-spectrum low-dimensional projection result including the first low-dimensional white point; The determining a first color adjustment parameter of the first image sensor according to the first multi-spectrum low-dimensional projection result, and color adjusting data of the first image according to the first color adjustment parameter to obtain the third image includes: acquiring a first white balance coefficient of the first image sensor according to the first low-dimensional white point, the color adjustment parameter including the first white balance coefficient; performing white balance processing on the first image according to the first white balance coefficient to obtain the third image.

4. The method of claim 2, wherein, The projecting data of the multi-spectrum image to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result includes: projecting the multi-spectrum image to a space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain a first low-dimensional color projection image, the first multi-spectrum low-dimensional projection result including the first low-dimensional color projection image; The determining a first color adjustment parameter of the first image sensor according to the first multi-spectrum low-dimensional projection result, and color adjusting data of the first image according to the first color adjustment parameter to obtain the third image includes: determining first low-dimensional color mapping parameters of the first image sensor according to the multi-spectrum low-dimensional color projection image, the first color adjustment parameter comprising the first low-dimensional color mapping parameters; color mapping the first image according to the first low-dimensional color mapping parameters to obtain the third image.

5. The method of claim 4, wherein, The method further comprises: obtaining third high-dimensional color mapping parameters; projecting color values of the multi-spectrum image into a standard space according to the third high-dimensional color mapping parameters to obtain a standard space projection image; The method further comprises: obtaining color mapping matrices through a color mapping network; 6. The method according to any one of claims 2-5, characterized in that, The method further comprises: updating the first color adjustment parameter according to the color mapping matrices to obtain updated first color adjustment parameters; color adjusting the first image according to the updated first color adjustment parameters to obtain the third image. The method further comprises: obtaining initial first high-dimensional color mapping parameters and initial second high-dimensional color mapping parameters according to the multi-spectrum image; 7. The method according to any one of claims 1 to 6, characterized in that, aligning the multi-spectrum image with the initial first high-dimensional color mapping parameters to obtain a first alignment result, and aligning the multi-spectrum image with the initial second high-dimensional color mapping parameters to obtain a second alignment result; updating the initial first high-dimensional color mapping parameters according to the first alignment result to obtain the first high-dimensional color mapping parameters, and updating the initial second high-dimensional color mapping parameters according to the second alignment result to obtain the second high-dimensional color mapping parameters. The method further comprises: obtaining color information from the multi-spectrum image, the color information comprising at least one of color temperature information, high-dimensional white points, light source reflection spectra, or light source spectra; 8. The method according to any one of claims 1-7, characterized in that, calculating the first high-dimensional color mapping parameters and the second high-dimensional color mapping parameters according to the color information. The method further comprises: obtaining color information corresponding to N high-dimensional light sources from the multi-spectrum image; 9. The method of claim 8, wherein, The method further comprises: obtaining a first spectral curve corresponding to the multi-spectrum sensor, a second spectral curve corresponding to the first image sensor, and a third spectral curve corresponding to the third image sensor; ​ ​ determine, based on the imaging model and color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve and the third spectral curve, a first color chart image corresponding to the multi-spectrum sensor, a second color chart image corresponding to the first image sensor and a third color chart image corresponding to the second image sensor; fit the first high-dimensional color mapping parameter according to the first color chart image and the second color chart image, and fit the second high-dimensional color mapping parameter according to the first color chart image and the third color chart image.

10. The method according to any one of claims 1-9, characterized in that, The first image is collected by the first image sensor, the second image is collected by the second image sensor, and the multi-spectrum image is collected by the multi-spectrum sensor, including: At the same time, the first image is collected by the first image sensor, the second image is collected by the second image sensor, and the multi-spectrum image is collected by the multi-spectrum sensor.

11. The method according to any one of claims 1-10, characterized in that, The electronic device further includes a third camera, or the electronic device further includes a third image sensor and a fourth image sensor, and the first image sensor, the second image sensor, the third image sensor and the fourth image sensor correspond to at least two focal lengths.

12. The method according to any one of claims 1-11, characterized in that, The first image sensor or the second image sensor includes at least one of the following sensors: a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera or an ultra-long-focus camera.

13. An imaging system characterized by, including: a first image sensor, a second image sensor, a multi-spectrum sensor and a processing unit; the first image sensor is used to collect a first image; the second image sensor is used to collect a second image; the multi-spectrum sensor is used to collect a multi-spectrum image; the processing unit is used to obtain a first high-dimensional color mapping parameter and a second high-dimensional color mapping parameter according to the multi-spectrum image, the first high-dimensional color mapping parameter representing a mapping relationship between colors of the multi-spectrum image and colors of the first image, and the second high-dimensional color mapping parameter representing a mapping relationship between colors of the multi-spectrum image and colors of the second image; the processing unit is further used to perform color adjustment on the first image according to the first high-dimensional color mapping parameter to obtain a third image, and perform color adjustment on the second image according to the second high-dimensional color mapping parameter to obtain a fourth image.

14. The imaging system of claim 13, wherein, The processing unit is specifically used to: project data in the multi-spectrum image to a space corresponding to the first image sensor according to the first high-dimensional color mapping parameter to obtain a first multi-spectrum low-dimensional projection result; determine a first color adjustment parameter of the first image sensor according to the low-dimensional projection result, and adjust the color of the first image according to the first color adjustment parameter to obtain the third image.

15. The imaging system of claim 14, wherein, The processing unit is specifically used to: obtain a high-dimensional white point in the multi-spectrum image; map the high-dimensional white point to a space corresponding to the first image according to the first high-dimensional color mapping parameter to obtain a first low-dimensional white point, and the multi-spectrum low-dimensional projection result includes the first low-dimensional white point; The first white balance coefficient of the first image sensor is obtained according to the first low-dimensional white point, and the first color adjustment parameter comprises the first white balance coefficient; The first image is subjected to white balance processing according to the first white balance coefficient, to obtain the third image.

16. The imaging system of claim 14, wherein, The processing unit is specifically configured to: The first low-dimensional color projection image is obtained by projecting the multi-spectrum image into a space corresponding to the first image according to the first high-dimensional color mapping parameter, and the first multi-spectrum low-dimensional projection result comprises the first low-dimensional color projection image; The low-dimensional color mapping parameter of the first image sensor is determined according to the multi-spectrum low-dimensional color projection image, and the color adjustment parameter comprises the first low-dimensional color mapping parameter; The first image is subjected to color mapping according to the first low-dimensional color mapping parameter, to obtain the third image.

17. The imaging system of claim 16, wherein, The processing unit is further configured to: Obtain a third high-dimensional color mapping parameter; Color values of the multi-spectrum image are projected into a standard space to obtain a standard space projection image according to the third high-dimensional color mapping parameter; The first low-dimensional color mapping parameter is determined according to the multi-spectrum low-dimensional image and the standard space projection image.

18. The imaging system of any of claims 14-17, wherein, The processing unit is further configured to: Obtain a first color mapping matrix through a color mapping network; The first color adjustment parameter is updated according to the first color mapping matrix, to obtain an updated first color adjustment parameter; The first image is subjected to color adjustment according to the updated first color adjustment parameter, to obtain the third image.

19. The imaging system of any of claims 13-18, wherein, The processing unit is specifically configured to: Obtain an initial first high-dimensional color mapping parameter and an initial second high-dimensional color mapping parameter according to the multi-spectrum image; Align the multi-spectrum image with the initial first high-dimensional color mapping parameter to obtain a first alignment result, and align the multi-spectrum image with the initial second high-dimensional color mapping parameter to obtain a second alignment result; The initial first high-dimensional color mapping parameter is updated according to the first alignment result to obtain the first high-dimensional color mapping parameter, and the initial second high-dimensional color mapping parameter is updated according to the second alignment result to obtain the second high-dimensional color mapping parameter.

20. The imaging system of any of claims 13-19, wherein, The processing unit is specifically configured to: Obtain color information from the multi-spectrum image, the color information comprising at least one of color temperature information, a high-dimensional white point, a light source reflection spectrum or a light source spectrum; The first high-dimensional color mapping parameter and the second high-dimensional color mapping parameter are calculated according to the color information.

21. The imaging system of claim 20, wherein, The processing unit is specifically configured to: Obtain color information corresponding to N high-dimensional light sources from the multi-spectrum image; Obtain a first spectral curve corresponding to the multi-spectrum sensor, a second spectral curve corresponding to the first image sensor and a third spectral curve corresponding to the third image sensor; determine, based on the imaging model and the color information corresponding to the N high-dimensional light sources, the first spectral curve, the second spectral curve and the third spectral curve, a first color chart image corresponding to the multi-spectral sensor, a second color chart image corresponding to the first image sensor and a third color chart image corresponding to the second image sensor; fit the first high-dimensional color mapping parameter according to the first color chart image and the second color chart image, and fit the second high-dimensional color mapping parameter according to the first color chart image and the third color chart image.

22. The imaging system of any one of claims 13-21, wherein, at the same time, the first image is captured by the first image sensor, the second image is captured by the second image sensor, and the multi-spectral image is captured by the multi-spectral sensor.

23. The imaging system of any of claims 13-22, wherein, The electronic device further comprises a third camera, or the electronic device further comprises a third image sensor and a fourth image sensor, and the first image sensor, the second image sensor, the third image sensor and the fourth image sensor correspond to at least two focal lengths.

24. The imaging system of any of claims 13-23, wherein, The first image sensor or the second image sensor comprises at least one of the following sensors: a main camera, a wide-angle camera, an ultra-wide-angle camera, a long-focus camera or an ultra-long-focus camera.

25. A chip, characterized by comprise a processor and a memory, the memory storing a program, when the program instruction stored in the memory is executed by the processor, the steps of the method of any one of claims 1-12 are implemented.

26. A terminal, characterized by comprise a plurality of image sensors for capturing a plurality of images, a multi-spectral sensor for capturing a multi-spectral image, a processor and a memory, the memory storing a program, when the program instruction stored in the memory is executed by the processor, the steps of the method of any one of claims 1-12 are implemented based on the plurality of images and the multi-spectral image.

27. A computer readable storage medium, characterized in that, comprise a program, when it is executed by a processing unit, the method of any one of claims 1-12 is executed.

28. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the method of any one of claims 1-12.

Images