Camera color calibration method and device and electronic equipment

Abstract

The invention provides a camera color calibration method applied to electronic equipment, the electronic equipment comprises a camera array (comprising a first camera) and a high-resolution multispectral camera, and the method comprises the following steps: synchronously acquiring an original image of a first scene through the first camera and the high-resolution multispectral camera, the first camera acquires a three-channel image (a first image), the high-resolution multispectral camera synchronously acquires an M-channel image (a second image), and M is a positive integer greater than or equal to 3; converting the second image from the M-channel image into a three-channel image through a first conversion function; and performing color calibration on the first image by taking the converted three-channel image as a reference to obtain a first color parameter for performing color adjustment on the first camera. In the method, the colors of the array cameras are all aligned to the colors sensed by the high-resolution multispectral camera, so that the authenticity and consistency of the colors of the images shot by the cameras in the camera array can be improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of camera color calibration, and more specifically, to a camera color calibration method, device, and electronic device. Background Art

[0002] Taking mobile phones as an example, the cameras of smart phones are generally presented in the form of camera arrays, which include multiple cameras. The main difference between these cameras is the focal length, which is mainly divided into ultra-wide-angle cameras, wide-angle cameras (main cameras) and telephoto cameras. Different mobile phones may be equipped with different cameras, but there are at least two cameras. These cameras can independently image and have the function of taking pictures and recording. The images or videos taken will be directly presented to users for browsing. Since the images taken by different cameras in the same scene are likely to have obvious color differences, it will seriously affect the user's shooting experience. Summary of the invention

[0003] The present application provides a method, device and electronic device for camera color calibration. Through this method, device and electronic device, the color of the image taken by the camera in the camera array is adjusted by taking the image taken by the high-resolution multispectral camera as the color reference, so that the color of the array camera is aligned with the color perceived by the high-resolution multispectral camera, which can improve the authenticity of the color of the image taken by the camera in the camera array.

[0004] In a first aspect, a method for camera color calibration is provided, the method being applied to an electronic device, the electronic device comprising a camera array and a high-resolution multispectral camera, the method comprising: when a first camera and the high-resolution multispectral camera are turned on at the same time, a first image is acquired through the first camera, the first image is an original image of a first scene, the first image is a three-channel image, and the camera array comprises the first camera; a second image is synchronously acquired through the high-resolution multispectral camera, the second image is a multispectral original image of the first scene, the second image is an M-channel image, wherein M is a positive integer greater than 3; through a first conversion function, the second image is converted from an M-channel image to a three-channel image; based on the three-channel image converted from the second image, color calibrate the first image to obtain a first color parameter; and send the first color parameter to an image signal processor ISP path of the first camera, so that the first camera performs color adjustment on the image acquired by the first camera according to the first color parameter.

[0005] In some embodiments, the first color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0006] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference to adjust the color of the image captured by the cameras in the camera array, so that the color of the array camera is aligned with the color perceived by the high-resolution multispectral camera, thereby improving the authenticity of the color of the image captured by the cameras in the camera array.

[0007] In combination with the first aspect, in a possible implementation, the camera array also includes a second camera, and the method also includes: acquiring a third image through the second camera, the third image being an original image of the first scene, and the third image being a three-channel image; performing color calibration on the third image based on the three-channel image converted from the second image to obtain second color parameters; and sending the second color parameters to an image signal processor ISP channel of the second camera, so that the second camera performs color adjustment on the image acquired by the second camera according to the second color parameters.

[0008] In some embodiments, the first camera, the second camera, and the high-resolution multi-spectral camera synchronously capture images of the first scene.

[0009] In some further embodiments, the first camera, the second camera, and the high-resolution multi-spectral camera asynchronously capture images of the first scene within a certain time threshold.

[0010] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0011] In combination with the first aspect, in a possible implementation, the camera array also includes a second camera, and the method also includes: when the second camera and the high-resolution multi-spectral camera are turned on at the same time, a third image is collected by the second camera, and the third image is an original image of the second scene, and the third image is a three-channel image; a fourth image is synchronously collected by the high-resolution multi-spectral camera, and the fourth image is a multi-spectral original image of the second scene, and the fourth image is an M-channel image; through a second conversion function, the fourth image is converted from an M-channel image to a three-channel image; based on the three-channel image converted from the fourth image, color calibrate the third image to obtain second color parameters; and send the second color parameters to the image signal processor ISP channel of the second camera, so that the second camera adjusts the color of the image collected by the second camera according to the second color parameters.

[0012] In some embodiments, the second color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0013] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0014] In combination with the first aspect, in a possible implementation, the camera array also includes a third camera, and the method also includes: when the third camera and the high-resolution multi-spectral camera are turned on at the same time, a fifth image is collected by the third camera, the fifth image is an original image of the first scene, and the fifth image is a three-channel image; a sixth image is synchronously collected by the high-resolution multi-spectral camera, the sixth image is a multi-spectral original image of the first scene, and the sixth image is an M-channel image, wherein M is a positive integer greater than 3; through a third conversion function, the sixth image is converted from an M-channel image to a three-channel image; based on the three-channel image converted from the sixth image, the fifth image is color calibrated to obtain third color parameters; the third color parameters are sent to the image signal processor ISP channel of the third camera, so that the third camera adjusts the color of the image collected by the third camera according to the third color parameters.

[0015] In some embodiments, the third color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0016] In some embodiments, the first camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, the second camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, and the third camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, wherein the first camera, the second camera, and the third camera are different from each other.

[0017] In some embodiments, the first camera is a main camera (wide-angle camera), the second camera is an ultra-wide-angle camera, and the third camera is a telephoto camera.

[0018] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0019] In combination with the first aspect, in a possible implementation method, the second image is converted from an M-channel image to a three-channel image through a first conversion function, including: downsampling from the M channels of the second image to the three channels of the first image according to the first conversion function, converting the second image from an M-channel image to a three-channel image, wherein the first conversion function is a function that uses a downsampling matrix of size M×3 to perform matrix multiplication with the second image.

[0020] In some embodiments, the further explanation of “converting the fourth image from an M-channel image to a three-channel image through a second conversion function” is similar to the above explanation of “converting the second image from an M-channel image to a three-channel image through a first conversion function”.

[0021] In some embodiments, the further explanation of “converting the sixth image from an M-channel image to a three-channel image through a third conversion function” is similar to the above explanation of “converting the second image from an M-channel image to a three-channel image through the first conversion function”.

[0022] In an embodiment of the present application, the M channels of the second image are downsampled to 3 channels through a downsampling matrix, and a mapping relationship is established between the second image and the color space where the image that can be obtained by the first camera is located. Since the second image contains a number of color channels of size M, the spectral information included is richer than the spectral information corresponding to the first image, it is mapped to the color space where the image that can be obtained by the first camera is located, and the obtained three-channel image is closer to the true color of the first scene. Then, the image after the downsampling of the second image channel can be used to adjust the color of the first image, thereby improving the color authenticity and consistency of the image taken by the first camera.

[0023] In combination with the first aspect, in a possible implementation, before converting the second image from an M-channel image to a three-channel image through a first conversion function, the method further includes: aligning the perspective FOV of the first image and the second image.

[0024] In some embodiments, before converting the fourth image from an M-channel image to a three-channel image through a second conversion function, the method further includes: aligning the third image and the fourth image by a viewing angle FOV.

[0025] In some embodiments, before converting the sixth image from an M-channel image to a three-channel image through a third conversion function, the method further includes: aligning the fifth image and the sixth image by performing a viewing angle FOV.

[0026] In an embodiment of the present application, before converting the second image from an M-channel image to a three-channel image (i.e., establishing a mapping relationship between the second image and the color space in which the first camera can obtain the image), the first image and the second image are first aligned in perspective, thereby avoiding the authenticity and consistency of the color adjustment of the first image being affected by the perspective difference between the first image and the second image.

[0027] In combination with the first aspect, in a possible implementation manner, the method further includes: determining the first conversion function based on the second image after FOV alignment.

[0028] In some embodiments, the method further includes: determining the second conversion function based on the FOV-aligned fourth image.

[0029] In some embodiments, the method further includes: determining the third conversion function based on the FOV-aligned sixth image.

[0030] In some embodiments, the estimation method of the downsampling matrix used by the first conversion function may be any one of the Bayesian method, the least squares method, and the deep learning method, or may be other estimation methods.

[0031] In combination with the first aspect, in a possible implementation method, the second image is converted from an M-channel image to a three-channel image through a first conversion function, including: performing spectral band super-resolution on the second image according to a spectral sensing curve corresponding to the second image to obtain a hyper-spectral image corresponding to the second image, wherein the hyper-spectral image corresponding to the second image is an N-channel image, wherein N is a positive integer much greater than M; down-sampling from the N channels of the hyper-spectral image corresponding to the second image to the three channels of the first image according to the first conversion function, thereby converting the hyper-spectral image corresponding to the second image from an N-channel image to a three-channel image, wherein the first conversion function is a function that uses a down-sampling matrix of size N×3 to perform matrix multiplication with the second image.

[0032] In some embodiments, the further explanation of “converting the fourth image from an M-channel image to a three-channel image through a second conversion function” is similar to the above explanation of “converting the second image from an M-channel image to a three-channel image through a first conversion function”.

[0033] In some embodiments, the further explanation of “converting the sixth image from an M-channel image to a three-channel image through a third conversion function” is similar to the above explanation of “converting the second image from an M-channel image to a three-channel image through the first conversion function”.

[0034] In an embodiment of the present application, the N channels of the hyperspectral image of the second image are downsampled to 3 channels through a downsampling matrix, and a mapping relationship between the hyperspectral image of the second image and the color space where the image that can be obtained by the first camera is located is established. Since the hyperspectral image of the second image contains a number of color channels of size N, the spectral information included is richer than the spectral information corresponding to the first image, it is mapped to the color space where the image that can be obtained by the first camera is located, and the obtained three-channel image is closer to the true color of the first scene, and then the image after the downsampling of the hyperspectral image channel of the second image can be used to adjust the color of the first image, thereby improving the color authenticity and consistency of the image taken by the first camera.

[0035] In combination with the first aspect, in a possible implementation manner, the method further includes: determining the first conversion function according to a hyperspectral image corresponding to the second image and a spectral sensing curve corresponding to the first image.

[0036] In some embodiments, the method further includes: determining a second conversion function according to a hyperspectral image corresponding to the fourth image and a spectral sensing curve corresponding to the third image.

[0037] In some embodiments, the method further includes: determining a third conversion function according to a hyperspectral image corresponding to the sixth image and a spectral sensing curve corresponding to the fifth image.

[0038] In some embodiments, the estimation method of the downsampling matrix used by the first conversion function / the second conversion function / the third conversion function may be any one of the Bayesian method, the least squares method, the deep learning method, or other estimation methods.

[0039] In a second aspect, an electronic device is provided, which includes a camera array, a high-resolution multispectral camera and a processor, wherein the camera array includes a first camera, wherein the first camera is used to capture a first image when the first camera and the high-resolution multispectral camera are turned on at the same time, the first image is an original image of a first scene, and the first image is a three-channel image; the high-resolution multispectral camera is used to synchronously capture a second image, the second image is a multispectral original image of the first scene, and the second image is an M-channel image, wherein M is a positive integer greater than 3; the processor is used to convert the second image from an M-channel image to a three-channel image through a first conversion function; the processor is also used to perform color calibration on the first image based on the three-channel image converted from the second image to obtain a first color parameter; the processor is also used to send the first color parameter to the image signal processor ISP channel of the first camera, so that the first camera can adjust the color of the image captured by the first camera according to the first color parameter.

[0040] In some embodiments, the first color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0041] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference to adjust the color of the image captured by the cameras in the camera array, so that the color of the array camera is aligned with the color perceived by the high-resolution multispectral camera, thereby improving the authenticity of the color of the image captured by the cameras in the camera array.

[0042] In combination with the second aspect, in a possible implementation, the camera array also includes a second camera, wherein: the second camera is used to capture a third image, the third image is an original image of the first scene, and the third image is a three-channel image; the processor is also used to perform color calibration on the third image based on the three-channel image converted from the second image to obtain second color parameters; the processor is also used to send the second color parameters to the image signal processor ISP channel of the second camera, so that the second camera adjusts the color of the image captured by the second camera according to the second color parameters.

[0043] In some embodiments, the first camera, the second camera, and the high-resolution multi-spectral camera synchronously capture images of the first scene.

[0044] In some further embodiments, the first camera, the second camera, and the high-resolution multi-spectral camera asynchronously capture images of the first scene within a certain time threshold.

[0045] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0046] In combination with the second aspect, in a possible implementation, the camera array also includes a second camera, wherein: the second camera is used to capture a third image through the second camera when the second camera and the high-resolution multi-spectral camera are turned on at the same time, and the third image is an original image of the second scene, and the third image is a three-channel image; the high-resolution multi-spectral camera is also used to synchronously capture a fourth image, and the fourth image is a multi-spectral original image of the second scene, and the fourth image is an M-channel image; the processor is also used to convert the fourth image from an M-channel image to a three-channel image through a second conversion function; the processor is also used to perform color calibration on the third image based on the three-channel image converted from the fourth image to obtain second color parameters; the processor is also used to send the second color parameters to the image signal processor ISP channel of the second camera, so that the second camera can adjust the color of the image captured by the second camera according to the second color parameters.

[0047] In some embodiments, the second color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0048] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0049] In combination with the second aspect, in a possible implementation, the camera array also includes a third camera, wherein: the third camera is used to capture a fifth image through the third camera when the third camera and the high-resolution multi-spectral camera are turned on at the same time, the fifth image is the original image of the first scene, and the fifth image is a three-channel image; the high-resolution multi-spectral camera is also used to synchronously capture a sixth image, the sixth image is the multi-spectral original image of the first scene, and the sixth image is an M-channel image, wherein M is a positive integer greater than 3; the processor is also used to convert the sixth image from an M-channel image to a three-channel image through a third conversion function; the processor is also used to perform color calibration on the fifth image based on the three-channel image converted from the sixth image to obtain a third color parameter; the processor is also used to send the third color parameter to the image signal processor ISP channel of the third camera, so that the third camera can adjust the color of the image captured by the third camera according to the third color parameter.

[0050] In some embodiments, the third color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

[0051] In some embodiments, the first camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, the second camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, and the third camera is one of the main camera, ultra-wide-angle camera, and telephoto camera, wherein the first camera, the second camera, and the third camera are different from each other.

[0052] In some embodiments, the first camera is a main camera (wide-angle camera), the second camera is an ultra-wide-angle camera, and the third camera is a telephoto camera.

[0053] In an embodiment of the present application, the image captured by a high-resolution multispectral camera is used as a color reference, and the colors of the images captured by each camera in the camera array are adjusted separately. The colors of the array cameras are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by the cameras in the camera array.

[0054] In combination with the second aspect, in a possible implementation method, the processor is specifically used to: convert the second image from an M-channel image to a three-channel image by downsampling from the M channels of the second image to the three channels of the first image according to the first conversion function, wherein the first conversion function is a function that uses a downsampling matrix of size M×3 to perform matrix multiplication with the second image.

[0055] In an embodiment of the present application, the M channels of the second image are downsampled to 3 channels through a downsampling matrix, and a mapping relationship is established between the second image and the color space where the image that can be obtained by the first camera is located. Since the second image contains a number of color channels of size M, the spectral information included is richer than the spectral information corresponding to the first image, it is mapped to the color space where the image that can be obtained by the first camera is located, and the obtained three-channel image is closer to the true color of the first scene. Then, the image after the downsampling of the second image channel can be used to adjust the color of the first image, thereby improving the color authenticity and consistency of the image taken by the first camera.

[0056] In combination with the second aspect, in a possible implementation, the processor is further specifically used to: before converting the second image from an M-channel image to a three-channel image through a first conversion function, align the perspective FOV of the first image and the second image.

[0057] In an embodiment of the present application, before converting the second image from an M-channel image to a three-channel image (i.e., establishing a mapping relationship between the second image and the color space in which the first camera can obtain the image), the first image and the second image are first aligned in perspective, thereby avoiding the authenticity and consistency of the color adjustment of the first image being affected by the perspective difference between the first image and the second image.

[0058] In combination with the second aspect, in a possible implementation manner, the processor is further used to: determine the first conversion function based on the second image after FOV alignment.

[0059] In some embodiments, the estimation method of the downsampling matrix used by the first conversion function may be any one of the Bayesian method, the least squares method, and the deep learning method, or may be other estimation methods.

[0060] In combination with the second aspect, in a possible implementation method, the processor is specifically used to: perform spectral band super-resolution on the second image according to the spectral sensing curve corresponding to the second image to obtain a hyper-spectral image corresponding to the second image, wherein the hyper-spectral image corresponding to the second image is an N-channel image, wherein N is a positive integer much greater than M; down-sample from the N channels of the hyper-spectral image corresponding to the second image to the three channels of the first image according to the first conversion function, thereby converting the hyper-spectral image corresponding to the second image from an N-channel image to a three-channel image, wherein the first conversion function is a function that uses a down-sampling matrix of size N×3 to perform matrix multiplication with the second image.

[0061] In an embodiment of the present application, the N channels of the hyperspectral image of the second image are downsampled to 3 channels through a downsampling matrix, and a mapping relationship between the hyperspectral image of the second image and the color space where the image that can be obtained by the first camera is located is established. Since the hyperspectral image of the second image contains a number of color channels of size N, the spectral information included is richer than the spectral information corresponding to the first image, it is mapped to the color space where the image that can be obtained by the first camera is located, and the obtained three-channel image is closer to the true color of the first scene, and then the image after the downsampling of the hyperspectral image channel of the second image can be used to adjust the color of the first image, thereby improving the color authenticity and consistency of the image taken by the first camera.

[0062] In combination with the second aspect, in a possible implementation manner, the processor is further used to: determine the first conversion function according to the hyperspectral image corresponding to the second image and the spectral sensing curve corresponding to the first image.

[0063] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0064] In a fourth aspect, a computer-readable storage medium is provided, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method in the above-mentioned first aspect or any possible implementation manner of the first aspect is implemented.

[0065] In a fifth aspect, a chip is provided, in which instructions are stored. When the instructions are executed on a device, the chip executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0066] In a sixth aspect, a computer program product is provided, in which a computer program or instructions are stored. When the computer program or instructions are executed, the method in the above-mentioned first aspect or any possible implementation manner of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0068] Figure 2 It is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0069] Figure 3 is a schematic diagram of an array camera provided in an embodiment of the present application;

[0070] Figure 4 is a schematic flow chart of a method for camera color calibration;

[0071] Figure 5 is a schematic flow chart of another method for camera color calibration;

[0072] Figure 6 This is another schematic diagram of an array camera provided in an embodiment of the present application;

[0073] Figure 7 It is a structural schematic diagram of a single-point multi-spectral camera provided in an embodiment of the present application;

[0074] Figure 8 It is a flowchart of a method for performing camera color calibration using a single-point multi-spectral camera provided in an embodiment of the present application;

[0075] Fig. 9 This is a schematic diagram of an RGGB sensor mode provided in an embodiment of the present application;

[0076] Fig.10 This is a schematic diagram of spectral perception of a high-resolution multi-spectral device provided in an embodiment of the present application;

[0077] Fig.11 is a schematic flow chart of a camera color calibration method provided in an embodiment of the present application;

[0078] Fig.12 is a schematic flow chart of another camera color calibration method provided in an embodiment of the present application;

[0079] Fig.13 It is a schematic diagram of a system framework corresponding to a camera color calibration method provided in an embodiment of the present application;

[0080] Fig.14 This is a flowchart corresponding to another camera color calibration method provided in an embodiment of the present application;

[0081] Fig.15 This is a flow chart corresponding to another camera color calibration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The technical solution in the present application will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0083] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" or "multiple" refers to two or more than two.

[0084] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0085] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0086] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "one embodiment", "some embodiments", "another embodiment", "some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0087] The method provided in the embodiments of the present application can be applied to electronic devices with a time display function or a time recognition function, for example, it can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), smart home devices and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0088] For example, Figure 11 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0089] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0090] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0091] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0092] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0093] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0094] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0095] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0096] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0097] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0098] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0099] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0100] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0101] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0102] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0103] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0104] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0105] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0106] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0107] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0108] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0109] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0110] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0111] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0112] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0113] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one App required for a function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0114] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0115] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0116] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0117] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0118] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0119] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0120] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0121] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0122] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0123] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an embedded SIM (eSIM) card, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0124] It should be understood that the phone card in the embodiments of the present application includes but is not limited to a SIM card, an eSIM card, a universal subscriber identity module (USIM), a universal integrated circuit card (UICC), and the like.

[0125] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0126] Figure 21 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0127] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0128] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0129] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0130] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0131] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0132] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0133] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0134] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0135] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0136] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0137] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0138] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0139] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0140] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0141] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0142] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0143] A 2D graphics engine is a drawing engine for 2D drawings.

[0144] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0145] It should be understood that the technical solutions in the embodiments of the present application can be used in Android, IOS, Hongmeng and other systems.

[0146] The technical solution of the embodiment of the present application can be applied to image processing scenarios of multiple cameras. For example, it can be applied to scenarios such as color calibration of images collected by array cameras.

[0147] Among them, the electronic device can be a television, a desktop computer, a laptop computer, or a portable electronic device such as a mobile phone, a folding screen, a tablet computer, a camera, a video camera, a video recorder, or a smart home device, such as a refrigerator, a washing machine, a sweeper, or any other electronic device with multiple cameras, or an electronic device in a 5G network or an electronic device in a future evolved public land mobile communication network (PLMN), etc.

[0148] For example, Figure 3 A schematic diagram of a camera setting provided in an embodiment of the present application is shown.

[0149] like Figure 3 As shown, the electronic device 300 is provided with a camera array, and the camera array includes a camera 310 , a camera 320 and a camera 330 .

[0150] In some embodiments, camera 310 is an ultra-wide-angle camera, camera 320 is a wide-angle camera which is also a main camera, and camera 330 is a telephoto camera.

[0151] Taking a mobile phone as an example, the camera of a smart phone is generally presented in the form of a camera array, which includes multiple cameras. The main difference between these cameras is the focal length, which is mainly divided into ultra-wide-angle camera, wide-angle camera (main camera) and telephoto camera. Different mobile phones may be equipped with different cameras, but there are at least two cameras. These cameras can independently image and have the function of taking pictures and recording. The images or videos taken will be directly presented to the user for browsing. Since the images taken by different cameras in the same scene are likely to have obvious color differences, this will seriously affect the user's shooting experience. Therefore, users have high requirements for the color consistency of the images taken by each camera in the camera array.

[0152] For example, Figure 4 A schematic flow chart of a method 400 for camera color calibration is shown. The method is a method for simultaneously starting multiple cameras, such as Figure 4 As shown, the method 400 includes:

[0153] S401: Using one camera (usually a main camera) among multiple cameras arranged on the first device as a reference, calibrate other cameras among the multiple cameras in real time.

[0154] Among them, the above-mentioned multiple cameras are all three-channel cameras, that is, the images captured by the above-mentioned multiple cameras are all three-channel images.

[0155] Specifically, one of the multiple cameras set on the first device may be used as a reference to calibrate each of the other cameras in the multiple cameras in real time.

[0156] Among them, the color of the image collected by the camera is calibrated in real time.

[0157] In this method, due to the differences in multiple dimensions such as shooting angle of view (FOV), shooting brightness, shooting contrast, shooting dynamic range, and sensor spectral response consistency among different camera modules, one of the multiple cameras is used as a benchmark to calibrate the other cameras. Since these differences between the cameras are not well taken into account, the accuracy of color calibration of the images taken by the cameras is relatively low. In addition, the calibration process requires at least two cameras to be turned on at the same time, which will result in greater power consumption.

[0158] For example, Figure 5 FIG. 5 is a schematic flow chart of another method 500 for camera color calibration. The method is an offline calibration method, such as Figure 5 As shown, the method 500 includes:

[0159] S501: Turn on a first camera and a second camera of an electronic device simultaneously, wherein the first camera serves as a reference camera for calibrating the second camera.

[0160] The first camera and the second camera are both three-channel cameras, that is, the images captured by the first camera and the second camera are both three-channel images.

[0161] S502: Determine a migration matrix / migration function between the first camera and the second camera according to the color perception distribution of the first camera and the color perception distribution of the second camera.

[0162] It can be understood that S501 and S502 are the process of calibrating the second camera based on the first camera. After the calibration is completed (that is, after the migration matrix / migration function between the first camera and the second camera is obtained), the first camera and the second camera are turned off.

[0163] S503: When photographing the first object through the first camera, turn on the first camera and collect a first image corresponding to the first object through the first camera.

[0164] S504: Determine a second image corresponding to the second camera according to a migration matrix / migration function between the first camera and the second camera and the first image.

[0165] Specifically, when photographing a first object through a first camera, the first camera is turned on, and the first camera estimates the spectrum of the current scene and determines the corresponding module migration parameters based on the migration matrix / migration function between the first camera and the second camera; then, the second image corresponding to the second camera is determined based on the module migration parameters and the first image.

[0166] In this method, since it is very difficult to estimate the scene spectrum and it is impossible to perceive the scene color in detail, it is difficult to determine the appropriate module migration parameters, resulting in a relatively low accuracy of camera color calibration.

[0167] In summary, the current camera color calibration methods all have the problems of low accuracy and high calibration cost (labor cost, power consumption cost), and the images taken by each camera in the camera array have inconsistent colors and unreal colors.

[0168] In view of this, the embodiments of the present application provide a method, device and electronic device for camera color calibration. In the method, a high-resolution multispectral camera is set on the electronic device, and the images collected by the high-resolution multispectral camera are used as a reference to perform color calibration on the cameras in the camera array set on the electronic device. This can avoid the problem of inaccurate calibration caused by calibration between array cameras, and can improve the accuracy and stability of camera color calibration, so that the colors of images output by multiple cameras in the camera array are more consistent and more realistic. In addition, the high-resolution multispectral camera is small in size and can also reduce the architectural redundancy of the electronic device. The power consumption generated by the calibration process is also lower, which can save calibration costs and thus improve the user's shooting experience.

[0169] For example, Figure 6 A schematic diagram of another camera setting provided in an embodiment of the present application is shown.

[0170] like Figure 6 As shown, the electronic device 600 is provided with a camera array and a high-resolution multi-spectral camera 610 , and the camera array includes a camera 310 , a camera 320 , and a camera 330 .

[0171] In some embodiments, camera 310 is an ultra-wide-angle camera, camera 320 is a wide-angle camera which is also a main camera, and camera 330 is a telephoto camera.

[0172] Among them, the spectral perception ability of the high-resolution multispectral camera is much stronger than that of the ultra-wide-angle camera, the wide-angle camera and the telephoto camera. In some embodiments, the scheme of the embodiment of the present application can be understood as: taking the multispectral image of the first scene captured by the high-resolution multispectral camera 610 as a reference, the red, green, blue (RGB) images of the first scene captured by the ultra-wide-angle camera 310, the wide-angle camera 320, and the telephoto camera 330 are color calibrated respectively.

[0173] It should be understood that this embodiment is only a schematic illustration of the camera setting method. The high-resolution multi-spectral camera 610 can be set independently of the camera array or in the camera array. This application does not limit the setting method of the high-resolution multi-spectral camera 610 on the electronic device.

[0174] In order to more clearly understand the advantages of high-resolution multispectral cameras for camera color calibration, for example, Figure 7 and Figure 8 , the process of camera color calibration using a single-point multispectral camera is described in detail.

[0175] Figure 7 The structure diagram of a single-point multi-spectral camera is shown in FIG. Figure 7 As shown, the single-point multi-spectral camera includes a light-homogenizing film and a single-point multi-spectral sensor arranged in parallel. Light incident from different directions passes through the light-homogenizing film. Due to the averaging effect of the light-homogenizing film, the amplitude and the angle of incidence of the light emitted from the light-homogenizing film are the same. The emitted light then passes through the single-point multi-spectral sensor to sense the average reflection spectrum of the scene. After multiple light rays emitted from the light-homogenizing film pass through the single-point multi-spectral sensor, a single-point spectral perception result with a dimension of 1×M is output, that is, a vector with a dimension of 1×M. Since the 1×M single-point spectral perception result has only one point (i.e., a single point) that is globally averaged in space and has spectral resolution, it is impossible to make a spatial correspondence, and it is impossible to clearly identify the specific spatial positions of each array camera corresponding to the single point, and it is impossible to estimate the spectral band downsampling mapping function (that is, it is impossible to convert a multi-channel image to a three-channel image).

[0176] Further, Figure 8 FIG. 8 is a flow chart showing a method 800 for performing camera color calibration using a single-point multi-spectral camera. Figure 8 As shown, the method 800 includes:

[0177] S801: The main camera / wide-angle camera / telephoto camera collects a RAW image of the current scene.

[0178] S802: The single-point camera synchronously obtains a single-point spectral perception result of the current scene, where the single-point spectral perception result is a single-point spectral perception result with a dimension of 1×M.

[0179] S803: Classify the light source based on the single-point spectral perception result, that is, determine the light source category of the current scene.

[0180] Among them, the light source classification based on the single-point spectral perception results has the problems of low accuracy and limited scenes in the determined light source category.

[0181] S804: Based on the light source category of the current scene, determine the color parameter according to a pre-calibrated color parameter-light source category association table.

[0182] The pre-calibrated color parameter-light source category association table refers to a pre-calibrated color parameter-light source category association table that maps the wide-angle / telephoto to the main camera.

[0183] S805: Send the determined color parameters to the ISP channel of the main camera / wide-angle / telephoto camera.

[0184] S806: Adjust the color of the wide-angle / telephoto camera according to the color parameter.

[0185] In this method, the light source classification based on the single-point spectral perception results has the problems of low accuracy and scene limitation in the determined light source category. Therefore, the camera color calibration using a single-point multi-spectral camera is discrete, inaccurate, and non-robust, and is easily affected by mixed light sources. Therefore, it is impossible to accurately perform color calibration and achieve multi-camera color consistency.

[0186] Therefore, in response to the technical problems existing in the above-mentioned single-point multispectral camera, the embodiment of the present application uses a high-resolution multispectral camera (a multi-point multispectral camera) to perform camera color calibration. In order to more clearly understand the camera color calibration method provided in the embodiment of the present application, the high-resolution multispectral camera (or high-resolution multispectral device) is first introduced in detail below.

[0187] At present, the structure of image sensors used in electronic devices is basically a Bayer Pattern structure. A basic color pixel includes 2×2=4 pixels, on which three different color filters (such as red, green and blue RGB, red, yellow and blue RYB) are placed to form an RGGB sensor or a RYYB sensor, or a color perception unit similar to RGGB or RYYB. The entire image sensor is composed of a large number of rows and columns of color perception units.

[0188] Among them, RGGB is a color mode, which is a mosaic color filter array formed by arranging RGB filters on the grid of light sensing components. In this arrangement, 50% is green, 25% is red, and the remaining 25% is blue; the RYYB sensor changes the arrangement of the bottom filter of the RGGB array, replacing two green pixels (G) with two yellow pixels (Y) to form a RYYB sensor.

[0189] The main difference between a high-resolution multispectral device and an RGGB device and an RYYB device is that its color perception unit has more spectral perception functions (i.e., more spectral bands). Specifically, the number of spectral perception functions (i.e., the number of spectral bands) of an RGGB device and an RYYB device is 3 (red, green, and blue or red, yellow, and blue), and the number of spectral perception functions of a high-resolution multispectral device is greater than 3, for example, it can be 3×3=9 spectral bands. Therefore, a high-resolution multispectral device can obtain a spectral perception capability far stronger than that of three channels without significantly reducing the spatial resolution. That is to say, compared with the three-channel spectral perception capability of an RGGB device and an RYYB device, the spectral perception capability of a high-resolution multispectral device is stronger, and the color authenticity of the perceived multispectral image is higher.

[0190] For example, taking the RGGB device as an example, Fig. 9 A schematic diagram of spectral perception of a three-channel spectral device provided in an embodiment of the present application is shown.

[0191] like Fig. 9 As shown, in the RGGB device, RGB filters are arranged on the light sensing component grid to form a mosaic color filter array. In this arrangement, 50% is green, 25% is red, and the remaining 25% is blue. Its corresponding spectral perception capability is a three-channel (red, green, and blue) spectral perception capability.

[0192] For example, Fig.10 A schematic diagram of spectral perception of a high-resolution multi-spectral device provided in an embodiment of the present application is shown.

[0193] like Fig.10 As shown in (a) of FIG. 1 , in one embodiment, the number of spectral bands sensed by the high-resolution multi-spectral device is 5, for example, including red, orange, yellow, blue, and purple, and the corresponding multi-spectral response curve can be, for example, as shown in FIG. Fig.10 As shown in (c) in Fig.10 (a) shows that the multi-spectral bands perceived by the high-resolution multi-spectral device are converted into Fig.10 The hyperspectral band shown in (b) in FIG. 1 , the corresponding hyperspectral response curve can be, for example, as shown in FIG. Fig.10 As shown in (d) in .

[0194] It can be seen that compared with the 3-channel spectrum sensed by the three-channel spectral device, the high-resolution multi-spectral device has a stronger spectral perception ability, and the color of the multi-spectral image collected by the high-resolution multi-spectral device is closer to the real color.

[0195] Specifically:

[0196] (1) Due to the limitations of material capabilities, the three-channel camera’s corresponding spectral perception curve is quite different from that of the human eye. The color restoration capability of the captured image is weak, and some colors are prone to color cast. For example, when capturing images of real red-orange, cyan, and blue-purple, the corresponding colors in the output image deviate from red-orange, cyan, and blue-purple.

[0197] The multispectral perception curve and hyperspectral perception curve corresponding to the high-resolution multispectral camera are slightly different from the spectral perception curve corresponding to the human eye, and the collected images have strong color restoration capabilities, so there is basically no color cast problem.

[0198] In an embodiment of the present application, a high-resolution multispectral camera can reduce the corresponding spectral perception function from a high dimension, that is, based on the perceived multispectral bands, it can perform spectral band downsampling to obtain a three-channel image with more realistic colors.

[0199] The spectral perception curve can also be described as a spectral response curve.

[0200] (2) The human eye and brain have the ability to adapt to colors, which is mainly related to the color of the ambient light source. It is difficult to accurately estimate the color of the light source from the three-channel color information collected by a three-channel (for example, RGB / RYB) camera. The output image is prone to white balance inaccuracy, resulting in color cast of the entire image. High-resolution multispectral cameras have more channels, which greatly increases the spatial color information corresponding to the RAW image collected by the high-resolution multispectral camera. The color of the output image is closer to the true color of the collected object. Correspondingly, the white balance capability is also significantly enhanced, which can solve the problem of image color cast to a large extent. White balance refers to the reference color used to determine the color of the image. When white balance is inaccurate (that is, the reference color is inaccurate), it will naturally lead to color cast of the entire image.

[0201] (3) The spectral perception function of the high-resolution multispectral camera can be reduced to the spectral perception function corresponding to the human eye, the spectral perception function corresponding to the RGB camera, and the spectral perception function corresponding to the RYB camera (it can also be understood as: the spectral perception function corresponding to the high-resolution multispectral camera is fitted with the spectral perception function corresponding to the human eye, the spectral perception function corresponding to the RGB camera, and the spectral perception function corresponding to the RYB camera respectively). In other words, the multi-channel image collected by the high-resolution multispectral camera can be converted into a three-channel image. Since the multi-channel image collected by the high-resolution multispectral camera has a high authenticity, the converted three-channel image is Compared with the three-channel image captured by the three-channel camera, the converted three-channel image is more authentic. The dimensionality reduction fitting of the spectral band from high dimension to low dimension is very accurate. Since the high-resolution multispectral camera can perform the above-mentioned spectral band dimensionality reduction, it is equivalent to the high-resolution multispectral camera being able to simultaneously capture the color map perceived by the human eye, the color map perceived by the ultra-wide-angle camera, the color map perceived by the wide-angle (main camera) camera, and the color map perceived by the telephoto camera. Moreover, there is no difference between the captured color maps in terms of FOV, brightness, contrast, dynamic range, overexposure, etc., which can achieve better color consistency and higher authenticity of the images captured by multiple cameras in the camera array.

[0202] Therefore, the colors perceived by the high-resolution multispectral camera are very accurate, and the distribution of original images (RAW images) of the ultra-wide-angle camera, wide-angle (main camera) camera and telephoto camera with the same information source can be simulated through spectral band dimensionality reduction (that is, through spectral band dimensionality reduction, the perceived multi-light channel image is converted into a three-channel image). There is no FOV difference between the simulated multiple RAW image distributions, and there is no problem of inaccurate white balance. Therefore, the high-resolution multispectral camera can be used as a global color observer. Different types of cameras (including ultra-wide-angle cameras, wide-angle (main camera) cameras and telephoto cameras, including the first and second supplies of each module, etc.) are aligned with the colors perceived by the high-resolution multispectral camera, so that the color performance is highly consistent and highly realistic.

[0203] For example, Fig.11 FIG. 1 is a schematic flow chart of a method 1100 for camera color calibration provided in an embodiment of the present application. Fig.11 As shown, the method 1100 includes:

[0204] S1101: Turn on a first camera and a high-resolution multi-spectral camera of an electronic device at the same time, and the first camera of the electronic device collects a first image, where the first image is an original image of a first scene.

[0205] The first image is a three-channel image, for example, it can be an RGB image or a RYB image.

[0206] Among them, the first camera of the electronic device may include one or more of a wide-angle camera, an ultra-wide-angle camera, and a telephoto camera.

[0207] S1102: While the first camera is capturing the first image, the high-resolution multispectral camera of the electronic device is capturing the second image, where the second image is a multispectral original image of the first scene.

[0208] The second image is an M-channel image, where M is a positive integer greater than 3.

[0209] It can also be understood as: the first image captured by the first camera corresponds to spectral information of 3 frequency bands; the second image captured by the high-resolution multispectral camera corresponds to spectral information of M frequency bands, and the spatial color information of the second image is much richer than the spatial color information of the first image. In other words, the spectral information perceived by the high-resolution multispectral camera is richer than the spectral information perceived by the first camera, and is closer to the true color of the first scene.

[0210] S1103: Convert the second image from a multi-channel image to a three-channel image through a first conversion function, wherein the first conversion function is a conversion function between the high-resolution multi-spectral camera and the first camera, and is used to represent a spectral band mapping relationship between the high-resolution multi-spectral camera and the first camera.

[0211] In a specific embodiment, according to the first conversion function, the second image is spectrally downsampled to convert the second image from a multi-channel image to a three-channel image.

[0212] The first conversion function is a function that uses the downsampling matrix to perform matrix multiplication with the second image.

[0213] In some embodiments, the estimation method of the downsampling matrix used by the first conversion function may be any one of the Bayesian method, the least squares method, and the deep learning method, or may be other estimation methods.

[0214] Among them, downsampling, also known as downsampling or decimation, is one of the basic contents in digital signal processing. It refers to sampling a sample value sequence once every few sample values, so that the new sequence obtained is the downsampling of the original sequence. In image processing, the main purpose of downsampling is to reduce the image size to match the size of the display area or to generate a thumbnail of the corresponding image. If the image in matrix form is considered, the image in the s×s window of the original image is converted into a pixel, and the value of this pixel is the average or maximum value of all pixels in the window.

[0215] In image processing, the principle of downsampling is: for an image I of size M×N, downsample it s times to obtain a resolution image of size (M / s)×(N / s), where s should be a common divisor of M and N.

[0216] S1104: Using the three-channel image converted from the second image as a reference, color calibrate the first image to obtain first color parameters.

[0217] S1105: Send the first color parameter to the ISP channel corresponding to the first camera, so that the first camera adjusts the color of the image captured by the first camera according to the first color parameter.

[0218] In some embodiments, the electronic device also includes a second camera, which is a three-channel camera. The second camera can be color calibrated based on a high-resolution multi-spectral camera, and the color calibration process is the same as the color calibration process for the first camera mentioned above.

[0219] In an embodiment of the present application, a method for color calibration of an array camera based on a high-resolution multispectral camera is provided. In this method, since the images captured by the high-resolution multispectral camera are multi-channel images, their spatial color information is richer and the corresponding image color authenticity is higher. The images captured by the high-resolution multispectral camera are used as color references to calibrate the colors of the images captured by each camera in the camera array respectively, which can improve the authenticity and consistency of the colors of the images captured by the multiple cameras in the camera array. At the same time, since the high-resolution multispectral camera is small in size and the power consumption generated during the calibration process is low, the architectural redundancy and calibration cost of electronic equipment in the color management development process can be reduced. The method provides a unified baseline for each camera and improves debugging efficiency.

[0220] For example, Fig.12 FIG. 1 is a schematic flow chart of another camera color calibration method 1200 provided in an embodiment of the present application. Fig.12 As shown, the electronic device includes a camera array, the camera array includes a main camera, an ultra-wide-angle camera, and a telephoto camera, and the method 1200 includes:

[0221] S1201 to S1205 are the processes of color calibration of the main camera of the electronic device based on the high-resolution multi-spectral camera.

[0222] S1201: Turn on a main camera and a high-resolution multi-spectral camera of an electronic device at the same time, and the main camera of the electronic device collects a first image, where the first image is an original image of a first scene.

[0223] The first image is a three-channel image, for example, it can be an RGB image or a RYB image.

[0224] S1202: While the main camera is capturing the first image, the high-resolution multispectral camera of the electronic device is capturing the second image, where the second image is a multispectral original image of the first scene.

[0225] The second image is an M-channel image, where M is a positive integer greater than 3.

[0226] It can also be understood that: the first image captured by the main camera is a three-channel image, and the second image captured by the high-resolution multispectral camera is a multi-channel image. The spatial color information of the second image is much richer than the spatial color information of the first image. That is to say, the spectral information perceived by the high-resolution multispectral camera is richer than the spectral information perceived by the main camera, and is closer to the true color of the first scene.

[0227] S1203: Convert the second image from a multi-channel image into a three-channel image through a first conversion function, wherein the first conversion function is a conversion function between the high-resolution multi-spectral camera and the main camera, and is used to represent a spectral band mapping relationship between the high-resolution multi-spectral camera and the main camera.

[0228] In a specific embodiment, the second image is spectrally downsampled according to the first conversion function, thereby converting the second image from a multi-channel image to a three-channel image.

[0229] In one implementation, according to the first conversion function, downsampling is performed from the M (or k×k) channels corresponding to the second image to three channels, thereby converting the second image from a multi-channel image to a three-channel image, wherein the first conversion function may be a multi-channel image of size M×3 (or k×k). 2 ×3) downsampling matrix and the second image to perform matrix multiplication.

[0230] S1204: Using the three-channel image converted from the second image as a reference, perform color calibration on the first image to obtain first color parameters.

[0231] In some embodiments, the first color parameter may include any one or more of auto white balance (AWB) parameters, color adjustment (CA) parameters, color calibration (CC) parameters, and 3D look up table (3DLUT). In addition, the first color parameter may also be other color-related parameters, which is not limited in the present application.

[0232] S1205: Send the first color parameter to the ISP channel corresponding to the main camera, so that the main camera adjusts the color of the image captured by the main camera according to the first color parameter.

[0233] In some embodiments, S1201 to S1204 are an offline calibration process. After the first color parameters corresponding to the main camera are obtained through the offline calibration process, when the main camera performs image capture, when the captured image is processed by the ISP corresponding to the main camera, the color of the captured image is adjusted according to the first color parameters to make the image output by the main camera more authentic.

[0234] In some other embodiments, S1201 to S1204 are an online calibration process. When the user captures an image through the main camera, the high-resolution multispectral camera is turned on synchronously when the main camera is turned on. While the main camera is capturing images, the high-resolution multispectral camera captures images of the same scene. The first color parameter is obtained through the above S1201 to S1204, and when the captured image is processed by the ISP corresponding to the main camera, the color of the captured image is adjusted according to the first color parameter to make the image output by the main camera more authentic.

[0235] In some embodiments, the color standard process of the main camera may occur each time the main camera captures an image, or it may occur in response to a user's calibration operation, or it may be calibrated offline (or online) after taking a certain number of images, or it may be calibrated every certain period of time; it may also be other calibration times. The present application does not limit the timing of the color calibration of the camera, nor does it limit the calibration scene. It can be an offline calibration or an online calibration.

[0236] In the embodiment of the present application, since the second image contains a number of color channels of size M, the spectral information included is richer than the spectral information corresponding to the first image, so mapping it to the main camera can obtain the color space where the image is located, and the obtained three-channel image is closer to the true color of the first scene, and then the image after downsampling the second image channel can be used to adjust the color of the first image, thereby improving the color accuracy and consistency of the image taken by the main camera.

[0237] S1206 to S1210 are the processes of color calibration of the ultra-wide-angle camera of the electronic device based on the high-resolution multi-spectral camera.

[0238] S1206: Turn on the ultra-wide-angle camera and the high-resolution multi-spectral camera of the electronic device at the same time, and the ultra-wide-angle camera of the electronic device collects a third image, which is an original image of the second scene.

[0239] The third image is a three-channel image, for example, it can be an RGB image or a RYB image.

[0240] S1207: While the ultra-wide-angle camera is capturing the third image, the high-resolution multispectral camera of the electronic device is capturing a fourth image, where the fourth image is a multispectral original image of the second scene.

[0241] The fourth image is an M-channel image, where M is a positive integer greater than 3.

[0242] It can also be understood that: the third image captured by the ultra-wide-angle camera is a three-channel image, and the fourth image captured by the high-resolution multispectral camera is a multi-channel image. The spatial color information of the fourth image is far richer than the spatial color information of the third image. In other words, the spectral information perceived by the high-resolution multispectral camera is richer than the spectral information perceived by the ultra-wide-angle camera, and is closer to the true color of the second scene.

[0243] S1208: Convert the fourth image from a multi-channel image to a three-channel image through a second conversion function, wherein the second conversion function is a conversion function between a high-resolution multi-spectral camera and an ultra-wide-angle camera, and is used to represent a spectral band mapping relationship between the high-resolution multi-spectral camera and the ultra-wide-angle camera.

[0244] In a specific embodiment, the fourth image is spectrally downsampled according to the second conversion function, thereby converting the fourth image from a multi-channel image to a three-channel image.

[0245] In some embodiments, the estimation method of the downsampling matrix used by the second conversion function may be any one of the Bayesian method, the least squares method, and the deep learning method, or may be other estimation methods.

[0246] In one implementation, according to the second conversion function, downsampling is performed from the M (or k×k) channels corresponding to the fourth image to three channels, thereby converting the fourth image from a multi-channel image to a three-channel image, wherein the second conversion function may be a multi-channel image of size M×3 (or k×k). 2 ×3) and the fourth image by performing matrix multiplication.

[0247] S1209: Using the three-channel image converted from the fourth image as a reference, color calibrate the third image to obtain second color parameters.

[0248] The explanation of the second color parameter is the same as the above explanation of the first color parameter, and will not be repeated here for the sake of brevity.

[0249] S1210: Send the second color parameter to the ISP channel corresponding to the ultra-wide-angle camera, so that the ultra-wide-angle camera adjusts the color of the image captured by the ultra-wide-angle camera according to the second color parameter.

[0250] In some embodiments, the second scene is the same as the first scene, and the fourth image and the second image are the same image. That is, in some implementations, when the ultra-wide-angle camera and the main camera both capture (either synchronously or asynchronously within a certain time threshold) the original image of the first scene, S1207 may not be executed, and the high-resolution multispectral camera only needs to synchronously capture the multispectral original image of the first scene once, and the color calibration of the ultra-wide-angle camera and the main camera is implemented through the multispectral original image of the first scene.

[0251] In some embodiments, S1206 to S1209 are an offline calibration process. After the second color parameters corresponding to the ultra-wide-angle camera are obtained through the offline calibration process, when the ultra-wide-angle camera captures an image, when the captured image is processed by the ISP corresponding to the ultra-wide-angle camera, the color of the captured image is adjusted according to the second color parameters to make the image output by the ultra-wide-angle camera more authentic.

[0252] In some other embodiments, S1206 to S1209 are an online calibration process. When a user captures images through the ultra-wide-angle camera, the high-resolution multi-spectral camera is turned on when the ultra-wide-angle camera is turned on. While the ultra-wide-angle camera is capturing images, the high-resolution multi-spectral camera captures images of the same scene. The second color parameters are obtained through the above S1206 to S1209. When the captured image is processed by the ISP corresponding to the ultra-wide-angle camera, the color of the captured image is adjusted according to the second color parameters to make the image output by the ultra-wide-angle camera more authentic.

[0253] In some embodiments, the color standard process of the ultra-wide-angle camera can occur each time the ultra-wide-angle camera captures an image, or it can occur in response to a user's calibration operation, or it can be calibrated offline (or online) after taking a certain number of images, or it can be calibrated every certain period of time; it can also be other calibration times. The present application does not limit the timing of the color calibration of the camera, nor does it limit the calibration scene. It can be an offline calibration or an online calibration.

[0254] In the embodiment of the present application, since the fourth image contains a number of color channels of size M, the spectral information included is richer than the spectral information corresponding to the third image, so mapping it to the ultra-wide-angle camera can obtain the color space where the image is located, and the obtained three-channel image is closer to the true color of the second scene, and then the image after downsampling the fourth image channel can be used to adjust the color of the third image, thereby improving the color accuracy and consistency of the image taken by the ultra-wide-angle camera.

[0255] S1211 to S1215 are the processes of color calibration of the telephoto camera of the electronic device based on the high-resolution multi-spectral camera.

[0256] S1211: Turn on the telephoto camera and the high-resolution multispectral camera of the electronic device at the same time, and the telephoto camera of the electronic device collects a fifth image, which is an original image of the third scene.

[0257] The fifth image is a three-channel image, for example, it may be an RGB image or a RYB image.

[0258] S1212: While the telephoto camera is capturing the fifth image, the high-resolution multispectral camera of the electronic device is capturing a sixth image, where the sixth image is a multispectral original image of the third scene.

[0259] The sixth image is an M-channel image, where M is a positive integer greater than 3.

[0260] It can also be understood that: the fifth image captured by the telephoto camera is a three-channel image, and the sixth image captured by the high-resolution multispectral camera is a multi-channel image. The spatial color information of the sixth image is far richer than the spatial color information of the fifth image. That is to say, the spectral information perceived by the high-resolution multispectral camera is richer than the spectral information perceived by the telephoto camera, and is closer to the true color of the third scene.

[0261] S1213: Convert the sixth image from a multi-channel image to a three-channel image through a third conversion function, wherein the third conversion function is a conversion function between a high-resolution multispectral camera and a telephoto camera, and is used to represent a spectral band mapping relationship between the high-resolution multispectral camera and the telephoto camera.

[0262] In a specific embodiment, the sixth image is spectrally downsampled according to the third conversion function, thereby converting the sixth image from a multi-channel image to a three-channel image.

[0263] In some embodiments, the estimation method of the downsampling matrix used by the third conversion function may be any one of the Bayesian method, the least squares method, and the deep learning method, or may be other estimation methods.

[0264] In some embodiments, the first / second / third conversion functions mentioned above may be understood as first / second / third mapping parameters.

[0265] In one implementation, according to the third conversion function, downsampling is performed from the M (or k×k) channels corresponding to the sixth image to three channels, thereby converting the sixth image from a multi-channel image to a three-channel image, wherein the third conversion function may be a multi-channel image of size M×3 (or k×k). 2 ×3) and the sixth image by performing matrix multiplication.

[0266] S1214: Using the three-channel image converted from the sixth image as a reference, color calibrate the fifth image to obtain third color parameters.

[0267] The explanation of the third color parameter is the same as the above explanation of the first color parameter, and will not be repeated here for the sake of brevity.

[0268] S1215: Send the third color parameter to the ISP channel corresponding to the telephoto camera, so that the telephoto camera adjusts the color of the image captured by the telephoto camera according to the third color parameter.

[0269] In some embodiments, the third scene is the same as the first scene, and the sixth image and the second image are the same image. That is, in some implementations, when the telephoto camera and the main camera both capture (either synchronously or asynchronously within a certain time threshold) the original image of the first scene, S1212 may not be executed, and the high-resolution multispectral camera only needs to synchronously capture the multispectral original image of the first scene once, and the color calibration of the telephoto camera and the main camera is implemented through the multispectral original image of the first scene.

[0270] In some other embodiments, the third scene is the same as the second scene, and the sixth image and the fourth image are the same image. That is, in some implementations, when the telephoto camera and the ultra-wide-angle camera both capture (either synchronously or asynchronously within a certain time threshold) the original image of the second scene, S1212 may not be executed, and the high-resolution multispectral camera only needs to synchronously capture the multispectral original image of the second scene once, and the color calibration of the telephoto camera and the ultra-wide-angle camera is implemented through the multispectral original image of the second scene.

[0271] In some other embodiments, the third scene, the second scene and the first scene are all the same, and the sixth image, the fourth image and the second image are all the same image. That is, in some implementations, when the ultra-wide-angle camera, the telephoto camera and the main camera all capture (either synchronously or asynchronously within a certain time threshold) the original image of the first scene, S1207 and S1212 may not be executed, and the high-resolution multispectral camera only needs to synchronously capture the multispectral original image of the first scene once, and the color calibration of the telephoto camera, the ultra-wide-angle camera and the main camera is implemented through the multispectral original image of the first scene.

[0272] In some embodiments, S1211 to S1214 are an offline calibration process. After the third color parameter corresponding to the telephoto camera is obtained through the offline calibration process, when the telephoto camera captures an image, when the captured image is processed by the ISP corresponding to the telephoto camera, the color of the captured image is adjusted according to the third color parameter to make the image output by the telephoto camera more authentic.

[0273] In some other embodiments, S1211 to S1214 are an online calibration process. When the user captures images through the telephoto camera, the high-resolution multispectral camera is turned on simultaneously when the telephoto camera is turned on. While the telephoto camera is capturing images, the high-resolution multispectral camera is capturing images of the same scene. The third color parameter is obtained through the above S1211 to S1214, and when the captured image is processed by the ISP corresponding to the telephoto camera, the color of the captured image is adjusted according to the third color parameter to make the image output by the telephoto camera more authentic.

[0274] In some embodiments, the color standard process of the telephoto camera may occur each time the telephoto camera is used to capture an image, or it may occur in response to a user's calibration operation, or it may be calibrated offline (or online) after taking a certain number of images, or it may be calibrated every certain period of time; it may also be other calibration times. The present application does not limit the timing of the color calibration of the camera, nor does it limit the calibration scene. It can be an offline calibration or an online calibration.

[0275] Among them, the parameter types of the first color parameter, the second color parameter and the third color parameter should be consistent; that is, the color-related modules in the ISP channel of the main camera, the color-related modules in the ISP channel of the ultra-wide-angle camera, and the color-related modules in the ISP channel of the telephoto camera are the same.

[0276] In the embodiment of the present application, since the images captured by the high-resolution multispectral camera are multi-channel images, their spatial color information is richer and the corresponding image color authenticity is higher. The images captured by the high-resolution multispectral camera are used as the color reference to calibrate the colors of the images captured by each camera in the camera array, so that the colors of the pictures captured by each camera are aligned with the colors perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the colors of the images captured by multiple cameras in the camera array.

[0277] For example, Fig.13 A schematic diagram of a system framework corresponding to a camera color calibration method provided in an embodiment of the present application is shown.

[0278] like Fig.13 As shown, the electronic device includes a main camera, a high-resolution multi-spectral camera, a color parameter determination module, and an image signal processor (ISP). The system framework is used to perform the following steps:

[0279] S1301: Turn on a main camera and a high-resolution multispectral camera of an electronic device at the same time, and the main camera collects a first RAW image, where the first RAW image is a RAW image of a first scene.

[0280] The first RAW image is a three-channel image, for example, it may be an RGB image or a RYB image.

[0281] S1302: While the main camera is capturing the first RAW image, the high-resolution multispectral camera is capturing the second RAW image, where the second RAW image is a RAW image of the first scene.

[0282] The second RAW image is an M-channel image, where M is a positive integer greater than 3.

[0283] S1303: The color parameter determination module determines a first color parameter according to the first RAW image and the second RAW image, and sends the first color parameter to the ISP channel corresponding to the main camera.

[0284] Specifically, the color parameter determination module converts the second RAW image from an M-channel image to a three-channel image through a first conversion function, wherein the first conversion function is a conversion function between a high-resolution multispectral camera and a main camera, and is used to represent a spectral band mapping relationship between the high-resolution multispectral camera and the main camera; and then, based on the three-channel image converted from the second RAW image, the first RAW image is color calibrated to obtain a first color parameter.

[0285] The explanation of the first color parameter has been given in Fig.11A detailed explanation is given in the illustrated embodiment, and for the sake of brevity, it will not be repeated here.

[0286] S1304: The main camera adjusts the color of the image captured by the main camera according to the first color parameter.

[0287] In some embodiments, the color calibration of the main camera by the high-resolution multispectral camera is an online calibration. In the process of processing the first RAW image through the ISP channel corresponding to the main camera, the first RAW image is color adjusted according to the first color parameter, and the color-adjusted image is output, that is, the main camera image.

[0288] In some other embodiments, the color calibration of the main camera by the high-resolution multispectral camera is offline calibration. After acquiring the first color parameter, when an image is captured by the main camera, in the process of processing the captured image through the ISP channel corresponding to the main camera, the color of the captured image is adjusted according to the first color parameter, and the color-adjusted image is output, i.e., the main camera image.

[0289] Furthermore, the electronic device also includes an ultra-wide-angle camera, and the ultra-wide-angle camera of the electronic device can be color calibrated based on the high-resolution multi-spectral camera. The system architecture diagram corresponding to the color calibration process is similar to the above-mentioned system architecture diagram for color calibration of the main camera of the electronic device based on the high-resolution multi-spectral camera. For the sake of brevity, it will not be repeated here.

[0290] Furthermore, the electronic device also includes a telephoto camera, and the high-resolution multispectral camera can be used as a reference to perform color calibration on the telephoto camera of the electronic device. The system architecture diagram corresponding to the color calibration process is similar to the system architecture diagram for color calibration of the main camera of the electronic device based on the high-resolution multispectral camera mentioned above, and for the sake of brevity, it will not be repeated here. In the embodiment of the present application, the image captured by the high-resolution multispectral camera is used as the color reference to calibrate the color of the image captured by each camera in the camera array, so that the color of the picture captured by each camera is aligned with the color perceived by the high-resolution multispectral camera, which can improve the authenticity and consistency of the color of the images captured by multiple cameras.

[0291] For example, Fig.14 FIG. 14 is a flow chart of another camera color calibration method 1400 provided in an embodiment of the present application. Fig.14 As shown, the method 1400 includes:

[0292] S1401 and S1402 Fig.13S1301 and S1302 in the illustrated embodiment are the same and will not be described again for the sake of brevity.

[0293] S1403: Align the first RAW image and the second RAW image in terms of field of view (FOV).

[0294] In one implementation, the FOV cropping and alignment of the first RAW image and the second RAW image are performed by calibration.

[0295] S1404: Estimate a first conversion function based on the second RAW image after FOV alignment. The first conversion function can also be understood as a spectral band mapping parameter between the high-resolution multi-spectral camera and the main camera.

[0296] In some embodiments, the first transfer function is a down-sampling matrix for down-sampling from the FOV-aligned second RAW image to the FOV-aligned first RAW image.

[0297] The method for estimating the down-sampling matrix used by the first conversion function has been described in detail in the aforementioned embodiment, and will not be repeated here for the sake of brevity.

[0298] 1405: Convert the second RAW image after FOV alignment from a multi-channel image to a three-channel image through a first conversion function.

[0299] It should be understood that the three-channel image converted from the second RAW image has higher color authenticity than the three-channel image captured by the main camera.

[0300] In one implementation, M=k×k, and according to the first conversion function, the spectral band is downsampled from the k×k channels corresponding to the second RAW image to the three-channel image, thereby converting the second RAW image from a multi-channel image to a three-channel image, wherein the first conversion function uses a size of k 2 A function for performing matrix multiplication of a downsampling matrix of ×3 and the second image, where the value of k is a positive integer greater than or equal to 3.

[0301] S1406: Taking the three-channel image converted from the second RAW image after FOV alignment as a reference, statistical correction is performed on the first RAW image after FOV alignment to obtain a first color parameter corresponding to the main camera.

[0302] The explanation of the first color parameter has been given in Fig.11 A detailed explanation is given in the illustrated embodiment, and for the sake of brevity, it will not be repeated here.

[0303] In some embodiments, the aligned first RAW image is statistically corrected, and the statistical correction method used may include white point correction and the like.

[0304] S1407: Send the first color parameter to the ISP channel corresponding to the main camera.

[0305] S1408: In the ISP of the first camera, color adjustment is performed on the first RAW image after the FOV alignment according to the first color parameter, and the color-adjusted image is output, that is, the main camera image.

[0306] In some embodiments, in the ISP of the first camera, after acquiring the first color parameter, the main camera performs color adjustment on the image captured by the main camera according to the first color parameter.

[0307] Furthermore, the electronic device also includes an ultra-wide-angle camera, and the ultra-wide-angle camera of the electronic device can be color calibrated based on the high-resolution multi-spectral camera. The color calibration process can be similar to the above-mentioned process of color calibrating the main camera of the electronic device based on the high-resolution multi-spectral camera (S1401 to S1408), and for the sake of brevity, it will not be repeated here.

[0308] Furthermore, the electronic device also includes a telephoto camera, and the telephoto camera of the electronic device can be color calibrated based on the high-resolution multispectral camera. The color calibration process is similar to the above-mentioned process of color calibrating the main camera of the electronic device based on the high-resolution multispectral camera (S1401 to S1408), and will not be repeated here for the sake of brevity.

[0309] In an embodiment of the present application, a mapping relationship between a high-spectral-band resolution image acquired by a high-resolution multispectral camera and a low-spectral-band resolution (RGB three-channel) image of each array camera can be respectively constructed, and the high-spectral-band resolution image is downsampled to the low-spectral-band resolution image based on the mapping relationship to obtain a three-channel image with higher color authenticity corresponding to the low-spectral-band resolution image. The low-spectral-band resolution image is calibrated by the three-channel image with higher color authenticity, and the color parameters corresponding to each array camera can be respectively obtained. Then, each array camera adjusts the color of the captured image according to the corresponding color parameters, so that the colors of the pictures captured by each camera are aligned with the colors perceived by the high-resolution multispectral camera, thereby improving the color authenticity and consistency of the images captured by multiple cameras in the camera array.

[0310] For example, Fig.15 FIG. 1 is a flow chart of another camera color calibration method 1500 provided in an embodiment of the present application. Fig.15As shown, the method 1500 includes:

[0311] S1501: Turn on a main camera and a high-resolution multispectral camera at the same time, and the main camera collects a first RAW image, where the first RAW image is a RAW image of a first scene.

[0312] Among them, the explanation of this step is Fig.13 The explanation of S1301 in the illustrated embodiments is the same and will not be repeated here for the sake of brevity.

[0313] S1502: Generate a first response curve corresponding to the first RAW image.

[0314] The first response curve can be understood as the spectral perception curve corresponding to the first RAW image. For details, see Fig.10 The embodiment shown.

[0315] S1503: While the main camera is capturing the first RAW image, the high-resolution multispectral camera is capturing the second RAW image, where the second RAW image is a RAW image of the first scene.

[0316] Among them, the explanation of this step is Fig.13 The explanation of S1302 in the illustrated embodiments is the same and will not be repeated here for the sake of brevity.

[0317] S1504: Generate a second response curve corresponding to the second RAW image.

[0318] The second response curve can be understood as the spectral perception curve corresponding to the second RAW image. For details, see Fig.10 The embodiment shown.

[0319] S1505: Based on the second response curve, perform spectral band super-resolution on the second RAW image to obtain a hyper-spectral image corresponding to the second RAW image.

[0320] It should be understood that the number of spectral bands corresponding to the hyperspectral image corresponding to the second RAW image is much greater than the number of spectral bands corresponding to the second RAW image.

[0321] In some embodiments, the second RAW image corresponds to M spectral bands, and the hyperspectral image corresponding to the second RAW image corresponds to N spectral bands, wherein N is much larger than M. When M is k 2 When N>>k 2 .

[0322] In some embodiments, the method for performing spectral band super-resolution on the second RAW image includes a super-resolution reconstruction method based on compressed sensing and / or a spectral band super-resolution method based on a neural network. In addition, it can also be other spectral band super-resolution methods, which are not limited in this application.

[0323] S1506: Determine a first conversion function according to the hyperspectral image corresponding to the second RAW image and the first response curve. The first conversion function can also be understood as a spectral band mapping parameter between the high-resolution multispectral camera and the main camera.

[0324] In some embodiments, the first conversion function is a function of performing matrix multiplication of a downsampling matrix for downsampling the hyperspectral image corresponding to the second RAW image to the first RAW image and the second image.

[0325] S1507: Convert the second RAW image into a three-channel image by using the first conversion function.

[0326] It should be understood that the three-channel image converted from the second RAW image has higher color authenticity than the three-channel image captured by the main camera.

[0327] In one implementation, based on a downsampling matrix of size N×3, spectral band downsampling is performed from N channels corresponding to the hyperspectral image corresponding to the second RAW image to three channels, and the hyperspectral image corresponding to the second RAW image is converted into a three-channel image, thereby converting the second RAW image from a multi-channel image to a three-channel image, where N is much greater than M.

[0328] S1508: Taking the converted three-channel image as a reference, statistically correct the first RAW image to obtain a first color parameter corresponding to the main camera.

[0329] The explanation of the first color parameter has been given in Fig.11 A detailed explanation is given in the illustrated embodiment, and for the sake of brevity, it will not be repeated here.

[0330] S1509: Send the first color parameter to the ISP channel corresponding to the main camera.

[0331] S1510: In the ISP of the first camera, color adjustment is performed on the first RAW image according to the first color parameter, and the color-adjusted image is output, that is, the main camera image.

[0332] In some embodiments, in the ISP of the main camera, after acquiring the first color parameter, the main camera performs color adjustment on the image captured by the main camera according to the first color parameter.

[0333] Furthermore, the electronic device also includes an ultra-wide-angle camera, and the ultra-wide-angle camera of the electronic device can be color calibrated based on the high-resolution multi-spectral camera. The color calibration process can be similar to the above-mentioned process of color calibrating the main camera of the electronic device based on the high-resolution multi-spectral camera (S1501 to S1510), and for the sake of brevity, it will not be repeated here.

[0334] Furthermore, the electronic device also includes a telephoto camera, and the telephoto camera of the electronic device can be color calibrated based on the high-resolution multispectral camera. The color calibration process is similar to the above-mentioned process of color calibrating the main camera of the electronic device based on the high-resolution multispectral camera (S1501 to S1510), and for the sake of brevity, it will not be repeated here.

[0335] In one implementation, the main camera performs online color calibration during the first shooting process to obtain a first color parameter, and performs color adjustment on the image obtained by the first shooting according to the first color parameter. Thereafter, when the main camera shoots an image again, the color of the captured image is directly adjusted according to the first color parameter.

[0336] In another implementation, the main camera obtains a first color parameter through online calibration or offline calibration in a first time period, and the images captured in the first time period are color-adjusted according to the first color parameter; the main camera obtains a second color parameter through online calibration or offline calibration in a second time period, and the images captured in the second time period are color-adjusted according to the second color parameter.

[0337] In yet another implementation, the main camera obtains the first color parameter in advance through offline calibration, and when the main camera captures an image, the color of the captured image is adjusted according to the first color parameter.

[0338] It should be understood that the multispectral camera described in the embodiments of the present application (or the above-mentioned high-resolution multispectral camera, or the above-mentioned second camera) can be a color filter array (CFA) that realizes differentiation through chemical dyes in terms of sensor function implementation; it can also be a CFA that realizes differentiation through coating interference; it can also be a CFA that realizes differentiation through a metasurface Wiener structure; it can also be other sensor function implementation methods, which are not limited in the present application.

[0339] It should also be understood that the multispectral camera described in the embodiments of the present application (or the high-resolution multispectral camera mentioned above, or the second camera mentioned above) can be independent of the existing cameras of the electronic device (for example, wide-angle, ultra-wide-angle, telephoto, etc.) in terms of module function implementation, and can also form a certain structural appearance with the existing camera, such as a hybrid structural appearance; it can also be other module function implementation methods, which are not limited in the present application.

[0340] The above-mentioned color parameters may include one or more parameters corresponding to the color module in the ISP channel of the camera, and may also include one or more parameters corresponding to the brightness module in the ISP channel of the camera.

[0341] One or more of the modules or units described herein can be implemented by software, hardware or a combination of the two. When any of the above modules or units are implemented in software, the software exists in the form of computer program instructions and is stored in a memory, and a processor can be used to execute the program instructions and implement the above method flow. The processor may include but is not limited to at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller (MCU), or an artificial intelligence processor and other types of computing devices that run software, each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor may be built into an SoC (system on chip) or an application specific integrated circuit (ASIC), or it may be an independent semiconductor chip. In addition to the core used to execute software instructions for operations or processing in the processor, necessary hardware accelerators may be further included, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or a logic circuit that implements a dedicated logic operation.

[0342] When the modules or units described in this document are implemented in hardware, the hardware can be any one or any combination of a CPU, a microprocessor, a DSP, an MCU, an artificial intelligence processor, an ASIC, a SoC, an FPGA, a PLD, a dedicated digital circuit, a hardware accelerator, or a non-integrated discrete device, which can run the necessary software or be independent of the software to execute the above method flow.

[0343] When the modules or units described herein are implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0344] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0345] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0346] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0347] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0348] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0349] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0350] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for camera color calibration, characterized in that: The method is applied to an electronic device, the electronic device comprising a camera array and a high-resolution multi-spectral camera, and the method comprises: When the first camera and the high-resolution multi-spectral camera are turned on at the same time, a first image is collected by the first camera, the first image is an original image of a first scene, the first image is a three-channel image, and the camera array includes the first camera; Synchronously collecting a second image through the high-resolution multispectral camera, where the second image is a multispectral original image of the first scene, and the second image is an M-channel image, where M is a positive integer greater than 3; Converting the second image from an M-channel image to a three-channel image by a first conversion function; Taking the three-channel image converted from the second image as a reference, color calibrating the first image to obtain first color parameters; The first color parameter is sent to an image signal processor ISP channel of the first camera, so that the first camera adjusts the color of the image captured by the first camera according to the first color parameter.

2. The method according to claim 1, characterized in that The camera array further includes a second camera, and the method further includes: Acquire a third image through the second camera, wherein the third image is an original image of the first scene and is a three-channel image; Taking the three-channel image converted from the second image as a reference, color calibrating the third image to obtain second color parameters; The second color parameter is sent to an image signal processor ISP channel of the second camera, so that the second camera adjusts the color of the image captured by the second camera according to the second color parameter.

3. The method according to claim 1, characterized in that The camera array further includes a second camera, and the method further includes: When the second camera and the high-resolution multi-spectral camera are turned on at the same time, a third image is collected by the second camera, where the third image is an original image of the second scene and is a three-channel image; Synchronously collecting a fourth image by the high-resolution multispectral camera, wherein the fourth image is a multispectral original image of the second scene, and the fourth image is an M-channel image; Converting the fourth image from an M-channel image to a three-channel image through a second conversion function; Taking the three-channel image converted from the fourth image as a reference, color calibrating the third image to obtain second color parameters; The second color parameter is sent to an image signal processor ISP channel of the second camera, so that the second camera adjusts the color of the image captured by the second camera according to the second color parameter.

4. The method according to claim 2 or 3, characterized in that: The first camera is any one of a main camera, an ultra-wide-angle camera, and a telephoto camera, and the second camera is any one of a main camera, an ultra-wide-angle camera, and a telephoto camera, and the first camera and the second camera are different.

5. The method according to any one of claims 1 to 4, characterized in that The converting the second image from an M-channel image to a three-channel image by using a first conversion function comprises: The second image is converted from an M-channel image to a three-channel image by downsampling from the M channels of the second image to the three channels of the first image according to the first conversion function, wherein the first conversion function is a function that uses a downsampling matrix of size M×3 to perform matrix multiplication with the second image.

6. The method according to claim 5, characterized in that Before converting the second image from an M-channel image to a three-channel image by using a first conversion function, the method further includes: The first image and the second image are aligned in terms of field of view (FOV).

7. The method according to claim 6, characterized in that The method further comprises: The first conversion function is determined based on the second image after FOV alignment.

8. The method according to any one of claims 1 to 4, characterized in that The converting the second image from an M-channel image to a three-channel image by using a first conversion function comprises: Performing spectral band super-resolution on the second image according to the spectral sensing curve corresponding to the second image to obtain a hyper-spectral image corresponding to the second image, wherein the hyper-spectral image corresponding to the second image is an N-channel image, where N is a positive integer much greater than M; The hyperspectral image corresponding to the second image is converted from an N-channel image to a three-channel image by downsampling from the N channels of the hyperspectral image corresponding to the second image to the three channels of the first image according to the first conversion function, wherein the first conversion function is a function that uses a downsampling matrix of size N×3 to perform matrix multiplication with the second image.

9. The method according to claim 8, characterized in that The method further comprises: The first conversion function is determined according to the hyperspectral image corresponding to the second image and the spectral sensing curve corresponding to the first image.

10. The method according to any one of claims 1 to 9, characterized in that The first color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

11. An electronic device, characterized in that: The electronic device comprises a camera array, a high-resolution multi-spectral camera and a processor, wherein the camera array comprises a first camera, wherein: The first camera is used to capture a first image when the first camera and the high-resolution multi-spectral camera are turned on at the same time, wherein the first image is an original image of a first scene and the first image is a three-channel image; The high-resolution multispectral camera is used to synchronously capture a second image, where the second image is a multispectral original image of the first scene, and the second image is an M-channel image, where M is a positive integer greater than 3; The processor is used to convert the second image from an M-channel image to a three-channel image through a first conversion function; The processor is further configured to perform color calibration on the first image based on the three-channel image converted from the second image to obtain first color parameters; The processor is further used to send the first color parameter to an image signal processor ISP channel of the first camera, so that the first camera can adjust the color of the image captured by the first camera according to the first color parameter.

12. The electronic device according to claim 11, characterized in that: The camera array further includes a second camera, wherein: The second camera is used to capture a third image, where the third image is an original image of the first scene and is a three-channel image; The processor is further configured to perform color calibration on the third image based on the three-channel image converted from the second image to obtain second color parameters; The processor is further used to send the second color parameter to the image signal processor ISP channel of the second camera, so that the second camera adjusts the color of the image captured by the second camera according to the second color parameter.

13. The electronic device according to claim 11, characterized in that: The camera array further includes a second camera, wherein: The second camera is used to collect a third image through the second camera when the second camera and the high-resolution multi-spectral camera are turned on at the same time, wherein the third image is an original image of the second scene and is a three-channel image; The high-resolution multispectral camera is further used to synchronously capture a fourth image, where the fourth image is a multispectral original image of the second scene, and the fourth image is an M-channel image; The processor is further used to convert the fourth image from an M-channel image to a three-channel image through a second conversion function; The processor is further configured to perform color calibration on the third image based on the three-channel image converted from the fourth image to obtain second color parameters; The processor is further used to send the second color parameter to the image signal processor ISP channel of the second camera, so that the second camera can adjust the color of the image captured by the second camera according to the second color parameter.

14. The electronic device according to claim 12 or 13, characterized in that: The first camera is any one of a main camera, an ultra-wide-angle camera, and a telephoto camera, and the second camera is any one of a main camera, an ultra-wide-angle camera, and a telephoto camera, and the first camera and the second camera are different.

15. The electronic device according to any one of claims 11 to 14, characterized in that: The processor is specifically used for: The second image is converted from an M-channel image to a three-channel image by downsampling from the M channels of the second image to the three channels of the first image according to the first conversion function, wherein the first conversion function is a function that uses a downsampling matrix of size M×3 to perform matrix multiplication with the second image.

16. The electronic device according to claim 15, characterized in that: The processor is further specifically configured to: Before converting the second image from an M-channel image to a three-channel image through a first conversion function, the first image and the second image are aligned with respect to the viewing angle FOV.

17. The electronic device according to claim 16, characterized in that: The processor is further configured to: The first conversion function is determined based on the second image after FOV alignment.

18. The electronic device according to any one of claims 11 to 14, characterized in that: The processor is specifically used for: Performing spectral band super-resolution on the second image according to the spectral sensing curve corresponding to the second image to obtain a hyper-spectral image corresponding to the second image, wherein the hyper-spectral image corresponding to the second image is an N-channel image, where N is a positive integer much greater than M; The hyperspectral image corresponding to the second image is converted from an N-channel image to a three-channel image by downsampling from the N channels of the hyperspectral image corresponding to the second image to the three channels of the first image according to the first conversion function, wherein the first conversion function is a function that uses a downsampling matrix of size N×3 to perform matrix multiplication with the second image.

19. The electronic device according to claim 18, characterized in that: The processor is further configured to: The first conversion function is determined according to the hyperspectral image corresponding to the second image and the spectral sensing curve corresponding to the first image.

20. The electronic device according to any one of claims 11 to 19, characterized in that: The first color parameter includes any one or more of an automatic white balance parameter, a color adjustment parameter, a color calibration parameter, and a 3D lookup table.

21. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 10 is implemented.

23. A chip, characterized in that: Instructions are stored in the chip, and when the instructions are executed, the method according to any one of claims 1 to 10 is implemented.

24. A computer program product, characterized in that The computer program product stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 10 is implemented.

Images