Multispectral sensors, camera modules, electronic devices and image processing methods

By setting two color filtering elements in a single pixel unit and using a microelectromechanical system (MEMS) to drive the components for dynamic switching, the problems of increased size and cost of multispectral sensor devices are solved, achieving higher spectral resolution and color reproduction accuracy.

CN123123686APending Publication Date: 2026-11-17VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611254708.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing multispectral sensors, while capable of sensing both visible light and infrared light, suffer from increased device size and cost. Furthermore, the fixed CFA arrangement in traditional multispectral sensors leads to a decrease in single-channel sampling frequency, which can easily cause zipper effect and false color problems.

Method used

Two color filter elements are set in a single pixel unit, and their dynamic switching between different CFA arrangement modes is controlled by a microelectromechanical system drive component. The precise position switching of the color filter elements is achieved by using a planar frame capacitor electrostatic driver. By combining Bayer filter array and multispectral filter array modes, different spectral bands can be dynamically switched to sense.

Benefits of technology

It achieves double the color channel information with the same total pixel size, improves spectral resolution and color reproduction accuracy, avoids the increase in device size and cost, and solves the problems of decreased single-channel sampling frequency and false color in traditional multispectral sensors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123686A_ABST
    Figure CN123123686A_ABST
Patent Text Reader

Abstract

This application discloses a multispectral sensor, camera module, electronic device, and image processing method, belonging to the field of image processing. The multispectral sensor includes: multiple pixel units, each pixel unit including a photoelectric conversion element, a first color filter element, and a second color filter element; and a driving component connected to the first color filter element and the second color filter element, used to drive the first color filter element or the second color filter element to move to a first working position or a second working position, so that the multispectral sensor switches between at least two color filter array modes; wherein, the first working position is located in the imaging optical path of the photoelectric conversion element, and the second working position is located in the blind zone of the imaging optical path.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a multispectral sensor, camera module, electronic device, and image processing method. Background Technology

[0002] Color reproduction in image sensors primarily relies on the color filter array (CFA) above the pixel unit. In related technologies, image sensors in terminal cameras often use Bayer filter arrays, meaning each pixel unit can only perceive one of the three colors: red, green, and blue. The remaining color information is estimated through interpolation algorithms, which can easily lead to image quality problems such as the zipper effect and false color.

[0003] While multispectral sensors can acquire richer spectral information and achieve more accurate color reproduction by increasing the number of color channels, the increase in the number of channels leads to a decrease in the sampling frequency of a single channel. In addition, the CFA arrangement mode in traditional multispectral sensors is fixed. If it is necessary to take into account multispectral sensing such as visible light and infrared, multiple multispectral sensors are often required in the image sensor, each equipped with a different CFA, which leads to an increase in device size and cost. Summary of the Invention

[0004] The purpose of this application is to provide a multispectral sensor, camera module, electronic device, and image processing method. By setting two color filtering elements in a single pixel unit and using a driving component to control them to enter or leave the optical path respectively, the same multispectral sensor can dynamically switch between different CFA arrangement modes according to usage requirements. This takes into account multiple spectral sensing needs and avoids the problems of increased device size and cost caused by setting multiple multispectral sensors equipped with different CFAs inside the image sensor.

[0005] In a first aspect, embodiments of this application provide a multispectral sensor, comprising: a plurality of pixel units, each pixel unit including a photoelectric conversion element, a first color filter element, and a second color filter element; and a driving component connected to the first color filter element and the second color filter element, configured to drive the first color filter element or the second color filter element to move to a first working position or a second working position, thereby enabling the multispectral sensor to switch between at least two color filter array modes; wherein the first working position is a position located in the imaging optical path of the photoelectric conversion element, and the second working position is a position located in the blind zone of the imaging optical path.

[0006] In one possible implementation, the driving component is a microelectromechanical system (MEMS) driving component, which includes a planar frame capacitor electrostatic driver, and the planar frame capacitor electrostatic driver is connected to the first color filter element and the second color filter element respectively.

[0007] This implementation uses a planar frame capacitor electrostatic driver in the microelectromechanical system (MEMS) drive assembly as the driving element. It has the advantages of small size, low power consumption, fast response speed and high control precision. It can accurately control the position switching of the color filter element in each pixel unit and is compatible with CMOS image sensor technology, which is convenient for integrated manufacturing.

[0008] In one possible implementation, the planar frame capacitor electrostatic driver includes an upper electrode plate and a lower electrode plate; the upper electrode plate is disposed on the first color filter element and the second color filter element; the lower electrode plate is disposed on the photosensitive side of the photoelectric conversion element and is disposed opposite to the upper electrode plate.

[0009] This implementation uses a flat-plate capacitor electrostatic driver consisting of an upper plate, a lower plate, and an insulating spring, resulting in a simple structure that is easy to manufacture. The upper plate moves with the color filter element, while the lower plate is fixed to the photosensitive side, ensuring the stability and reliability of the electrostatic drive.

[0010] In one possible implementation, the at least two color filter array modes include a first color filter array mode and a second color filter array mode, wherein the color filter array in the first color filter array mode includes a first set of color channels, and the color filter array in the second color filter array mode includes a second set of color channels, and the first set of color channels and the second set of color channels are at least partially different.

[0011] This implementation method sets two sets of multispectral modes with different color channels in the same multispectral sensor and dynamically switches them using a driving component, enabling the acquisition of two different multispectral images on a single sensor. Compared to traditional multispectral sensors with a fixed CFA arrangement, this implementation method can obtain twice the color channel information with the same total pixel size, thereby achieving finer spectral resolution and more accurate color reproduction.

[0012] In one possible implementation, the first set of color channels and the second set of color channels are used to sense light in different spectral bands; the different spectral bands include at least two of the visible light band, infrared band, and ultraviolet band.

[0013] This implementation design allows a single multispectral sensor to simultaneously cover multiple spectral bands, such as visible light, infrared, and ultraviolet light, by designing the color channels of the two multispectral modes to sense different spectral bands. Compared to traditional solutions that require multiple sensors to cover different bands, this approach significantly reduces device size while expanding the spectral sensing coverage.

[0014] In one possible implementation, the at least two color filter array modes include a Bayer filter array mode and a multispectral filter array mode; the Bayer filter array mode includes red, green, and blue filter elements; and the multispectral filter array mode includes filter elements of at least four colors.

[0015] This implementation dynamically switches between Bayer filter array mode and multispectral filter array mode within the same image sensor. This allows a single sensor to function as a traditional RGB sensor for regular imaging, while also switching to multispectral mode to acquire rich spectral information. In subsequent image processing, the images acquired in both modes are combined to output a pixel-level color-corrected RGB image, exhibiting significantly higher color accuracy than traditional Bayer images. Furthermore, compared to a dual-camera solution that combines a traditional camera and a multispectral camera within the image sensor, this implementation avoids field-of-view alignment issues between multiple sensors, achieving more accurate pixel-level color reproduction.

[0016] Secondly, embodiments of this application provide a camera module, including the multispectral sensor described in the first aspect.

[0017] Thirdly, embodiments of this application provide an electronic device, including: a housing; and a camera module, the camera module including the multispectral sensor described in the first aspect.

[0018] Fourthly, embodiments of this application provide an image processing method executed by the electronic device described in the third aspect, comprising: controlling the multispectral sensor to expose in a first color filter array mode to acquire a first image; wherein, in the first color filter array mode, the first color filter element is located at a first working position and the second color filter element is located at a second working position; controlling the multispectral sensor to expose in a second color filter array mode to acquire a second image; wherein, in the second color filter array mode, the first color filter element is located at the second working position and the second color filter element is located at the first working position; performing downsampling processing on the first image and the second image to extract single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode respectively; inputting the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode respectively into an image processing model to output a third image.

[0019] This implementation acquires images under two different CFA modes through two exposures, then downsamples and extracts single-channel images of each color channel as input to the image processing model. This allows the model to perform inference based on richer color channel information, resulting in a higher-quality third image. Compared to traditional fixed CFA single-frame image processing methods, this implementation utilizes dual-mode switching to acquire more spectral channel information, providing a more sufficient data foundation for subsequent image processing.

[0020] In one possible implementation, the third image is an RGB image that has undergone color restoration correction, or the third image is the spectral information of the shooting scene corresponding to the first image and the second image.

[0021] This implementation method allows the same image processing method to output different types of image processing results by setting different training objectives, thereby meeting different application needs such as color restoration and spectral analysis, and expanding the applicability of the technical solution of this application.

[0022] In one possible implementation, the method further includes: acquiring RGB images of the shooting scene of the first image and the second image using an auxiliary camera; the step of inputting the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode into the image processing model includes: inputting the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode and the RGB image together into the image processing model.

[0023] This implementation uses an auxiliary camera to simultaneously acquire RGB images of the same scene as supplementary information for the image processing model's input. This allows the model to combine the texture information of traditional RGB images with the spectral information of multispectral images, achieving more accurate pixel-level color reproduction. The RGB images from the auxiliary camera can serve as additional feature input, enhancing the image processing model's ability to perceive scene colors and lighting.

[0024] In one possible implementation, the training process of the image processing model includes: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode, and RGB images acquired by the auxiliary camera; the training target set is a reference RGB image.

[0025] This implementation introduces RGB images of the same scene simultaneously captured by an auxiliary camera as additional input, enabling the image processing model to obtain dual information from both traditional RGB images and multispectral images during both training and inference phases. On one hand, the RGB images provided by the auxiliary camera contain rich texture information and color distribution features, compensating for the spatial resolution loss due to the low single-channel sampling rate of the multispectral sensor. On the other hand, the single-channel images of multiple color channels provided by the multispectral sensor contain detailed spectral information, reflecting the spectral characteristics of different substances in the scene. By inputting both into the deep learning model, the model can fuse the texture features of the RGB image and the spectral features of the multispectral image, thereby achieving pixel-level accurate color reproduction. Compared to methods that only use multispectral images as model input, the technical solution in this embodiment can output RGB images with more accurate colors and richer details, effectively improving the quality and robustness of color reproduction.

[0026] In one possible implementation, the training process of the image processing model includes: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode; and the training target set is the spectral curve acquired by the spectrometer.

[0027] This implementation uses real spectral curves acquired by a spectrometer as the training target, enabling the trained model to accurately reconstruct the spectral information of a scene based on channel images obtained from a multispectral sensor. This method requires no auxiliary camera and is suitable for applications that require detailed spectral data rather than RGB images, such as color recognition, light source detection, food safety analysis, skin analysis, and water quality testing.

[0028] In one possible implementation, the training process of the image processing model includes: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes RGB images or single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode, and single-channel images of each color channel acquired in the second color filter array mode; the training target set is a reference RGB image.

[0029] This implementation method allows model training and inference to be completed using images acquired by the multispectral sensor itself in different modes. This saves on the hardware costs and device size overhead associated with auxiliary cameras and their related circuitry, making it advantageous for implementing multispectral imaging and color reproduction functions in devices with limited hardware resources. The Bayer RGB image acquired in the first color filter array mode has the same color expression as the traditional RGB image. After downsampling to extract the R, G, and B single-channel images, these images, along with the single-channel images of each color channel of the multispectral image acquired in the second color filter array mode, constitute the training input. This allows the model to directly establish the mapping relationship between the Bayer RGB channels and the multispectral channels to the reference RGB image, improving the accuracy of color reproduction without adding additional hardware.

[0030] Fifthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the fourth aspect.

[0031] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method described in the fourth aspect.

[0032] In a seventh aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the fourth aspect.

[0033] In this embodiment, by setting two color filtering elements in a single pixel unit and using a driving component to control them to enter or leave the optical path respectively, the same multispectral sensor can dynamically switch between different CFA arrangement modes according to usage requirements, thereby taking into account multiple spectral sensing needs and avoiding the problems of increased device size and cost caused by setting multiple multispectral sensors equipped with different CFAs inside the image sensor.

[0034] The second and third aspects mentioned above have various possible structural designs similar to the first aspect and any possible implementation of the first aspect, and can produce corresponding technical effects, which will not be elaborated here. Attached Figure Description

[0035] Figure 1 A schematic diagram of a single pixel structure of a multispectral sensor provided for some embodiments of this application; Figure 2 Schematic diagrams of MEMS planar capacitor electrostatic drivers provided for some embodiments of this application; Figure 3 A schematic flowchart illustrating an image processing method provided for some embodiments of this application; Figure 4 A schematic diagram of the CFA arrangement for switching between multispectral mode 1 and multispectral mode 2 provided for some embodiments of this application; Figure 5 A schematic diagram illustrating the extraction of a single-channel image from two frames of multispectral images, provided for some embodiments of this application; Figure 6 A schematic diagram of the CFA arrangement for switching between Bayer mode and multispectral mode provided for some embodiments of this application; Figure 7 Schematic diagrams of the structure of electronic devices provided for some embodiments of this application; Figure 8 A schematic diagram of the structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0037] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0038] To achieve finer spectral resolution or a wider spectral sensing range, multispectral sensors need more color channels to sense different wavelengths of light. However, increasing the number of color channels reduces the number of pixels per color channel for the same total pixel size, resulting in a lower single-channel sampling frequency. Using a conventional Bayer filter array as a reference, the red channel pixels account for only 1 / 4 of the total pixels, and adjacent red pixels are separated by a green pixel in the row and column directions. In the final RGB imaging, the remaining 3 / 4 of the red pixel information for each pixel needs to be estimated using demosaic interpolation algorithms. Interpolation predictions inevitably contain errors; common interpolation problems include the zipper effect, false color, and moiré patterns. For example, in a multispectral sensor with 9 (3×3) color channels, adjacent pixels in the same color channel are separated by two pixels from other color channels in the row and column directions, meaning each color channel only accounts for 1 / 9 of the total pixels. The single-channel resolution decreases significantly compared to a traditional RGB sensor. Even when using interpolation algorithms to try to improve single-channel resolution, the aforementioned interpolation artifacts are more likely to occur.

[0039] The purpose of this application is to provide a multispectral sensor, camera module, electronic device, and image processing method. By controlling the movement of a color filter array relative to the photosensitive surface on the sensor through a driving component, the sensor can dynamically switch between different color filter array modes. Compared to traditional multispectral sensors with fixed color filter array modes, the variable color filter array of this application can switch between at least two different multispectral modes, fully utilizing the color information of two multispectral modes on a single image sensor. For example, one mode primarily senses visible light, while another mode primarily senses infrared or ultraviolet light. This allows the multispectral sensor to achieve finer spectral resolution or a wider spectral sensing range while saving device size and achieving higher sampling accuracy. Furthermore, by switching between multispectral modes and traditional Bayer modes, a single image sensor can function as both a traditional RGB sensor for imaging and fully utilize the color information of multispectral modes, thereby achieving pixel-to-pixel accurate color reproduction.

[0040] The multispectral sensor and image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0041] This application provides a multispectral sensor, including multiple pixel units. For example... Figure 1 As shown, each pixel unit may include a photoelectric conversion element 10, a first color filter element 20, and a second color filter element 30. The first color filter element 20 and the second color filter element 30 are disposed above the photoelectric conversion element 10 and are used to filter the incident light at different wavelengths. The photoelectric conversion element 10 may be a photodiode.

[0042] The multispectral sensor also includes a driving component 40, which is connected to a first color filter element 20 and a second color filter element 30. The driving component 40 drives either the first color filter element 20 or the second color filter element 30 to move to a first working position or a second working position. The first working position is where the color filter element is located within the imaging optical path of the photoelectric conversion element 10, and the second working position is where the color filter element is located in the blind zone of the imaging optical path. Alternatively, the first working position is where the color filter element is parallel to and above the photosensitive surface of the photoelectric conversion element 10 within the imaging optical path, and the second working position is where the color filter element is perpendicular to the photosensitive surface and away from the imaging optical path. The color filter element is either the first color filter element 20 or the second color filter element 30. In other words, the first working position is within the imaging optical path of the photoelectric conversion element, and the second working position is in the blind zone of the imaging optical path. That is, the color filter element filters the incident light in the first working position, and in the second working position, it has virtually no impact on the incident light reaching the photoelectric conversion element.

[0043] By controlling the positional state combination of the first color filter element 20 and the second color filter element 30 in all pixel units through the driving component 40, the multispectral sensor can switch between at least two color filter array modes.

[0044] This application embodiment sets two color filtering elements in a single pixel unit and uses a driving component to control them to enter or leave the optical path respectively. This allows the same multispectral sensor to dynamically switch between different CFA arrangement modes according to usage requirements, thereby taking into account multiple spectral sensing needs and avoiding the problems of increased device size and cost caused by setting multiple multispectral sensors equipped with different CFAs inside the image sensor.

[0045] Compared to conventional image sensors, the multispectral sensor provided in this application embodiment can have multiple color filtering elements in each pixel unit, each driven independently by a corresponding driving component. As an optional implementation of this application, each pixel unit can have two color filtering elements, designated as the first color filtering element 20 and the second color filtering element 30, respectively. These are driven independently by their respective driving components 40. It should be noted that the number of color filtering elements in a pixel unit is not limited to two. In other embodiments, each pixel unit can also have three, four, or more color filtering elements. The driving components can selectively drive at least two of these color filtering elements to a first working position in the optical path and a second working position away from the optical path, thereby achieving switching between two or more color filtering array modes. Those skilled in the art will understand that the above variations are all within the scope of this application.

[0046] Furthermore, in various embodiments of this application, all or most of the pixel units in the plurality of pixel units can adopt the above-described structure, that is, all or most of the pixel units are respectively provided with a plurality of color filtering elements and corresponding driving components. In embodiments where most pixel units adopt the above-described structure, the remaining pixel units that are not provided with a plurality of color filtering elements can be used as reference pixels or auxiliary pixels, without affecting the overall switching function of the multispectral sensor between at least two color filtering array modes.

[0047] In some embodiments of this application, the driving component 40 can be a microelectromechanical system (MEMS) driving component 40. This MEMS driving component 40 includes a planar frame capacitor electrostatic driver, which is connected to the first color filter element 20 and the second color filter element 30, respectively.

[0048] When a voltage is applied to the upper and lower plates of the flat plate capacitor, the resulting electrostatic force drives the upper plate to move relative to the lower plate, thereby causing the first color filter element 20 or the second color filter element 30 connected to it to move between a first working position in the optical path and a second working position away from the optical path.

[0049] Specifically, when a voltage is applied to the upper electrode plate above the pixel unit and the lower electrode plate below it, the resulting electrostatic force causes them to attract each other, thereby lowering the color filter element and allowing it to enter the optical path. The two upper electrodes, respectively connected to the first color filter element 20 and the second color filter element 30, combine with the lower electrode plate to control the lowering of the two color filter elements. Figure 1As can be seen, the planar frame capacitor electrostatic driver in this embodiment may further include an insulating spring 43. Specifically, an insulating spring 43 is disposed between the first color filter element 20 and the lower electrode plate 42; simultaneously, an insulating spring 43 is disposed between the second color filter element 30 and the lower electrode plate 42; the insulating spring 43 can control the springing of the two color filter elements away from the optical path. In addition, the planar frame capacitor electrostatic driver may also include a voltage source V (not numbered).

[0050] This application embodiment uses a flat-plate frame capacitor electrostatic driver in the microelectromechanical system driving component 40 as the driving element. It has the advantages of small size, low power consumption, fast response speed and high control precision. It can accurately control the position switching of the color filtering element in each pixel unit, and is compatible with CMOS image sensor technology, which is convenient for integrated manufacturing.

[0051] In some embodiments of this application, such as Figure 2 and Figure 3 As shown, in this embodiment of the application, the flat-plate frame capacitor electrostatic driver includes an upper electrode plate 41 and a lower electrode plate 42. The upper electrode plate 41 is disposed on the first color filter element 20 and the second color filter element 30, and moves together with the color filter elements; the lower electrode plate 42 is disposed on the photosensitive side of the photoelectric conversion element 10, opposite to the upper electrode plate 41, and is fixed.

[0052] When a voltage is applied between the upper electrode 41 and the lower electrode 42, an electrostatic force is generated between them, driving the upper electrode 41 to move relative to the lower electrode 42, thereby moving the color filter element to the first working position. When the voltage is removed, the upper electrode 41 returns to the second working position under the elastic restoring force of the insulating spring 43. When two upper electrodes 41 are provided in the same pixel unit to correspond to two color filter elements respectively, the two upper electrodes 41 can share a lower electrode 42, or they can correspond to different lower electrodes 42. This application embodiment does not limit this.

[0053] In this embodiment of the application, electrostatic force With driving voltage Plate area Electrode spacing and dielectric constant The relationship between them is shown in formula (1).

[0054] (1) in: It is electrostatic force; , where is the dielectric constant of the medium between the plates. The vacuum permittivity, It is the relative permittivity; This represents the area of ​​the two plates facing each other. The voltage applied between the two plates; , where is the distance between the two plates.

[0055] This embodiment of the application uses a flat-plate frame capacitor electrostatic driver in the form of an upper electrode, a lower electrode, and an insulating spring, resulting in a simple structure that is easy to manufacture. The upper electrode moves with the color filter element, while the lower electrode is fixed to the photosensitive side, ensuring the stability and reliability of the electrostatic drive.

[0056] Furthermore, in some embodiments of this application, the MEMS driving component 40 corresponding to the first color filter element 20 and the MEMS driving component 40 corresponding to the second color filter element 30 are respectively connected to different voltage sources (unnumbered) to achieve independent control. When it is necessary to switch the position of the first color filter element 20, a driving voltage is applied through its corresponding voltage source; when it is necessary to switch the position of the second color filter element 30, a driving voltage is applied through its corresponding voltage source.

[0057] It should be noted that in practical applications, the voltage source in multiple pixel units can be shared. That is, the first color filter element 20 in all pixel units under the same color filter array mode can share one voltage source, and the second color filter element 30 in all pixel units can share another voltage source. By uniformly controlling the position switching of the corresponding color filter elements in all pixel units through a common voltage source, the synchronous operation of all pixels can be ensured, while significantly saving the chip area and power consumption occupied by the driving circuit, reducing device costs, and facilitating the integration of high pixel density on image sensor chips with limited area.

[0058] In some embodiments of this application, at least two color filter array modes include a first color filter array mode and a second color filter array mode. The color filter array in the first color filter array mode includes a first group of color channels, and the color filter array in the second color filter array mode includes a second group of color channels. The first group of color channels and the second group of color channels are at least partially different; for example, the two groups of color channels can be completely different, or only some color channels can be the same, as long as at least one color channel is different.

[0059] For example, the first set of color channels can be nine different color channels, such as dark skin tone, light skin tone, sky blue, orange, bluish-purple, medium red, blue, green, and red; the second set of color channels can be another nine different color channels, such as leaf color, blue flower color, bluish-green, purple, yellowish-green, orange-yellow, yellow, magenta, and cyan. It should be noted that the number and types of color channels mentioned above are merely illustrative examples; in actual applications, they can be freely combined according to usage requirements and are not limited by the above examples.

[0060] This application embodiment sets two sets of multispectral modes with different color channels in the same multispectral sensor and dynamically switches them using the MEMS driving component 40, enabling the acquisition of two different multispectral images on a single sensor. Compared to traditional multispectral sensors with fixed CFA arrangement, this application embodiment can obtain twice the color channel information with the same total pixel size, thereby achieving finer spectral resolution and more accurate color reproduction.

[0061] In some embodiments of this application, the first set of color channels and the second set of color channels are used to sense light in different spectral bands; the different spectral bands include at least two of the visible light band, infrared band and ultraviolet band.

[0062] For example, the first set of color channels can be primarily designed to sense light in the visible light band, while the second set of color channels can be primarily designed to sense light in the infrared or ultraviolet band. Through the complementarity of the two sets of color channels in the spectral bands, the multispectral sensor can cover a wider spectral range.

[0063] This application embodiment designs the color channels of two multispectral modes to sense different spectral bands, enabling a single multispectral sensor to simultaneously meet the spectral sensing needs of multiple bands, including visible light, infrared, and ultraviolet. Compared to traditional solutions that require multiple sensors to cover different bands, this application embodiment significantly reduces device size while expanding the spectral sensing coverage.

[0064] In some embodiments of this application, at least two color filter array modes include a Bayer filter array mode and a multispectral filter array mode. The Bayer filter array mode includes red, green, and blue filter elements arranged in a conventional Bayer array, such as RGGB. The multispectral filter array mode includes filter elements for at least four colors, such as 9 color channels, 12 color channels, or 16 color channels.

[0065] For example, in the first color filter array mode, the MEMS driving component 40 controls the first color filter element 20 to be lowered, i.e., placed in the optical path, while the second color filter element 30 is raised by the elastic force of the insulating spring 43, i.e., removed from the optical path, resulting in a conventional Bayer array arrangement RGB image. In the second color filter array mode, the MEMS driving component 40 controls the second color filter element 30 to be lowered, i.e., placed in the optical path, while the first color filter element 20 is raised by the elastic force of the insulating spring 43, i.e., removed from the optical path, resulting in a multispectral image, such as a multispectral image composed of 16 different color channels: dark skin, light skin, sky blue, leaf color, orange, purplish-blue, medium red, purple, blue, white, red, yellow, green, light gray, medium gray, and dark gray. It should be noted that the number and type of color channels mentioned above are only illustrative examples, and in actual applications, they can be freely combined according to usage requirements and are not limited by the above examples.

[0066] This application embodiment achieves dynamic switching between Bayer filter array mode and multispectral filter array mode within the same image sensor. This allows a single sensor to function as a traditional RGB sensor for conventional imaging, while also switching to multispectral mode to acquire rich spectral information. In subsequent image processing, the images acquired in both modes are combined to output an RGB image with pixel-level color restoration correction, exhibiting significantly higher color accuracy than traditional Bayer images. Furthermore, compared to a dual-camera solution that combines a traditional camera and a multispectral camera within the image sensor, this application embodiment avoids the field-of-view alignment problem between multiple sensors, achieving more accurate pixel-level color restoration.

[0067] Based on the same inventive concept, this application also provides a camera module, including the multispectral sensor described in any of the above embodiments.

[0068] Based on the same inventive concept, this application also provides an electronic device, including: a housing; a camera module, wherein the camera module includes the multispectral sensor described in any of the above embodiments.

[0069] Based on the same inventive concept, embodiments of this application also provide an image processing method, which can be executed by the aforementioned electronic device. For example... Figure 3 As shown, the method 300 may include the following steps.

[0070] Step 302: Control the multispectral sensor to expose in the first color filter array mode to acquire the first image; wherein, in the first color filter array mode, the first color filter element is located at the first working position and the second color filter element is located at the second working position.

[0071] For example, in the first color filter array mode, the MEMS driving component 40 controls the first color filter element 20 in all pixel units to be lowered, that is, located in the optical path, and the second color filter element 30 in all pixel units is lifted by the elastic force of the insulating spring 43, that is, left the optical path, so that the multispectral sensor is exposed in the first color filter array mode to obtain the first image.

[0072] Step 304: Control the multispectral sensor to expose in the second color filter array mode to acquire the second image; wherein, in the second color filter array mode, the first color filter element is located in the second working position, and the second color filter element is located in the first working position.

[0073] For example, in the second color filter array mode, the MEMS driving component 40 controls the second color filter element 30 in all pixel units to be lowered, that is, located in the optical path, and the first color filter element 20 in all pixel units is lifted by the elastic force of the insulating spring 43, that is, left the optical path, so that the multispectral sensor is exposed in the second color filter array mode to obtain the second image.

[0074] Step 306: Perform downsampling processing on the first image and the second image to extract the single-channel image of each color channel corresponding to the first color filter array mode and the second color filter array mode respectively.

[0075] Since each pixel can only perceive information from one color channel, by downsampling, the single-channel images of each color channel under the first color filter array mode can be extracted from the first image, and the single-channel images of each color channel under the second color filter array mode can be extracted from the second image.

[0076] In some embodiments of this application, such as Figure 4 As shown, the first color filter array is a 9-channel multispectral filter array 100, and the second color filter array is a 9-channel multispectral filter array 200. The first image is a multispectral mode 1 image obtained under the first color filter array mode, and the second image is a multispectral mode 2 image obtained under the second color filter array mode. Figure 5 As shown, by downsampling, single-channel images of each color channel are extracted from the two images, resulting in a total of 18 single-channel images of different colors, while a traditional fixed CFA sensor of the same size can only obtain 9. For example, in... Figure 5In the example, the first image is downsampled to obtain nine single-channel images of colors: dark skin tone, light skin tone, sky blue, orange, purplish-blue, medium red, blue, green, and red. The second image is downsampled to obtain nine more single-channel images of colors: leaf color, blue flower color, bluish-green, purple, yellowish-green, orange-yellow, yellow, magenta, and cyan. It should be noted that the number and types of color channels mentioned above are for illustrative purposes only; in actual applications, they can be freely combined according to usage requirements and are not limited by the examples above.

[0077] In other embodiments, such as Figure 6 As shown, the first color filtering array is a Bayer filter array 300, and the second color filtering array is a 16-channel multispectral filter array 400. The first image is an RGB image obtained by exposure in Bayer filter array mode, and the second image is a multispectral image obtained by exposure in multispectral filter array mode. Through downsampling processing, single-channel images of the red, green, and blue color channels are extracted from the first image, and single-channel images of each color channel in multispectral mode are extracted from the second image. For example, the second image can extract single-channel images of 16 different color channels, including dark skin, light skin, sky blue, leaf color, orange, purplish-blue, medium red, purple, blue, white, red, yellow, green, light gray, medium gray, and dark gray. The specific number and type of color channels can be freely combined according to actual application needs. Thus, a total of 3 RGB single-channel images and 16 multispectral single-channel images are obtained, totaling 19 single-channel images of different colors, providing rich color information input for subsequent image processing models.

[0078] Step 308: Input the single-channel image of each color channel corresponding to the first color filter array mode and the second color filter array mode into the image processing model, and output the third image.

[0079] This application embodiment acquires images under two different CFA modes through two exposures, and then extracts single-channel images of each color channel through downsampling as input to the image processing model. This allows the image processing model to perform inference based on richer color channel information, thereby outputting a higher-quality third image. Compared to the traditional fixed CFA single-frame image processing method, this application embodiment utilizes dual-mode switching to acquire more spectral channel information, providing a more sufficient data foundation for subsequent image processing.

[0080] In some embodiments of this application, the third image is an RGB image that has been color restored and corrected, or the third image is the spectral information of the shooting scene corresponding to the first image and the second image.

[0081] When the image processing model is trained using the target RGB image as the training target, the third image output by the image processing model is an RGB image that has undergone pixel-level color restoration correction, which can be used to assist traditional cameras such as the main camera to achieve more accurate color restoration.

[0082] When the image processing model is trained using the spectral curves of the same shooting scene collected by a spectrometer as the training target, the third image output by the image processing model is the spectral information of the shooting scene corresponding to the first image and the second image, which can be used for application scenarios such as color recognition, light source detection, food safety analysis, and skin analysis.

[0083] By setting different training objectives, the same image processing method can output different types of image processing results, which can meet different application needs such as color restoration and spectral analysis, thus expanding the applicability of the technical solution of this application.

[0084] In some embodiments of this application, method 300 may further include the following steps: acquiring RGB images of the shooting scene of the first image and the second image through an auxiliary camera; step 308, inputting the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode into the image processing model, includes: inputting the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode and the RGB image together into the image processing model.

[0085] Specifically, while the multispectral sensor performs two exposures in the first and second color filter array modes respectively, an additional conventional camera within the image sensor acquires RGB images of the scene being captured for both the first and second images. In subsequent image processing, the single-channel images of each color channel acquired by the multispectral sensor and the RGB images acquired by the auxiliary camera are input together into the image processing model. The model then combines the information from both to output the processed third image.

[0086] This application embodiment uses an auxiliary camera to simultaneously acquire RGB images of the same scene as supplementary information for the image processing model input. This allows the image processing model to combine the texture information of traditional RGB images with the spectral information of multispectral images, achieving more accurate pixel-level color reproduction. Especially in Figure 4 In the illustrated embodiment, the RGB image from the auxiliary camera can be used as additional feature input to enhance the image processing model's ability to perceive scene color and lighting.

[0087] In some embodiments of this application, the above method may further include a training process for an image processing model. The training process includes the following steps: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode, and RGB images acquired by the auxiliary camera; the training target set is a reference RGB image.

[0088] This scheme corresponds to a training method that uses RGB images from an auxiliary camera as additional input. Specifically, it involves collecting a large amount of real-world image data, using single-channel images of each color channel acquired by a multispectral sensor in two modes and RGB images of the same scene acquired by the auxiliary camera as input, and using a reference RGB image as the training target. This forms training, validation, and test sets for supervised learning training of the image processing model. After obtaining the trained model, in subsequent photography, both the single-channel images of each color channel acquired by the multispectral sensor and the RGB images acquired by the auxiliary camera are used as input to the model, allowing it to output an RGB image with pixel-level color restoration and correction.

[0089] This embodiment introduces an auxiliary camera simultaneously capturing RGB images of the same scene as additional input, enabling the image processing model to obtain dual information from both traditional RGB images and multispectral images during both training and inference phases. On one hand, the RGB images provided by the auxiliary camera contain rich texture information and color distribution features, compensating for the spatial resolution loss due to the low single-channel sampling rate of the multispectral sensor. On the other hand, the single-channel images of multiple color channels provided by the multispectral sensor contain detailed spectral information, reflecting the spectral characteristics of different substances in the scene. By inputting both into the deep learning model, the model can fuse the texture features of the RGB image and the spectral features of the multispectral image, thereby achieving pixel-level accurate color reproduction. Compared to using only multispectral images as model input, the technical solution of this embodiment can output RGB images with more accurate colors and richer details, effectively improving the quality and robustness of color reproduction.

[0090] In some embodiments of this application, the above method may further include a training process for an image processing model. The training process includes the following steps: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode; and the training target set is the spectral curve acquired by the spectrometer.

[0091] Specifically, by collecting a large amount of real-world image data, using single-channel images of each color channel acquired by a multispectral sensor in two different modes as input, and spectral curves of the same scene acquired by a spectrometer as the training target, a training set, a validation set, and a test set are constructed. Supervised learning training is performed using a classic deep learning network architecture. After obtaining the trained model, it is integrated into the algorithm module. In subsequent photography, using single-channel images of each color channel acquired by the multispectral sensor as input, the model can output the spectral information of the current scene.

[0092] This application's embodiments use real spectral curves acquired by a spectrometer as the training target, enabling the trained model to accurately reconstruct the spectral information of a scene based on channel images obtained from a multispectral sensor. This solution requires no auxiliary camera and is suitable for applications that require detailed spectral data rather than RGB images, such as color recognition, light source detection, food safety analysis, skin analysis, and water quality testing.

[0093] The embodiments of this application do not rely on auxiliary cameras. Model training and inference can be completed solely through images acquired by the multispectral sensor itself in different modes. This saves on the hardware costs and device size overhead associated with auxiliary cameras and their related circuits, and is beneficial for realizing multispectral imaging and color restoration functions in devices with limited hardware resources.

[0094] In some embodiments of this application, the above method may further include a training process for an image processing model. The training process includes the following steps: acquiring multiple sets of real-scene image samples; constructing a training input set and a training target set based on the real-scene image samples; inputting the training input set and the training target set into the image processing model for supervised learning training; wherein, the training input set includes RGB images or single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode, and single-channel images of each color channel acquired in the second color filter array mode; the training target set is a reference RGB image.

[0095] This solution is particularly suitable for Figure 6In the illustrated embodiment, the scenario of switching between multispectral mode and Bayer mode is that the Bayer RGB image obtained by the first color filter array mode is downsampled to obtain three single-channel images of R, G, and B, which together with the single-channel images of each color channel obtained by the second color filter array mode constitute the training input.

[0096] This application's embodiments do not rely on an auxiliary camera; model training and inference can be completed solely through images acquired by the multispectral sensor in different modes. This saves on the hardware costs and device size overhead associated with auxiliary cameras and their related circuitry, making it advantageous for implementing multispectral imaging and color reproduction functions in devices with limited hardware resources. It is particularly suitable for... Figure 6 The scenario shown in the multispectral mode and Bayer mode switching example has the same color expression form as the traditional RGB image obtained in the first color filter array mode. After downsampling to extract the three single-channel images of R, G, and B, they are combined with the single-channel images of each color channel of the multispectral mode obtained in the second color filter array mode to form the training input. This allows the model to directly establish the mapping relationship between the Bayer RGB channel and the multispectral channel to the reference RGB image, improving the accuracy of color reproduction without adding extra hardware.

[0097] The multispectral sensor and image processing method provided in this application, by setting multiple color filtering elements in a single pixel unit and using a MEMS planar frame capacitor electrostatic driver to control their movement between a first working position in the optical path and a second working position away from the optical path, achieves dynamic switching of at least two CFA arrangement modes in the same image sensor. Compared with traditional multispectral sensors with fixed CFA modes, the technical solution of this application has at least the following beneficial effects: Flexible mode switching: It can switch between two multispectral modes or between Bayer mode and multispectral mode according to usage needs, making a single sensor adaptable to a variety of application scenarios.

[0098] Saves device size: Achieving the same spectral sensing effect that previously required multiple sensors equipped with different CFAs with a single sensor significantly reduces the overall size of the image sensor and lowers costs.

[0099] Improved spectral resolution: By switching and fusing two sets of multispectral modes with different color channels, more color channel information can be obtained under the same total pixel size, achieving finer spectral resolution and wider spectral coverage.

[0100] Improved sampling efficiency: By acquiring two sets of multispectral images with different color channels through two exposures, and after downsampling and model fusion, it is equivalent to obtaining twice the effective channel sampling information under the same total pixel size, which alleviates the problem of decreased single-channel sampling frequency caused by the increase in the number of channels in traditional multispectral sensors.

[0101] Improved image quality: By combining multispectral channel information obtained through dual-mode switching with deep learning models, it can output RGB images with pixel-level color restoration or accurate spectral information, effectively avoiding image quality problems such as zipper effect, false color, and moiré patterns caused by traditional interpolation algorithms.

[0102] Precise field of view alignment: Compared with the dual-camera combination of a traditional camera and a multispectral camera in the traditional solution, this application achieves multi-mode imaging by switching the same sensor in time, which naturally avoids the field of view alignment problem between multiple sensors and provides a reliable guarantee for pixel-level color reproduction.

[0103] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0104] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0105] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0106] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0107] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0108] The processor 810 is configured to control the multispectral sensor to expose in a first color filter array mode to acquire a first image; wherein, in the first color filter array mode, the first color filter element is located at the first working position and the second color filter element is located at the second working position; control the multispectral sensor to expose in a second color filter array mode to acquire a second image; wherein, in the second color filter array mode, the first color filter element is located at the second working position and the second color filter element is located at the first working position; perform downsampling processing on the first image and the second image to extract single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode respectively; input the single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode respectively into an image processing model to output a third image.

[0109] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0110] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0111] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0112] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0113] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer ROM, RAM, magnetic disk, or optical disk.

[0114] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0115] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0116] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0117] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0119] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A multispectral sensor, characterized in that, include: Multiple pixel units, wherein each pixel unit includes a photoelectric conversion element, a first color filter element, and a second color filter element; A driving component, connected to the first color filter element and the second color filter element, is used to drive the first color filter element or the second color filter element to move to a first working position or a second working position, so that the multispectral sensor switches between at least two color filter array modes. The first working position is located in the imaging optical path of the photoelectric conversion element, and the second working position is located in the blind zone of the imaging optical path.

2. The multispectral sensor according to claim 1, characterized in that, The driving component is a microelectromechanical system (MEMS) driving component, which includes a flat-plate frame capacitor electrostatic driver. The flat-plate frame capacitor electrostatic driver is connected to the first color filter element and the second color filter element, respectively.

3. The multispectral sensor according to claim 2, characterized in that, The flat-plate frame capacitor electrostatic driver includes an upper electrode plate and a lower electrode plate; the upper electrode plate is disposed on the first color filter element and the second color filter element; the lower electrode plate is disposed on the photosensitive side of the photoelectric conversion element and is disposed opposite to the upper electrode plate.

4. The multispectral sensor according to any one of claims 1 to 3, characterized in that, The at least two color filter array modes include a first color filter array mode and a second color filter array mode. The color filter array in the first color filter array mode includes a first group of color channels, and the color filter array in the second color filter array mode includes a second group of color channels. The first group of color channels and the second group of color channels are at least partially different.

5. The multispectral sensor according to claim 4, characterized in that, The first set of color channels and the second set of color channels are used to sense light in different spectral bands; the different spectral bands include at least two of the visible light band, infrared band and ultraviolet band.

6. The multispectral sensor according to any one of claims 1 to 3, characterized in that, The at least two color filter array modes include a Bayer filter array mode and a multispectral filter array mode; the Bayer filter array mode includes red, green and blue filter elements; the multispectral filter array mode includes filter elements of at least four colors.

7. A camera module, characterized in that, The multispectral sensor includes any one of claims 1 to 6.

8. An electronic device, characterized in that, include: case; A camera module comprising the multispectral sensor according to any one of claims 1 to 6.

9. An image processing method, performed by the electronic device of claim 8, characterized in that, include: The multispectral sensor is controlled to expose in a first color filter array mode to acquire a first image; wherein, in the first color filter array mode, the first color filter element is located at the first working position and the second color filter element is located at the second working position. The multispectral sensor is controlled to expose in the second color filter array mode to acquire a second image; wherein, in the second color filter array mode, the first color filter element is located in the second working position, and the second color filter element is located in the first working position; The first image and the second image are downsampled to extract the single-channel image of each color channel corresponding to the first color filter array mode and the second color filter array mode, respectively. The single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode are input into the image processing model, and the third image is output.

10. The method according to claim 9, characterized in that, The third image is an RGB image that has undergone color restoration correction, or the third image is the spectral information of the shooting scene corresponding to the first image and the second image.

11. The method according to claim 9, characterized in that, The method further includes: An auxiliary camera is used to capture RGB images of the shooting scenes of the first and second images; The step of inputting the single-channel image of each color channel corresponding to the first color filter array mode and the second color filter array mode into the image processing model includes: The single-channel images of each color channel corresponding to the first color filter array mode and the second color filter array mode, along with the RGB image, are input into the image processing model.

12. The method according to claim 11, characterized in that, The training process of the image processing model includes: Collect multiple sets of real-scene image samples; Based on the real-world image samples, a training input set and a training target set are constructed; The training input set and the training target set are input into the image processing model for supervised learning training. The training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode, and RGB images acquired by the auxiliary camera; the training target set is a reference RGB image.

13. The method according to claim 9, characterized in that, The training process of the image processing model includes: Collect multiple sets of real-scene image samples; Based on the real-world image samples, a training input set and a training target set are constructed; The training input set and the training target set are input into the image processing model for supervised learning training. The training input set includes single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode and the second color filter array mode; the training target set is the spectral curves acquired by the spectrometer.

14. The method according to claim 9, characterized in that, The training process of the image processing model includes: Collect multiple sets of real-scene image samples; Based on the real-world image samples, a training input set and a training target set are constructed; The training input set and the training target set are input into the image processing model for supervised learning training. The training input set includes RGB images or single-channel images of each color channel acquired by the multispectral sensor in the first color filter array mode, and single-channel images of each color channel acquired in the second color filter array mode; the training target set is a reference RGB image.