Image processing method and apparatus

Abstract

The application discloses an image processing method and device, and belongs to the field of image processing. The method is applied to an electronic device, and the electronic device comprises a first camera and a second camera. The first camera comprises a microlens layer, a filter layer and a photosensitive layer which are arranged in layers, and the filter layer is used for multispectral filtering and polarization filtering. The second camera is an imaging camera. The method comprises the following steps: the electronic device acquires multispectral information and polarization information through the first camera; the electronic device acquires image information through the second camera; and the electronic device processes the image information according to the multispectral information and the polarization information. In the above scheme, the polarization information and the multispectral information are acquired through a camera other than the imaging camera (i.e. the second camera), and the polarization information is effectively acquired and used through the first camera without losing the photosensitivity of the imaging camera and reducing the resolving power of the image, so that the image quality is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method and apparatus. Background Technology

[0002] During the imaging process of a scene, processing the image using the scene's polarization information can improve image sharpness and contrast. For example, electronic devices can acquire polarization information through the polarization structure in an imaging camera and process the image based on this information, thereby improving image sharpness and contrast. However, the polarization structure reduces the overall amount of light entering the imaging camera and causes uneven light reception, resulting in a loss of the camera's sensitivity, reduced image resolution, and a decline in image quality.

[0003] Therefore, how to effectively improve image quality is an urgent problem to be solved. Summary of the Invention

[0004] This application provides an image processing method and apparatus that can effectively improve image quality.

[0005] In a first aspect, an image processing method is provided. This method is applied to an electronic device, which includes a first camera and a second camera. The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer has at least four spectral channels. The second camera is an imaging camera. The method includes: the electronic device acquiring multispectral information and polarization information through the first camera; the electronic device acquiring image information through the second camera; and the electronic device processing the image information based on the multispectral information and the polarization information.

[0006] Based on the above scheme, the first camera can be equipped with a filter layer that has multispectral filtering and polarization filtering functions. In this way, the electronic device can acquire multispectral and polarization information through the first camera, and process the image information acquired by the second camera, thereby improving the contrast, clarity, and color accuracy of the image information. In the above scheme, the polarization and multispectral information are acquired by a camera other than the imaging camera (i.e., the second camera). Therefore, the above scheme effectively acquires and uses polarization information through the first camera without sacrificing the sensitivity of the imaging camera or reducing the image resolution, thus effectively improving image quality.

[0007] In some implementations, the filter layer includes a multispectral layer and a polarization filter layer.

[0008] Based on the above scheme, the filter layer can include a multispectral layer and a polarization filter layer. The multispectral layer provides multispectral information, and the polarization filter layer provides polarization information. Thus, by using these two layers, a filter layer with both multispectral and polarization filtering functions can be easily implemented.

[0009] In some implementations, the filter layer has a micro / nano structure.

[0010] Based on the above scheme, the filter layer can have a micro / nano structure, which can have either multispectral filtering or polarization filtering functions, or both. Therefore, the filter layer can be a single layer, saving space.

[0011] In some implementations, the first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

[0012] Based on the above scheme, multiple first pixels with the same spectral channel are arranged adjacently. This allows the electronic device to simultaneously read multiple first pixels, thereby quickly acquiring multispectral information. The multiple first pixels have different polarization angles. This allows the electronic device to read each pixel of the multiple first pixels one by one, thus simultaneously acquiring multispectral and polarization information. Therefore, the above scheme supports the electronic device in conveniently and flexibly acquiring only multispectral information, or simultaneously acquiring multispectral and polarization information, through two different reading methods.

[0013] In some implementations, the filter layer has at least two polarization channels.

[0014] In some implementations, the second camera is an RGB camera, a RYB camera, or an RGBW camera.

[0015] In some implementations, the electronic device acquires multispectral information and polarization information through the first camera, including: the electronic device performs visible light imaging through the first camera to acquire the multispectral information and polarization information.

[0016] In some implementations, before the electronic device processes the image information based on the multispectral information and the polarization information, the method further includes: the electronic device determining, based on the image information, that the image information corresponds to a preset scene.

[0017] Secondly, an image processing apparatus is provided. This apparatus is applied to an electronic device, which includes a first camera and a second camera. The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer has at least four spectral channels. The second camera is an imaging camera.

[0018] As an example, the image processing apparatus may include modules, units, or means corresponding one-to-one with the methods / operations / steps / actions described in the first aspect and any possible implementation of the first aspect, which may be hardware circuits, software, or a combination of hardware circuits and software implementation.

[0019] For example, the image processing apparatus includes an acquisition unit and a processing unit; the acquisition unit is used to acquire multispectral information and polarization information through the first camera; the acquisition unit is also used to acquire image information through the second camera; the processing unit is used to process the image information based on the multispectral information and the polarization information.

[0020] In some implementations, the filter layer includes a multispectral layer and a polarization filter layer.

[0021] In some implementations, the filter layer has a micro / nano structure.

[0022] In some implementations, the first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

[0023] In some implementations, the filter layer has at least two polarization channels.

[0024] In some implementations, the second camera is an RGB camera, a RYB camera, or an RGBW camera.

[0025] In some implementations, the processing unit is specifically used to perform visible light imaging through the first camera to acquire the multispectral information and the polarization information.

[0026] In some implementations, the processing unit is also used to determine, based on the image information, that the image information corresponds to a preset scene.

[0027] Thirdly, an electronic device is provided, comprising: a first camera, a second camera, and a processor (or processing circuitry). The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer, the filter layer being located between the microlens layer and the photosensitive layer, the filter layer being used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer having at least four spectral channels; the second camera is an imaging camera; and the processor is used to execute the first aspect and any possible method thereof.

[0028] In some implementations, the electronic device also includes memory. In one possible implementation, the processor and memory are integrated together. In another possible implementation, the memory is located outside the electronic device. The processor may include one or more processors.

[0029] In some implementations, the processor is used to communicate with other devices via interface circuitry and to execute the first aspect and any possible method of the first aspect described above.

[0030] Fourthly, a computer-readable storage medium is provided that stores a computer program or instructions that, when executed on a computer, cause the first aspect and any possible method of the first aspect to be performed.

[0031] Fifthly, a computer program product is provided, comprising a computer program or instructions that, when executed on a computer, cause the first aspect and any possible method of the first aspect to be performed (or implemented).

[0032] In a sixth aspect, a chip is provided, including a processor for invoking a computer program or computer instructions in memory to cause the processor to execute any of the implementations in the first aspect described above.

[0033] In some implementations, the processor is coupled to the memory via an interface.

[0034] The description of the beneficial effects of any of the second to sixth aspects can be referred to the description of the beneficial effects of the first aspect. Attached Figure Description

[0035] Figure 1 This is a schematic structural diagram of an electronic device according to an embodiment of this application.

[0036] Figure 2 This is a schematic exploded view of a camera provided in an embodiment of this application.

[0037] Figure 3This is a schematic cross-sectional view of a camera provided in an embodiment of this application.

[0038] Figure 4 This is a schematic diagram of a cross-sectional view of the image sensor provided in an embodiment of this application.

[0039] Figure 5 This is another schematic diagram of a cross-sectional view of the image sensor provided in the embodiments of this application.

[0040] Figure 6 This is a schematic diagram of the smallest repeating unit of the sensor pattern of the image sensor provided in the embodiments of this application.

[0041] Figure 7 This is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0042] Figure 8 This is a schematic flowchart of another image processing method provided in the embodiments of this application.

[0043] Figure 9 This is a schematic flowchart of another image processing method provided in the embodiments of this application.

[0044] Figure 10 This is a schematic block diagram of another electronic device according to an embodiment of this application.

[0045] Figure 11 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application. Detailed Implementation

[0046] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0047] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "at least one" and "one or more" refer to one, two, or more than two. The singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise.

[0048] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0049] In the description of the embodiments of this application, the terms "upper," "lower," "inner," "outer," "vertical," and "horizontal," etc., indicate orientations or positional relationships relative to the indicated placement of components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and not to indicate or imply a specific orientation that the device or component must have, or its construction and operation in a specific orientation. They can change accordingly depending on the orientation of the components in the accompanying drawings, and therefore should not be construed as limiting this application. Furthermore, "vertical" in this application is not strictly vertical, but within the allowable error range. "Parallel" is not strictly parallel, but within the allowable error range.

[0050] In the embodiments of this application, the same reference numerals are used to denote the same component or part. Furthermore, the parts in the drawings are not drawn to scale, and the dimensions and sizes of the parts shown are merely exemplary and should not be construed as limiting the scope of this application.

[0051] This application's embodiments may relate to electronic devices. As an example, the aforementioned electronic devices may be electronic devices with imaging capabilities, such as mobile phones, personal digital assistants (PDAs), tablet computers, laptop computers, cameras, video recorders, smartwatches, smart wristbands, point-of-sale (POS) terminals, in-vehicle systems, televisions (e.g., smart screens), or wearable devices. This application's embodiments do not impose special limitations on the specific form of the electronic device. For ease of explanation and understanding, the following description, in conjunction with... Figure 1 This example uses a mobile phone as an example of an electronic device.

[0052] Figure 1 A schematic structural diagram of an electronic device according to an embodiment of this application is shown.

[0053] For example, Figure 1 Images (a) and (b) schematically show the front and back of the electronic device 100, respectively. Figure 1 As shown, the electronic device 100 may include a housing 101, a display panel (DP) 102, and a camera array 103.

[0054] As an example, housing 101 can form a receiving space for accommodating components of electronic device 100. Housing 101 also serves to protect electronic device 100 and support the entire device. Display screen 102 and camera array 103 can be disposed within the receiving space of housing 101 and connected to housing 101. In some examples, housing 101 may include a back cover opposite to display screen 102 and a mid-frame disposed between the back cover and display screen 102, with display screen 102 and camera array 103 fixed to the mid-frame. Housing 101 can be made of metal, plastic, ceramic, or glass, etc.

[0055] As an example, display screen 102 can be used to display images, such as images captured by camera array 103. Display screen 102 can be a liquid crystal display (LCD) screen, or an organic light emitting diode (OLED) screen, etc. Display screen 102 can be a regular screen, an irregularly shaped screen, or a foldable screen, etc. Display screen 102 can be located on the front and / or back of electronic device 100. Here, the front of electronic device 100 can be understood as the side facing the user when using electronic device 100, and the back of electronic device 100 can be understood as the side facing away from the user when using electronic device 100.

[0056] As an example, camera array 103 can be used to capture still images or video. Camera array 103 includes multiple camera compact modules (CCMs) (or simply cameras). For example, as... Figure 1 As shown, the camera array 103 may include at least one imaging camera 103a and at least one multispectral camera 103b.

[0057] For example, the imaging camera 103a can be a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a depth-sensing camera.

[0058] For example, the multispectral camera 103b may include a lens assembly and an image sensor (or, a multispectral image sensor). Exemplarily, the multispectral sensor may include a filter array with multiple transmittances to obtain more comprehensive spectral information (or multispectral information, or color information) of the subject. As an example, this multispectral information can provide a basis for color reproduction of the subject in an image, and can also be used for spectral detection.

[0059] For example, multispectral image sensors can use multiple filter materials in their filter arrays, each with a different transmittance. Thus, the pixel (or the photosensitive unit or photodiode within that pixel) corresponding to each filter material can only acquire spectral information within the band corresponding to that transmittance. Compared to traditional image sensors, multispectral image sensors can obtain more spectral information (called multispectral information), thereby enabling color reproduction.

[0060] For example, spectral detection can be understood as follows: because different objects have different spectral characteristics (e.g., emission or reflection spectra), multispectral image sensors can determine the spectral information of objects by analyzing and calibrating the spectra, thereby detecting the type or characteristics of the objects. For instance, spectral detection may include freshness detection, or sunscreen detection, and so on.

[0061] Multispectral sensors can have multiple spectral channels (or color channels, or filter channels). For example, spectral channels can be defined based on peak wavelengths and transmittance. As an example, for practical applications, a range of peak wavelengths and transmittances can be defined, with each range corresponding to a spectral channel. This application does not limit the specific values ​​for defining spectral channels. In this way, different spectral channels have different transmittance curves, ensuring that a pixel corresponding to a particular spectral channel (or, the photosensitive unit or photodiode within that pixel) can only acquire (or, in other words, detect) spectral information within a fixed wavelength band.

[0062] In some examples, the number of spectral channels in a multispectral sensor can be greater than 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, or other values. As a specific example, the spectral channels of a multispectral sensor may include at least one of the following: green channel, yellow channel, blue channel, white channel, red channel, cyan channel, magenta channel, gray channel, or violet channel.

[0063] The following describes an exemplary meaning of multispectral information. Multispectral information can be spectral information obtained from a multispectral image sensor. Taking a spectral channel count of 4 as an example, the transmittance of the 4 spectral channels can vary significantly. Therefore, a multispectral image sensor can obtain the spectral characteristics of the subject in four ways. After inverse processing using an algorithm, the multispectral information of the subject can be obtained, thus enabling color restoration.

[0064] When the camera array 103 includes multiple multispectral cameras 103b, the multiple multispectral cameras 103b can be completely identical or different. For example, the multiple multispectral cameras 103b may have different focal lengths, different optical structures, or different optical parameters.

[0065] Similarly, when the camera array 103 includes multiple imaging cameras 103a, the multiple imaging cameras 103a can be completely identical or different. For example, the multiple imaging cameras 103a may have different focal lengths, different optical structures, or different optical parameters.

[0066] In some examples, each of the at least one multispectral camera 103b is capable of independent imaging, and the captured images can be directly presented to the user for viewing.

[0067] In some examples, each of the at least one imaging camera 103a is capable of independent imaging, and the captured images can be directly presented to the user for viewing. One or more of the at least one imaging camera 103a can serve as the main camera. Typically, the main camera is responsible for the primary shooting task, usually having the highest pixel count, providing higher resolution and a more powerful sensor, thereby meeting the user's photography needs in different scenarios.

[0068] In some examples, images acquired by multispectral camera 103b can be used to perform color calibration on images acquired by imaging camera 103a. For instance, an electronic device can use a multispectral image of a scene acquired by multispectral camera 103b (e.g., containing color distribution information of the scene) as a reference to perform color calibration on an image of the same scene acquired by imaging camera 103a. Exemplarily, imaging camera 103a can be a red-green-blue (RGB) camera, a red-yellow-blue (RYB) camera, or a red-green-blue-white (RGBW) camera, etc. Accordingly, images acquired by imaging camera 103a can be RGB images, RYB images, or RGBW images, etc.

[0069] One possible principle for color calibration of images acquired by imaging camera 103a using images acquired by multispectral camera 103b is that the spectral response curve of imaging camera 103a differs somewhat from that of the human eye, and it can perceive a smaller number of spectral bands, resulting in weaker color reproduction and a tendency for certain colors to be off-color in the acquired images. In contrast, the spectral response curve of multispectral camera 103b can be made less different from that of the human eye, and it has a stronger spectral perception capability (i.e., it can perceive more than 3 spectral bands, or even more). Therefore, it has a stronger color reproduction capability, and the colors in the acquired images are closer to the true colors after restoration. Thus, using images acquired by multispectral camera 103b as a benchmark for color calibration of imaging camera 103a can improve the color fidelity, or in other words, the color accuracy, of the output images from imaging camera 103a.

[0070] In some examples, the multispectral camera 103b or imaging camera 103a can be a vertical module or a foldable module (or periscope camera). A vertical camera can be understood as light entering the camera directly hitting the image sensor without bending the light path. A foldable camera can be understood as light entering the camera needing to pass through optical elements such as mirrors, lenses, and prisms before hitting the image sensor, resulting in a folded light path.

[0071] In some examples, the camera array 103 can be positioned on the front and / or back of the electronic device 100. When the camera array 103 is positioned on the front of the electronic device 100, it can also be referred to as a front-facing camera. When the camera array 103 is positioned on the back of the electronic device 100, it can also be referred to as a rear-facing camera. In some examples, when the display 102 can be folded, the camera array 103 can function as either a front-facing or rear-facing camera as the display 102 folds. It is understood that the placement of the camera array 103 can be determined based on actual needs. Figure 1 The installation locations shown are for illustrative purposes only.

[0072] In some examples, the electronic device 100 may also include a protective lens 104 for protecting the cameras in the camera array 103. The protective lens 104 is disposed on the housing 101 and covers the cameras. For example, when the protective lens 104 is used to protect the front-facing camera, the protective lens 104 may cover only the front-facing camera or cover the entire front of the electronic device 100. When the protective lens 104 covers the entire front of the electronic device 100, it can also be used to protect the display screen 102, in which case the protective lens 104 is the cover glass (CG). As another example, when the protective lens 104 is used to protect the rear-facing camera, the protective lens 104 may cover the entire back of the electronic device 100 or may be disposed only at the position corresponding to the rear-facing camera.

[0073] In some examples, the protective lens 104 may be made of glass, sapphire, ceramic, etc., and this application does not impose any special limitations on it. For example, the protective lens 104 may be transparent, allowing light from outside the electronic device 100 to enter the camera through the protective lens 104.

[0074] In some examples, electronic device 100 may also include a circuit board and an image processor. Figure 1 (Not shown in the diagram). Exemplarily, the circuit board and image processor can be located within the receiving space formed by the housing 101, and the image processor can be fixed to and electrically connected to the circuit board. The image processor can be communicatively connected to a camera (e.g., a multispectral camera 103b or an imaging camera 103a) in the camera array 103. The image processor can be used to acquire image data from the camera and process the image data. The communication connection between the camera and the image processor can include data transmission via electrical connections such as wiring, or data transmission via coupling or other means. It is understood that the camera and the image processor can also be communicatively connected through other methods capable of data transmission.

[0075] In some examples, electronic device 100 may also include an analog-to-digital converter (also known as an A / D converter). Figure 1 (Not shown in the image). An analog-to-digital converter (ADC) can be connected between the camera and the image processor. The ADC can be used to convert the signal generated by the camera into a digital image signal and transmit it to the image processor. The image processor then processes the digital image signal and finally displays the image or video on the display screen 102.

[0076] In some examples, electronic device 100 may also include a memory ( Figure 1(Not shown in the image) The memory can be communicatively connected to the image processor. After processing the digital image signal, the image processor can transfer the image to the memory so that the image can be retrieved from the memory and displayed on the display screen 102 when needed later. In some examples, the image processor can also compress the processed digital image signal before storing it in the memory to save memory space.

[0077] For example, the image processor described above may also be called a processor or other names, which are not limited in this application.

[0078] Understandable Figure 1 The structure shown in the diagram does not constitute a specific limitation on the electronic device 100. The electronic device 100 may include more or fewer components than shown in the diagram. For example, the electronic device 100 may also include one or more of the following components: battery, flash, earpiece, buttons, sensors, etc. Alternatively, the electronic device 100 may not include the display screen 102, or the electronic device 100 may have a different component arrangement than shown in the diagram.

[0079] Figure 2 and Figure 3 A schematic diagram of the structure of a camera provided in an embodiment of this application is shown. Figure 2 This is a schematic exploded view of camera 200. Figure 3 This is a schematic cross-sectional view of camera 200. Figure 2 The camera 200 in the middle can be Figure 1 An exemplary structure of a multispectral camera or imaging camera is described below. Figure 2 and Figure 3 A brief introduction to the structure of camera 200.

[0080] For ease of description, the optical axis direction of the camera 200 is defined as the Z direction, and the two directions perpendicular to the optical axis are the X direction and the Y direction, with the X direction perpendicular to the Y direction. In the Z direction, the side facing the object being photographed is the front side, and the side facing away from the object is the rear side. In the X and Y directions, the direction closer to the optical axis is the inner side, and the direction facing away from the optical axis is the outer side. In this embodiment, the optical axis direction is the direction in which the optical system transmits light.

[0081] Here, the definitions of X, Y, Z directions and front, back, inside, and outside also apply to the various figures described below. It should be noted that the above definitions of X, Y, Z directions and front, back, inside, and outside are merely for the convenience of describing the positional, connection, or motion relationships between the components in the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0082] like Figure 2 and Figure 3 As shown, the camera module 200 may include a housing 210, a lens assembly 220, a lens actuator 230, and a light sensing component 240.

[0083] The housing 210 has a receiving space for accommodating the lens assembly 220, lens actuator 230, light sensing component 240, etc. Additionally, the housing 210 also serves a protective and supporting function. This is understandable. Figure 2 and Figure 3 The structure of the housing 210 shown is merely exemplary and should not be construed as limiting the scope of this application. Those skilled in the art can design the shape of the housing 210 according to actual needs.

[0084] The lens assembly 220 (or optical lens) mainly includes a lens group 221 and a lens barrel 222, with the lens group 221 housed within the receiving space formed by the lens barrel 222. The lens assembly 220 is used to image the scene on the object side onto the image plane on the image side. In some embodiments, the lens assembly 220 can also perform certain processing on the received imaging beam, such as aberration correction and chromatic aberration elimination. Here, the imaging beam refers to the beam formed by the light incident on the camera 200.

[0085] Lens group 221 may include at least one lens. The at least one lens may be different or at least partially the same. This application embodiment does not specifically limit the number of lenses included in lens group 221. Those skilled in the art can set the number of lenses according to actual needs, such as 1, 2, 3, 5, 8 or more.

[0086] The focal length of lens group 221 can be fixed, and correspondingly, lens assembly 220 is a prime lens. Alternatively, the focal length of lens group 221 can be adjusted, and correspondingly, lens assembly 220 is a zoom lens. For example, the focal length of lens group 221 can be adjusted by changing the relative positions of the lenses within it.

[0087] The lens barrel 222 has a receiving space, primarily for accommodating the lens assembly 221. In some embodiments, the lens barrel 222 can be a single unit, with the lens assembly 221 housed within this single unit. In other embodiments, the lens barrel 222 may also comprise multiple lens barrel sections, with the lenses of the lens assembly 221 grouped and disposed within these multiple lens barrel sections, wherein each lens barrel section and the lenses housed therein can be referred to as a lens group. Exemplarily, the relative positions between these multiple lens barrel sections can be adjusted, enabling optical zoom by adjusting the relative positions of the lenses.

[0088] Understandable. Figure 2 and Figure 3The structure of the lens barrel 222 and the connection method between the lens group 221 and the lens barrel 222 are merely exemplary and do not limit the embodiments of this application.

[0089] The lens actuator 230 is used to move the lens assembly 220 to achieve autofocus and / or optical image stabilization. In some embodiments, the lens actuator 230 may also be referred to as a lens motor, or simply a motor.

[0090] like Figure 3 As shown, the lens assembly actuator 230 may include a motor (hereinafter referred to as the AF motor) 231 for moving the lens assembly 220 for autofocus (AF) and / or a motor (hereinafter referred to as the OIS motor) 232 for moving the lens assembly 220 for optical image stabilization (OIS). Specifically, the AF motor 231 may be used to move the lens assembly 220 for autofocus in the Z direction, and the OIS motor 232 may be used to move the lens assembly 220 for optical image stabilization in the X and / or Y directions. In some embodiments, the AF motor 231 and the OIS motor 232 may be two independent components, each independently driving the lens assembly 220 for AF and OIS respectively. Alternatively, the AF motor 231 and the OIS motor 232 may be integrated into one unit, with a single motor driving the lens assembly 220 for AF and OIS. Figure 3 The lens assembly actuator 230 is illustrated by way of separate AF motor 231 and OIS motor 232, but it should be understood that the embodiments of this application are not limited thereto.

[0091] In some embodiments, the AF motor 231 or the OIS motor 232 can be used to move the entire lens assembly 220, or it can be used to move a part of the lens assembly 220. For example, if a part of the lens assembly 220 (such as the first lens group) is relatively fixed and another part (such as the second lens group) is movable, the AF motor 231 or the OIS motor 232 can drive the movable part to move, thereby changing the optical path to achieve the desired function.

[0092] In some embodiments, the AF motor 231 or the OIS motor 232 may be a voice coil motor (VCM), a shape memory alloy (SMA) motor, a stepping motor, a piezoelectric motor, etc. It should be understood that the specific structure of the AF motor 231 or the OIS motor 232 may be designed and selected according to the selected driving method, and the embodiments of this application do not limit this.

[0093] The light sensing component 240 is disposed on the rear side of the lens assembly 220 and is mainly used for imaging. Exemplarily, the light sensing component 240 may include a filter 241, an image sensor 242, and a circuit board 243.

[0094] A filter 241 can be disposed between the lens assembly 220 and the image sensor 242. The filter 241 can eliminate unwanted light projected onto the image sensor 242, preventing problems such as ghosting, stray light, and color cast during image formation. For example, the filter 241 can be an infrared cut-off filter (filtering out long-wavelength light other than visible light), a bandpass filter (allowing light of a specific wavelength to pass through while blocking light outside the passband), or a filter for filtering out other wavelengths of light. The filter 241 can have different effective ranges depending on the application scenario.

[0095] Image sensor 242 can be a semiconductor chip used to convert collected external light signals into electrical signals. Specifically, the surface of image sensor 242 contains hundreds of thousands to millions of photodiodes, which generate charges when illuminated, thereby converting the light signals collected by lens assembly 220 into electrical signals. For example, image sensor 242 can be a CCD or a CMOS sensor.

[0096] The image sensor 242 includes multiple photosensitive units, which can convert light signals into electrical charges to form an electronic image corresponding to the scene. Each photosensitive unit corresponds to one pixel; the more pixels, the clearer the image.

[0097] As an example, in the image sensor 242 used in the imaging camera 130a, each photosensitive unit includes three different spectral channels, such as red, green, and blue filter channels, or red, yellow, and blue filter channels, etc. For example, different spectral channels can be formed by placing three different colored filters on the photosensitive unit.

[0098] As an example, in the image sensor 242 (hereinafter referred to as multispectral image sensor for ease of description) used in the multispectral camera 130b, each photosensitive unit includes more spectral channels (the number of spectral channels is more than 3, 4, 5, 6, 7, 8, 9, 10 or other values). For example, different color filter arrays (CFAs) can be formed by chemical dyes, coating interference, or metasurface micro / nano structures. Therefore, the multispectral image sensor can extract spectral information from multiple channels to acquire images. Because the multispectral image sensor has more spectral channels and stronger spectral sensing capabilities, the color fidelity of the perceived multispectral image is higher. The multispectral image sensor can acquire multispectral images through multiple spectral channels and can transmit the acquired multispectral images to the color restoration module for analysis, realizing color imaging or color information extraction. The aforementioned multispectral image can also be referred to as multispectral information or other names.

[0099] For example, a multispectral image sensor may include a spectral modulation region and a photoelectric conversion region. The spectral modulation region forms multiple spectral channels through various materials or optical structures to split the incident light. The photoelectric conversion region converts the split light signal into an electrical signal, and then outputs a digital signal or encoding through analog-to-digital conversion. A color restoration module (e.g., an image signal processor) is electrically connected to the multispectral image sensor. The color restoration module can calculate spectral information or color information based on the light signal and pixel position information input from the multispectral image sensor. Based on the color transformation matrix from signals of multiple spectral channels to color space, the color restoration module can also convert the multispectral signals acquired by the multispectral image sensor into color space information of the image (e.g., RGB information, XYZ tristimulus values, or YUV information). Here, the color transformation matrix from signals of multiple spectral channels to color space is established based on the visual response curve of the human eye (e.g., the spectral tristimulus value curve, including the response curve of the human eye to red, green, and blue light) and the spectral response curve of the multispectral camera (including the spectral response curve of the multispectral camera to different wavelengths). In some embodiments, the color restoration module can also use the spectral information acquired by the multispectral camera to perform color calibration on other cameras (e.g., imaging cameras). For example, based on the transformation relationship between the spectral response curve of the multispectral camera and the spectral response curve of other cameras, the color space information corresponding to the multispectral camera is mapped onto other cameras to correct their colors, making the image colors of other cameras more accurate.

[0100] In some embodiments, camera modules with different focal lengths may have their own image sensors, or camera modules with different focal lengths may share the same image sensor.

[0101] In some embodiments, the light sensing component 240 may include an image signal processing (ISP) module. The ISP module is used to process the signals acquired by the image sensor 242, such as performing linear correction, noise cancellation, automatic white balance, automatic exposure control, color correction, etc., converting the raw data acquired by the image sensor 242 into a format supported by the algorithm. Exemplarily, the image signal processing unit may be integrated with the image sensor 242.

[0102] Circuit board 243 is used to transmit electrical signals, and it can be a flexible printed circuit (FPC) or a printed circuit board (PCB). Image sensor 242 can be electrically connected to circuit board 243 via wires to extract signals.

[0103] In some embodiments, the light sensing component 240 may further include a microelectromechanical system (MEMS) actuator 244, which drives the image sensor 242 to move along the optical axis and / or perpendicular to the optical axis, thereby achieving autofocus and / or optical image stabilization. The MEMS actuator 244 can be driven by electrostatic force, magnetoelectric force, piezoelectric force, thermoelectric force, etc. It should be understood that the specific structure of the MEMS actuator 244 can be designed and selected according to the chosen driving method, and this application does not limit it in this regard.

[0104] It should be understood that Figure 2 and Figure 3 The structure shown in the diagram does not constitute a specific limitation on camera 200. Camera 200 may include more or fewer components than shown. For example, camera 200 may also include connectors and peripheral electronic components, or camera 200 may not include lens actuator 230, which will not be described in detail here.

[0105] When electronic devices image a scene, processing the image using the scene's polarization information can improve image sharpness and contrast. For example, in a scene containing mirrors such as a lake or a mirror, the electronic device can use the scene's polarization information to eliminate the decrease in sharpness caused by reflected light from the mirrors, thus improving image sharpness. As another example, in a scene containing metallic objects in grass, the electronic device can use the scene's polarization information to further enhance the contrast between the metallic objects and the grass, thereby improving image contrast.

[0106] In some related solutions, the camera used for image capture (referred to as the imaging camera) can incorporate a polarization structure. For example, a pixelated polarization array can be incorporated into the complementary metal-oxide-semiconductor (CMOS) image sensor of an electronic device. This allows the electronic device to acquire and utilize polarization information simultaneously with image capture, thereby improving image sharpness and contrast. Alternatively, a polarizing mirror or similar component can be placed in front of the CMOS image sensor or charge-coupled device (CCD) image sensor of an electronic device. This allows the electronic device to obtain polarization information of the subject from various angles using temporal or spatial domain techniques, generating images with high contrast and sharpness.

[0107] In other words, in the aforementioned solutions, electronic devices can acquire polarization information through the polarization structure in the imaging camera. However, the polarization structure reduces the overall amount of light entering the imaging camera and causes uneven light reception. Therefore, these solutions reduce the camera's sensitivity and image resolution, resulting in a decrease in image quality.

[0108] Therefore, how to effectively improve image quality is an urgent problem to be solved.

[0109] In view of this, embodiments of this application provide an image processing method. This method can be applied to an electronic device. The electronic device may include a first camera and a second camera. The first camera is capable of providing multispectral information and polarization information. The second camera may be an imaging camera capable of providing image information. Thus, the electronic device can acquire multispectral and polarization information through the first camera and image information through the second camera. Furthermore, the electronic device can use the multispectral and polarization information to process the image information, thereby improving the image's contrast, sharpness, and color accuracy, that is, effectively improving image quality.

[0110] The following are examples of the first camera.

[0111] As an example, the first camera can be obtained by further designing the aforementioned multispectral camera 103b. For instance, based on the aforementioned multispectral camera 103b, a polarization structure can be added, enabling the multispectral camera 103b to provide polarization information and multispectral information. The meaning of multispectral information can be found in the preceding text. Figure 1 The description of the multispectral camera 103b in the original text will not be repeated here.

[0112] As an example, polarization information may include the degree of polarization, a polarization image, the polarization angle, or other polarization-related information.

[0113] For example, the first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer may be located between the microlens layer and the photosensitive layer. The filter layer can be used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer may have at least four spectral channels. For ease of understanding, the following description... Figure 4 The structure of the first camera will be described in detail.

[0114] Figure 4 This is a schematic cross-sectional view of the image sensor 400 provided in an embodiment of this application.

[0115] For example, the first camera may include an image sensor 400, which may include a microlens layer 410, a filter layer 420, and a photosensitive layer 430 stacked together. The filter layer 420 may be located between the microlens layer 410 and the photosensitive layer 430.

[0116] As an example, the microlens layer 410 can be used to converge light.

[0117] As an example, the filter layer 420 can be used to perform multispectral filtering and polarization filtering on light from the microlens layer 410. Exemplarily, the filter layer 420 has at least four spectral channels, or at least five spectral channels, or at least six spectral channels, or at least seven spectral channels, or at least eight spectral channels, or at least nine spectral channels, or at least ten spectral channels, or at least eleven spectral channels, or at least twelve spectral channels, or at least thirteen spectral channels, or at least fourteen spectral channels, or at least fifteen spectral channels, or at least sixteen spectral channels.

[0118] Taking filter layer 420 with five spectral channels as an example, these five spectral channels can be different from each other; in other words, any two of the five spectral channels can be different. For example, the five spectral channels can include a blue channel, a red channel, a green channel, a yellow channel, and a white channel. Examples of other numbers and specific forms of spectral channels will not be elaborated further.

[0119] The aforementioned spectral channels may also be referred to as color channels, multispectral channels, or other names, and this application does not limit them to any particular name.

[0120] For example, the filter layer 420 may have at least two polarization channels, or at least three polarization channels, or at least four polarization channels. Different polarization channels may correspond to different polarization angles. Taking the filter layer 420 having two polarization channels as an example, the filter layer 420 can obtain polarization information at two different polarization angles based on the two polarization channels.

[0121] As an example, the photosensitive layer 430 can be used to perform photoelectric conversion on light from the filter layer 420. For instance, the photosensitive layer 430 can convert an optical signal into an electrical signal.

[0122] In some examples, the first camera may also include other components, such as... Figure 2 Some or all of the components in the camera 200. For example, the image sensor 400 in the first camera may be located at the position of the image sensor 242 in the camera 200, or at other positions, which is not limited in this application.

[0123] The image sensor 400 may also be called a multispectral image sensor, a multispectral polarization image sensor, or other names. This application does not limit the specific name of the image sensor 400.

[0124] To facilitate understanding, the following will be combined with Figure 5 This section provides examples of some structures of the filter layer 420.

[0125] Figure 5 This is another schematic cross-sectional view of the image sensor 400 provided in this application embodiment. For the sake of brevity, the reference numerals for the microlens layer 410 and the photosensitive layer 430 are shown in the accompanying drawings. Figure 4 , Figure 5 It will not be shown again.

[0126] In some examples, filter layer 420 may include two layers: a multispectral layer 422 and a polarization filter layer 424. For example, Figure 5 (a), (b), and (c) in the examples illustrate some of the cases described above.

[0127] This application does not limit the relative position between the multispectral layer 422 and the polarization filter layer 424. For example, see... Figure 5 In (a) and (b), the multispectral layer 422 can be located closer to the microlens layer 410, and the polarization filter layer 424 can be located farther from the microlens layer 410. See also, for example... Figure 5 In (c), the multispectral layer 422 can be located further away from the microlens layer 410, and the polarization filter layer 424 can be located closer to the microlens layer 410. Alternatively, the positions of the multispectral layer 422 and the polarization filter layer 424 can be interchanged.

[0128] As an example, the multispectral layer 422 can be achieved through dyes, coatings, or micro / nano structures.

[0129] As an example, the polarization filter layer 424 can be implemented using gratings, micro / nano structures, or other polarization sensing structures.

[0130] Image sensor 400 may include one or more pixels. Exemplarily, a pixel may include a microlens in microlens layer 410, a multispectral component and / or polarization filter in filter layer 420, and a photodiode (or, photosensitive unit) in photosensitive layer 430. For example, Figure 5 (a) to (d) in the figure show 4 pixels each.

[0131] For example, a multispectral component can be a component with color filtering function that is implemented in the filter layer 420 by dye, coating or micro / nano structure.

[0132] For example, the polarization filter component can be a component with polarization filtering function that is implemented by a grating or micro / nano structure in the filter layer 420.

[0133] As an example, a pixel in the image sensor 400 may have both a multispectral component and a polarization filter component. As another example, a pixel in the image sensor 400 may have either a multispectral component or a polarization filter component.

[0134] For example, see Figure 5 In (a) and (c), the four pixels in the image sensor 400 can simultaneously have both multispectral and polarization filtering components. See also, for example... Figure 5 In (b), pixels 401 and 404 can simultaneously have both multispectral and polarization filter components; pixel 402 can have a multispectral component but not a polarization filter component; and pixel 403 can have a polarization filter component but not a multispectral component. Thus, the electronic device can read pixels 401 and 404 when both multispectral and polarization information are needed; the electronic device can read pixel 402 when only multispectral information is needed; and the electronic device can read pixel 403 when only polarization information is needed.

[0135] As an example, when a pixel does not have a multispectral component, the pixel may include a planar material in the multispectral layer 422. When a pixel does not have a polarization filter component, the pixel may include a planar material in the polarization filter layer 424.

[0136] In some examples, the filter layer 420 can be a single layer. The filter layer 420 can have a micro / nano structure. The micro / nano structure can have one of the functions of multispectral filtering or polarization filtering, or it can have both multispectral filtering and polarization filtering functions simultaneously.

[0137] For example, such as Figure 5 As shown in (d) above, the shape of the cross-section of the micro / nanostructure can be triangular or rectangular. This application does not limit this; the shape of the cross-section can also be other shapes, such as trapezoidal, elliptical, or other shapes. It is understood that by designing the size, period, shape, or fabrication conditions of the micro / nanostructure, it is possible to make the micro / nanostructure simultaneously possess polarization filtering and multispectral filtering functions.

[0138] It is understandable that, as mentioned earlier, a pixel with color filtering functionality implemented through micro-nano structures in the filter layer 420 can be called a multispectral component; a pixel with polarization filtering functionality implemented through micro-nano structures in the filter layer 420 can be called a polarization filtering component. Combined with... Figure 5 In the scheme shown in (d), when the micro-nano structure of a pixel only has color filtering or polarization filtering functions, the pixel can be understood as having a multispectral component or a polarization filtering component; when the micro-nano structure of a pixel has both color filtering and polarization filtering functions, the pixel can be understood as having both multispectral components and polarization filtering components.

[0139] For example, the arrangement of the spectral channels and polarization angles of the individual pixels in the image sensor 400 of the first camera can be referred to as a "sensor pattern". In some examples, the sensor pattern can be obtained by repeating the smallest repeating unit multiple times on the plane where the filter layer 420 is located. In other examples, the sensor pattern itself can be regarded as the smallest repeating unit of the sensor pattern.

[0140] To facilitate understanding, the following will be combined with Figure 6 This section introduces some examples of the smallest repeating unit of a sensor pattern.

[0141] Figure 6 This is a schematic diagram of the smallest repeating unit of the sensor pattern of the image sensor 400 provided in this application embodiment. Figure 6 In this context, one square corresponds to one pixel. Alternatively, one square can represent a polarization filter and / or multispectral component within a pixel. Figure 6 The shadows in different directions can represent polarization filter components with different polarization angles. Different polarization angles can correspond to different polarization channels. Figure 6A scenario with four polarization channels is shown, but those skilled in the art will understand that the image sensor 400 of this application can have more or fewer polarization channels. Figure 6 The numbers in the table represent spectral channel numbers; the same number indicates the same spectral channel, and different numbers indicate different spectral channels. To clearly represent the numbers, Figure 6 In cases where a square contains both shadow and a number (i.e., the number corresponding to the square simultaneously has both a polarizing filter and a multispectral component), the number is surrounded by white fill. Those skilled in the art will understand that the white fill is merely for clarity and does not imply a change to the actual structure of the polarizing filter.

[0142] As an example, the smallest repeating unit of a sensor pattern may include two, three, four, or more pixels with polarization filters. See, for example, [link to relevant documentation]. Figure 6 In (a) to (c), the smallest repeating unit of the sensor pattern may include four pixels with polarization filters. See also, for example, […]. Figure 6 In (d), the smallest repeating unit of the sensor pattern can include nine pixels with polarization filters. For example, see [reference 1]. Figure 6 In (e), the smallest repeating unit of the sensor pattern may include 16 pixels with polarization filters. For example, see [reference 1]. Figure 6 In (f), the smallest repeating unit of the sensor pattern can include 64 pixels with polarization filters.

[0143] For example, a grating or micro / nano structure can be provided in the polarization filter layer 424 of the image sensor 400 as a polarization filter component, so that only light with a certain polarization angle can pass through the polarization filter component and be detected by the photosensitive layer 430. As an example, the difference in polarization angle between two polarization filter components with different polarization angles in the smallest repeating unit of the sensor pattern is ±45°, ±60°, ±90°, ±135° or other polarization angles. "±" indicates "positive or negative".

[0144] This application does not limit the number of spectral channels included in the smallest repeating unit of the sensor pattern. For example, see Figure 6 In (a), (b), and (d), the smallest repeating unit of the sensor pattern can include five spectral channels. For another example, see... Figure 6 In (e), the smallest repeating unit of the sensor pattern can include 7 spectral channels. For another example, see... Figure 6 In (f), the smallest repeating unit of the sensor pattern can include 8 spectral channels. For another example, see [reference 1]. Figure 6 In (c), the smallest repeating unit of the sensor pattern can include 9 spectral channels.

[0145] Furthermore, this application does not limit the number of pixels with multispectral components in the smallest repeating unit of the sensor pattern. For example, see... Figure 6 In (a), (b), and (d), the smallest repeating unit of the sensor pattern can include five pixels with multispectral components. See also, for example, […]. Figure 6 In (c), the smallest repeating unit of the sensor pattern may include nine pixels with multispectral components. For example, see [link to example]. Figure 6 In (e), the smallest repeating unit of the sensor pattern may include 16 pixels with multispectral components. For example, see [reference 1]. Figure 6 In (f), the smallest repeating unit of the sensor pattern may include 64 pixels with multispectral components.

[0146] A pixel in the smallest repeating unit of a sensor pattern can have one of the following components: a polarization filter or a multispectral component, or both.

[0147] Considering the application scenarios and manufacturing difficulties of multispectral and polarization information, the following section combines... Figure 6 This paper introduces some specific examples of the smallest repeating unit of the sensor pattern provided in the embodiments of this application.

[0148] For example, assume that the smallest repeating unit of the sensor pattern is an N*N square matrix. Here, N is an integer greater than 1, and "*" represents a multiplication sign.

[0149] In some examples, 2, 3, or 4 pixels are selected within the smallest repeating unit of an N*N array for polarization detection (or, to provide polarization information). That is, 2, 3, or 4 pixels in the smallest repeating unit of the N*N array have polarization filters. The remaining pixels can be used for multispectral color detection (or, to provide multispectral information). That is, all pixels in the smallest repeating unit of the N*N array except those with polarization filters have multispectral components. For example, pixels with polarization filters can be adjacent; see [reference needed]. Figure 6 (a) in the example. For instance, pixels with polarizing filters may not be adjacent; see also (see...). Figure 6 (b) in the middle.

[0150] Building upon the example above, each pixel in the smallest repeating unit of N*N is provided with a polarization filter or a multispectral component. Thus, some pixels can have both a polarization filter and a multispectral component simultaneously, while others have only one of either a polarization filter or a multispectral component. For example, see... Figure 6 (c) in the middle.

[0151] In other examples, each pixel of the smallest repeating unit of N*N simultaneously has both polarization filtering and multispectral components. For example, see Figure 6 (e) in. Figure 6 In the case shown in (e), the electronic device can simultaneously obtain multispectral and polarization information by reading the pixels on the image sensor obtained by repeating the smallest repeating unit of the sensor pattern. Furthermore, the electronic device can use the multispectral and polarization information to process image information, or it can extract one of the multispectral and polarization information using an algorithm and use that information (i.e., the multispectral information or the polarization information) to process the image information.

[0152] For example, an N*N minimum repeating unit includes adjacent pixels with the same spectral channel and different polarization angles. Since an image sensor can be obtained by repeating an N*N minimum repeating unit once or multiple times, or it can be the N*N minimum repeating unit itself, the above example can also be described as: a first camera includes multiple first pixels arranged adjacently, the multiple first pixels having the same spectral channel and different polarization angles.

[0153] As an example, see Figure 6 In (f), the aforementioned multiple first pixels can be four adjacent pixels with the same spectral channel, for example, Figure 6 The four pixels in the top left corner of (f) are spectral channels numbered 1 and have different polarization angles.

[0154] based on Figure 6 The image sensor obtained by the smallest repeating unit shown in (f) can be accessed by an electronic device through different readout methods (or circuit readout methods) to obtain either polarization information or multispectral information. For example, by reading a single pixel at a time, the electronic device can simultaneously obtain polarization and multispectral information. Alternatively, by reading four pixels at a time (e.g., four adjacent pixels with the same spectral channel), the electronic device can obtain only multispectral information.

[0155] To facilitate understanding, the following will be combined with Figure 7 This section introduces a flowchart of an image processing method.

[0156] Figure 7 This is a schematic flowchart of an image processing method 700 provided in an embodiment of this application. Method 700 can be applied to an electronic device. The electronic device may include a first camera and a second camera. The first camera can be found in the foregoing description. Figures 1 to 6 Related examples will not be repeated. The second camera can be an imaging camera; see [link to relevant documentation] for details. Figures 1 to 3 Related examples are omitted. The optional operations in method 700 are... Figure 7The middle part is indicated by a dashed line. The following is in conjunction with... Figure 7 This section introduces the various operations of method 700.

[0157] S710: The electronic device acquires multispectral and polarization information through a first camera.

[0158] In some examples, the electronic device can control a first camera to perform visible light imaging, acquiring multispectral and polarization information. That is, the multispectral information may include (or only include) the spectral information of visible light. In other examples, the multispectral information may include the spectral information of other light sources, such as infrared light, and / or ultraviolet light.

[0159] In other examples, S710 described above can be replaced by: the electronic device acquiring either multispectral information or polarization information via the first camera. For example, the electronic device can acquire multispectral information via the first camera to perform spectral detection. Examples of spectral detection are provided above. Figure 1 The description of the multispectral camera 103b will not be repeated here. For example, an electronic device can acquire polarization information through the first camera to perform polarization detection. As an example, the aforementioned polarization detection may include scratch detection, roughness detection, and so on.

[0160] In the S720, electronic devices acquire image information through a second camera.

[0161] For example, electronic devices can control a second camera to capture images and thus obtain image information.

[0162] This application does not limit the execution order of S710 and S720. For example, S710 may be executed before S720, or S720 may be executed before S721, or S710 and S720 may be executed simultaneously.

[0163] The S730 is an electronic device that processes image information based on multispectral and polarization information.

[0164] For example, electronic devices can improve color accuracy by processing image information based on multispectral information. In other words, the color accuracy of the image information processed in S730 can be higher than the color accuracy of the image information before processing in S720.

[0165] For example, electronic devices can improve the contrast and sharpness of image information by processing it based on polarization information. In other words, the contrast and sharpness of the image information processed in S730 can be higher than the contrast and sharpness of the image information before processing in S720.

[0166] In other examples, S730 described above can be replaced by: the electronic device processing image information based on either multispectral information or polarization information. For example, if S710 is replaced by the electronic device acquiring polarization information via a first camera, or if the electronic device extracts polarization information using an algorithm after obtaining multispectral and polarization information through S710, then S730 can be replaced by: the electronic device processing image information based on polarization information.

[0167] For example, if the aforementioned S710 is replaced by the case where the electronic device can acquire multispectral information through the first camera, or if the electronic device extracts multispectral information through an algorithm after obtaining multispectral and polarization information by executing S710, then the aforementioned S730 can be replaced by: the electronic device processes image information based on multispectral information.

[0168] In some possible implementations, method 700 also includes S725 prior to S730.

[0169] S720: The electronic device determines that the image information corresponds to a preset scene based on the image information.

[0170] As an example, the aforementioned preset scene can be a typical scenario where polarization imaging is advantageous. For instance, the preset scene could include a mirror, a water surface, or other scenes with specular reflection. After the electronic device detects the aforementioned scene, it can execute S730, that is, process the image information using polarization information to eliminate the imaging effect of reflected light on the object in front of the mirror (or on the water surface), thereby significantly improving the contrast and sharpness of the object.

[0171] It is understandable that many objects in nature exhibit differences in polarization due to variations in their materials. In other examples, when the processing load on electronic devices is relatively light or when there is a user demand, the electronic device can use a first camera to simultaneously acquire polarization and multispectral information, thereby improving image quality.

[0172] To facilitate understanding, the following will be combined with Figure 8 This section presents a specific example of method 700.

[0173] Figure 8 This is a schematic flowchart of another image processing method provided in the embodiments of this application.

[0174] See Figure 8Electronic devices can simultaneously perceive a subject using a first camera and a second camera. The first camera can provide multispectral and polarization information. For example, the first camera can acquire an image containing both multispectral and polarization information. Using an inverse kinematics algorithm, combined with calibration data, the spectral and polarization information of the subject can be extracted from this image. The second camera can provide image information. The electronic device fuses the multispectral, polarization, and image information using an algorithm to obtain image information with improved contrast, sharpness, and color accuracy, thereby enhancing the imaging effect of previews, videos, or finished products.

[0175] See Figure 8 The electronic device can access the first camera to acquire multispectral and polarization information of the object being photographed. Furthermore, based on the multispectral and polarization information, the electronic device can perform spectral and polarization detection of the substance. Examples of spectral and polarization detection can be found in the relevant content of S710 above, and will not be elaborated further here.

[0176] To facilitate understanding, the following will be combined with Figure 9 Here is another concrete example of method 700.

[0177] Figure 9 This is a schematic flowchart of another image processing method provided in the embodiments of this application.

[0178] In some examples, the first camera can operate in multiple modes; for instance, it can acquire only multispectral or polarization information. As an example, if polarization enhancement is required, both multispectral and polarization information can be acquired and then fused with the image information acquired by the second camera using an algorithm to obtain image information with improved contrast, sharpness, and color accuracy (i.e., image information with improved quality).

[0179] For example, the aforementioned situations requiring polarization enhancement may include: the image information corresponding to a preset scene, or other situations, such as user-defined settings. As an example, the electronic device can perform content detection on the image information acquired by the second camera, and determine that polarization enhancement is required if it identifies that the image information corresponds to a preset scene. Specific examples are described in the preceding S725 section and will not be repeated here.

[0180] As another example, if no polarization enhancement is required, multispectral information can be acquired and then fused with the image information acquired by the second camera based on an algorithm to obtain image information with improved color accuracy (i.e., image information with improved quality).

[0181] The following describes the apparatus embodiments corresponding to the method embodiments of this application. Only a brief description of the apparatus is provided below; for specific implementation steps and details, please refer to the preceding method embodiments.

[0182] To achieve the functions of the methods provided in this application, the communication device may include hardware structures and / or software modules, implementing the aforementioned functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0183] Figure 10 This is a schematic block diagram of another electronic device 1000 according to an embodiment of this application. The electronic device 1000 includes a processor 1010, a first camera 1020, and a second camera 1030. Optionally, the processor 1010, the first camera 1020, and the second camera 1030 can be interconnected via a bus. As an example, Figure 10 The electronic device 1000 may also include a device for... Figure 1 Other components described in the electronic device 1000. For example, the electronic device 1000 may also include a housing 101, a display screen 102, or... Figure 1 Other components besides these.

[0184] Optionally, the electronic device 1000 may further include a memory 1040. The memory 1040 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, erasable programmable read-only memory (EPROM), synchronous dynamic random access memory (SDRAM), hard disk drive (HDD), solid-state drive (SSD), or compact disc read-only memory (CD-ROM). The memory 1040 is used to store related instructions and / or data. The memory 1040 may be integrated with the processor 1010 or disposed separately.

[0185] Processor 1010 may be one or more central processing units (CPUs). When processor 1010 is a CPU, it may be a single-core CPU or a multi-core CPU. However, this application is not limited in this respect; processor 1010 may also be one or more graphics processing units (GPUs). Processor 1010 may be an image processor, a chip, or other integrated circuit capable of implementing the methods of this application, or a portion of the circuitry within the aforementioned processor, chip, or integrated circuit for processing functions.

[0186] For example, the processor 1010 is configured to perform the following operations: acquire multispectral information and polarization information through the first camera; acquire image information through the second camera; and process the image information based on the multispectral information and the polarization information.

[0187] The above description is for illustrative purposes only. The electronic device 1000 can be responsible for executing the methods described in the foregoing embodiments. Figures 7 to 9 The relevant methods or steps.

[0188] For details on other implementation methods, please refer to the foregoing. Figures 7 to 9 The detailed description of the embodiments shown will not be repeated here. It should be understood that the specific processes by which each component performs the corresponding processes described above have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0189] Figure 11 This is a schematic block diagram of an image processing apparatus 1100 according to an embodiment of this application. The image processing apparatus 1100 can be an electronic device 1000, or a chip or module of the electronic device 1000, used to implement... Figures 7 to 9 The methods involved in the embodiments shown are described in detail in the above-described method embodiments.

[0190] The image processing apparatus 1100 includes an acquisition unit 1110 and a processing unit 1120.

[0191] For example, the acquisition unit 1110 is used to acquire multispectral information and polarization information through the first camera; the acquisition unit 1110 is also used to acquire image information through the second camera; the processing unit 1120 is used to process the image information according to the multispectral information and the polarization information.

[0192] In some possible implementations, the processing unit 1120 is specifically used to perform visible light imaging through the first camera to acquire the multispectral information and the polarization information.

[0193] In some possible implementations, the processing unit 1120 is further configured to determine, based on the image information, that the image information corresponds to a preset scene.

[0194] The above description is for illustrative purposes only. The image processing device 1100 will be responsible for executing the relevant methods or steps in the foregoing method embodiments. It should be understood that the specific processes by which each component performs the corresponding processes have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0195] This application also provides a chip, including a processor, for calling and executing instructions stored in a memory, causing an electronic device on which the chip is installed to perform the methods in the examples above.

[0196] This application also provides another chip, including: an input interface, an output interface, and a processor. The input interface, output interface, and processor are connected via an internal connection path. The processor is used to execute code in memory. When the code is executed, the processor is used to perform the methods in the examples described above. Optionally, the chip also includes a memory for storing computer programs or code.

[0197] This application also provides a processor for coupling with a memory for performing the methods and functions of an image processing apparatus or electronic device in any of the above embodiments.

[0198] In another embodiment of this application, a computer program product comprising a computer program or instructions is provided, wherein the method of the foregoing embodiments is implemented when the computer program product is run on a computer.

[0199] This application also provides a computer program product that, when run on a computer, enables the implementation of the methods described in the foregoing embodiments.

[0200] In another embodiment of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a computer, implements the methods described in the foregoing embodiments.

[0201] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0206] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0207] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, which includes a first camera and a second camera; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The method includes: The electronic device acquires multispectral and polarization information through the first camera; The electronic device acquires image information through the second camera; The electronic device processes the image information based on the multispectral information and the polarization information.

2. The method according to claim 1, characterized in that, The filter layer includes a multispectral layer and a polarization filter layer.

3. The method according to claim 1 or 2, characterized in that, The filter layer has a micro-nano structure.

4. The method according to any one of claims 1 to 3, characterized in that, The first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

5. The method according to any one of claims 1 to 4, characterized in that, The filter layer has at least two polarization channels.

6. The method according to any one of claims 1 to 5, characterized in that, The second camera is: a red-green-blue RGB camera, a red-yellow-blue RYB camera, or a red-green-blue-white RGBW camera.

7. The method according to any one of claims 1 to 6, characterized in that, The electronic device acquires multispectral and polarization information through the first camera, including: The electronic device performs visible light imaging through the first camera to acquire the multispectral information and the polarization information.

8. The method according to any one of claims 1 to 7, characterized in that, Before the electronic device processes the image information based on the multispectral information and the polarization information, the method further includes: The electronic device determines that the image information corresponds to a preset scene based on the image information.

9. An image processing apparatus, characterized in that, The device is applied to an electronic device, which includes a first camera and a second camera; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The device includes an acquisition unit and a processing unit; The acquisition unit is used to acquire multispectral information and polarization information through the first camera; The acquisition unit is further configured to acquire image information through the second camera; The processing unit is used to process the image information based on the multispectral information and the polarization information.

10. The apparatus according to claim 9, characterized in that, The filter layer includes a multispectral layer and a polarization filter layer.

11. The apparatus according to claim 9 or 10, characterized in that, The filter layer has a micro-nano structure.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The filter layer has at least two polarization channels.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The second camera is: a red-green-blue RGB camera, a red-yellow-blue RYB camera, or a red-green-blue-white RGBW camera.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The processing unit is specifically used to perform visible light imaging through the first camera to obtain the multispectral information and the polarization information.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The processing unit is further configured to determine, based on the image information, that the image information corresponds to a preset scene.

17. An electronic device, characterized in that, include: First camera, second camera, and processor; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The processor is configured to execute the method of any one of claims 1 to 8 by executing a computer program or instructions.

18. The electronic device according to claim 17, characterized in that, The electronic device further includes a memory for storing the computer program or the instructions.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, and when the computer program or instructions are run, the method of any one of claims 1 to 8 is performed.

20. A computer program product, characterized in that, It includes a computer program or instructions that, when run, implement the method as described in any one of claims 1 to 8.

Images