Image processing method, camera and electronic equipment

Abstract

The invention provides an image processing method, a camera and electronic equipment. The camera comprises a hybrid image sensor and an optical lens, the hybrid image sensor is located on the image side of the optical lens to receive light emitted from the optical lens, the hybrid image sensor comprises first pixels and second pixels, any first pixel can independently obtain light intensity changes and output event signals, and any second pixel can independently obtain light intensity changes and output event signals. The number of color channels of the second pixel is greater than or equal to 7. According to the camera provided by the invention, through the design of the hybrid image sensor, the functions of an event camera and a multispectral camera are integrated into the camera, lens holes of the electronic equipment are reduced, and the overall size design of the electronic equipment can be prevented from being interfered.

Description

Technical Field

[0001] This application relates to the field of terminal technology, specifically to an image processing method, a camera, and an electronic device. Background Technology

[0002] With technological advancements, electronic devices with camera capabilities are increasingly sophisticated, often employing auxiliary cameras to supplement the main camera's three-channel imaging capabilities. For example, multispectral and event cameras can be added to assist the main camera. However, electronic devices typically employ multiple three-channel cameras, including a main camera, a wide-angle camera, and a telephoto camera. Therefore, the design of these auxiliary cameras increases the lens openings on the electronic device. This results in an excessive number of lens openings when multispectral and event cameras are added simultaneously, negatively impacting the device's appearance. Furthermore, the increased number of cameras also affects the size design of each camera module, interfering with the overall size design of the electronic device. Summary of the Invention

[0003] This application provides an image processing method, a camera, and an electronic device. The camera includes a hybrid image sensor and an optical lens. The hybrid image sensor includes event pixels and multispectral pixels. By designing a hybrid image sensor, the functions of an event camera and a multispectral camera are integrated into the hybrid camera, reducing the lens opening of the electronic device, reducing the space required to install the camera module in the electronic device, and avoiding interference with the overall size design of the electronic device.

[0004] It should be noted that in the following implementation or embodiments, the first camera is a hybrid camera, the second camera is a three-channel camera, the first pixel is an event pixel, the event pixel has the function of acquiring event data and outputting event signals by a single pixel, and the second pixel is a multispectral pixel.

[0005] In a first aspect, this application provides a camera, which includes a hybrid image sensor and an optical lens. The hybrid image sensor is located on the image side of the optical lens to receive light incident from the optical lens. The hybrid image sensor includes a first pixel and a second pixel. Each first pixel can independently acquire light intensity changes and output an event signal. The second pixel has a color channel number greater than or equal to 7.

[0006] In this application, a hybrid image sensor design is employed to combine the functions of an event camera and a multispectral camera. Since the hybrid camera requires only one lens assembly, AF and / or OIS motor assembly, it not only reduces the number of openings on the electronic device's casing, improving its appearance, but also reduces the size of the camera module and the area of ​​the corresponding readout circuitry, thus improving the overall size of the camera configuration and minimizing interference with the overall design of the electronic device. The multispectral pixels have at least 7 color channels, enabling the hybrid camera to achieve excellent color acquisition performance.

[0007] In some possible implementations, the first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the total area occupied by the second pixel.

[0008] In this implementation, in the first arrangement, the total area occupied by the event pixels is smaller than the total area occupied by the multispectral pixels, so that the event pixels can achieve auxiliary functions with a small number of pixel occupancy, leaving enough pixel occupancy for the multispectral pixels to have high resolution, so that the multispectral pixels can achieve imaging assistance.

[0009] In some other possible implementations, the first pixel and the second pixel have a second arrangement, wherein the total area occupied by the first pixel is greater than the total area occupied by the second pixel.

[0010] In this implementation, the ratio of the total area occupied by event pixels to the total area occupied by multispectral pixels is larger in the second arrangement compared to the first arrangement, enabling the event pixels to perform more functions. In the second arrangement, the total area occupied by event pixels can be greater than the total area occupied by multispectral pixels, allowing the hybrid image sensor to primarily utilize the functions of event pixels, with the functions of multispectral pixels playing a secondary role.

[0011] In some possible implementations, in the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the first pixel, where M is a positive integer less than or equal to 5. For example, M can be, but is not limited to, 1, 2, 3, 4, or 5.

[0012] In this implementation, the pixel placement design described above ensures that the multispectral pixels have sufficient resolution to achieve the effect of prioritizing multispectral functionality while using event functionality as a secondary function. M should not be too large to avoid the total area occupied by the event pixels being too small, which would affect the working effect of the event pixels.

[0013] In some other possible implementations, in the second arrangement, the total area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5. For example, N can be, but is not limited to, 1, 2, 3, 4, or 5.

[0014] In this implementation, the pixel placement design described above ensures that the event pixels have sufficient resolution to achieve the effect of prioritizing event functionality while using multispectral functionality as a secondary function. N should not be too large to avoid the total area occupied by the event pixels becoming too small, thus affecting their performance.

[0015] Among some possible implementations, the camera has a first working mode, a second working mode, and a third working mode.

[0016] In the first working mode, the first pixel operates independently, while the hybrid camera is used to collect event data and output an event data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for gesture detection and eye tracking, outputting the detected gesture or eye position and providing it to the corresponding algorithm. The algorithm then calculates and activates the function of the electronic device corresponding to the gesture, or calculates the eye position and activates the function icon or application of the electronic device at the eye position.

[0017] In the second working mode, the second pixel operates independently, and the hybrid camera is used to acquire multispectral data and output a multispectral data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for multispectral detection to capture a multispectral image of an object, which can then be output, for instance, stored in a gallery.

[0018] In the third working mode, both the first and second pixels are active. The hybrid camera collects event data and multispectral data, outputting event data streams and multispectral data streams respectively. The event data stream uses different algorithms depending on the application scenario, and the multispectral data stream uses different algorithms for the same scenario. Both calculation results are then output. For example, when dynamically acquiring color, event pixels are used to identify the dynamic object, while multispectral pixels are used to identify the color. Event pixels can help make the color image acquired by multispectral pixels clearer. As another example, when the hybrid camera uses multispectral pixels for spectral detection, event pixels can be used for gesture detection to activate different function options or enable shooting.

[0019] It should be noted that in the first aspect, the camera is a hybrid camera.

[0020] Secondly, this application provides an electronic device that includes a camera in any of the implementations of the first aspect.

[0021] In this application, the design of a hybrid image sensor enables the integration of the functions of an event camera and a multispectral camera. Furthermore, since the hybrid camera only requires one set of lens assembly, AF and / or OIS motor assembly, it not only reduces the number of openings on the electronic device housing and improves the appearance of the electronic device, but also reduces the size of a camera module and the area of ​​the corresponding readout circuit, thereby improving the overall size of the camera configuration of the electronic device and reducing interference with the overall size design of the electronic device.

[0022] In some possible implementations, the electronic device also includes a second camera, which is a three-channel camera.

[0023] In this implementation, the function of an event camera and a multispectral camera is hybridized through the design of a hybrid image sensor. This allows the hybrid image sensor to assist the three-channel camera in both dynamic shooting and color reproduction, thus solving the aforementioned shooting problems of the three-channel camera.

[0024] The first camera can be positioned adjacent to the second camera. It should be noted that "adjacent" here means that there are no other cameras, flashes, or auxiliary sensors or other imaging devices between the first and second cameras.

[0025] In this implementation, placing the hybrid camera next to the three-channel camera shortens the communication distance between them, thereby improving the communication efficiency of the hybrid camera assisting the three-channel camera in shooting. Furthermore, by reducing the distance between the hybrid camera and the three-channel camera, when the hybrid camera uses spectral detection for assisted shooting, the captured image from the hybrid camera tends to be consistent with that captured by the three-channel camera, improving the accuracy of spectral-assisted shooting.

[0026] In some possible implementations, the light-incident ends of the first camera and the second camera are located on different surfaces of the electronic device.

[0027] In this implementation, a hybrid camera is used for detection to activate functions related to the three-channel camera, thereby assisting the three-channel camera in its operation.

[0028] For example, a hybrid camera can function as a front-facing camera, while a three-channel camera can function as a rear-facing camera. In some examples, the hybrid camera can activate the camera app via gesture detection or eye tracking to start the three-channel camera. In other examples, when the three-channel camera is on, shooting can be activated via gesture detection or eye tracking.

[0029] For example, the electronic device can be a foldable electronic device, and the hybrid camera and the three-channel camera can be located in different foldable parts so that when the electronic device is unfolded, the hybrid camera and the three-channel camera are located on the same side of the electronic device, thereby enabling the hybrid camera to assist the three-channel camera in taking pictures.

[0030] In some possible implementations, the light-incident ends of the first and second cameras are located on the same side of the electronic device, so that the hybrid camera can assist the three-channel camera in taking pictures.

[0031] In some possible implementations, there are multiple second cameras, and each of the multiple second cameras is set adjacent to the first camera.

[0032] In this implementation, by placing the hybrid camera adjacent to each of the three-channel cameras, the hybrid camera can provide shooting assistance to each of the three-channel cameras, which helps to balance the efficiency of shooting assistance to each three-channel camera. Furthermore, when providing spectral assistance, the images captured by the hybrid camera tend to be consistent with those captured by each of the three-channel cameras, thus balancing the spectral assistance effect of the hybrid camera on each of the three-channel cameras.

[0033] In some possible implementations, the distance between each of the multiple second cameras and the first camera is the same.

[0034] In this implementation, by setting equal intervals, the efficiency of the hybrid camera in assisting each three-channel camera in shooting, as well as the spectral assistance shooting effect of each three-channel camera, can be further balanced.

[0035] In some possible implementations, the first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the total area occupied by the second pixel.

[0036] In this implementation, in the first arrangement, the total area occupied by the event pixels is smaller than the total area occupied by the multispectral pixels, so that the event pixels can achieve auxiliary functions with a small number of pixel occupancy, leaving enough pixel occupancy for the multispectral pixels to have high resolution, so that the multispectral pixels can achieve imaging assistance.

[0037] In some other possible implementations, the first pixel and the second pixel have a second arrangement, wherein the total area occupied by the first pixel is greater than the total area occupied by the second pixel.

[0038] In this implementation, in the second arrangement, the total area occupied by the event pixels can be greater than the total area occupied by the multispectral pixels, so that the hybrid image sensor primarily uses the function of the event pixels and secondarily uses the function of the multispectral pixels.

[0039] In some possible implementations, in the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the first pixel, where M is a positive integer less than or equal to 5. For example, M can be, but is not limited to, 1, 2, 3, 4, or 5.

[0040] In this implementation, the pixel placement design described above ensures that the multispectral pixels have sufficient resolution to achieve the effect of prioritizing multispectral functionality while using event functionality as a secondary function. M should not be too large to avoid the total area occupied by the event pixels being too small, which would affect the working effect of the event pixels.

[0041] In some other possible implementations, in the second arrangement, the total area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5. For example, N can be, but is not limited to, 1, 2, 3, 4, or 5.

[0042] In this implementation, the pixel placement design described above ensures that the event pixels have sufficient resolution to achieve the effect of prioritizing event functionality while using multispectral functionality as a secondary function. N should not be too large to avoid the total area occupied by the event pixels becoming too small, thus affecting their performance.

[0043] In some possible implementations, the first camera has a first working mode, a second working mode, and a third working mode.

[0044] In the first working mode, the first pixel operates independently, while the hybrid camera is used to collect event data and output an event data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for gesture detection and eye tracking, outputting the detected gesture or eye position and providing it to the corresponding algorithm. The algorithm then calculates and activates the function of the electronic device corresponding to the gesture, or calculates the eye position and activates the function icon or application of the electronic device at the eye position.

[0045] In the second working mode, the second pixel operates independently, and the hybrid camera is used to acquire multispectral data and output a multispectral data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for multispectral detection to capture a multispectral image of an object, which can then be output, for instance, stored in a gallery.

[0046] In the third working mode, both the first and second pixels are active. The hybrid camera collects event data and multispectral data, outputting event data streams and multispectral data streams respectively. The event data stream uses different algorithms depending on the application scenario, and the multispectral data stream uses different algorithms for the same scenario. Both calculation results are then output. For example, when dynamically acquiring color, event pixels are used to identify the dynamic object, while multispectral pixels are used to identify the color. Event pixels can help make the color image acquired by multispectral pixels clearer. As another example, when the hybrid camera uses multispectral pixels for spectral detection, event pixels can be used for gesture detection to activate different function options or enable shooting.

[0047] In some possible implementations, a first camera is used to acquire first event data, a second camera is used to acquire first RGB data, and an electronic device is used to generate a first image based on the first event data and the first RGB data; and / or, a first camera is used to acquire first multispectral data, a second camera is used to acquire second RGB data, and an electronic device is used to generate a second image based on the first multispectral data and the second RGB data.

[0048] It should be noted that the first event data is the event data collected by the event pixels, and the first multispectral data is the multispectral data collected by the multispectral pixels.

[0049] The hybrid camera collects event data through event pixels to form an event data stream. After processing by the ESP (Electronic Stability Program), different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the ISP (Image Signal Processor) for image preprocessing. The ISP combines the RGB data stream and the relevant results to perform image processing and generate an image. The event pixels provide functional assistance to the three-channel camera, and these scenarios can include: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and assisted OIS (Optical Image Interpretation).

[0050] The hybrid camera acquires event data through event pixels to form an event data stream. This stream is processed by the ESP (Electronic Stability Program) and then subjected to different algorithms depending on the scene to obtain the event image. The RGB data stream undergoes image preprocessing by the ISP (Image Signal Processor) to form the original RGB image. The ISP then uses an image fusion algorithm to fuse the event image and the original RGB image to generate the final image. The event pixels provide imaging assistance to the three-channel camera, and their applications can include: deblurring, video frame interpolation / slow motion, assisted EIS (Electronic Image Sensor), assisted OIS (Optical Image Sensor), depth detection, portrait blurring, and Livephoto.

[0051] The hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After processing by the M-ISP, different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the ISP's image preprocessing algorithm. The ISP combines the RGB data stream and the relevant results to perform image preprocessing and generate an image. The multispectral pixels provide functional assistance to the three-channel camera, and their applications can include spectral inversion and high-dimensional white point estimation.

[0052] The hybrid camera uses multispectral pixels to acquire multispectral data, forming a multispectral data stream. This stream is then processed by an M-ISP (Multi-Spectral Image Processor), which applies different algorithms based on the scene to obtain relevant results. The RGB data stream undergoes pre-processing by the ISP to obtain the original RGB image. The ISP then uses post-processing algorithms to apply the relevant results to the original RGB image to generate the final image. The multispectral pixels provide functional assistance to the three-channel camera, and its applications include: ambient light spectrum estimation, true color estimation, and infrared and ultraviolet applications.

[0053] The hybrid camera uses multispectral pixels to acquire multispectral data, forming a multispectral data stream. This stream is then processed by an M-ISP (Multi-Spectral Image Processor), which applies different algorithms depending on the scene to obtain a multispectral image. The RGB data stream undergoes preprocessing by the ISP to produce the original RGB image. Finally, the ISP uses an image fusion algorithm to fuse the multispectral and original RGB images to generate the final image. The multispectral pixels provide imaging assistance to the three-channel camera, and its applications include: object spectral decomposition, color layering, tonal gradation, beautification and makeup effects, and creating interactive avatars.

[0054] In some possible implementations, event pixels and multispectral pixels are arranged in a first configuration, the first event data is collected in a first scene, and the first multispectral data is collected in a second scene. The first scene includes any of the following: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and assisted OIS. The second scene includes any of the following: spectral inversion, high-dimensional white point estimation, ambient light source spectral estimation, True color estimation, infrared and ultraviolet applications, object spectral decomposition, color layering, tonal style, beautification and makeup, and shooting and creating interactive avatars.

[0055] In this implementation, the main purpose of the hybrid image sensor is to meet the needs of multispectral devices. Event pixels occupy a small amount of pixel area and mainly perform some auxiliary functions.

[0056] It should be noted that the first scenario is the scenario where event pixels are used to achieve functional assistance, and the second scenario includes the scenario where multispectral pixels can be used to achieve imaging assistance. Since multispectral pixels have higher resolution and can be compatible with functional assistance, the second scenario also includes the scenario where multispectral pixels can be used to achieve functional assistance.

[0057] In some possible implementations, event pixels and multispectral pixels are arranged in a second configuration. The first event data is collected in a third scenario, and the first multispectral data is collected in a fourth scenario. The third scenario includes any of the following: stroboscopic frequency and position detection, stroboscopic mask segmentation, motion region detection / optical flow, moving target detection, deblur, video frame interpolation / slow motion, assisted EIS, assisted OIS, detection depth, portrait blurring, and Livephoto. The fourth scenario includes any of the following: spectral inverse, high-dimensional white point estimation, ambient light source spectral estimation, True color estimation, and infrared and ultraviolet applications.

[0058] In this implementation, since the total area occupied by the event pixels is larger, the event pixels can be used to improve the image quality of the three-channel camera. Since the total area occupied by the multispectral pixels is smaller, they can be used as auxiliary devices to achieve some auxiliary functions.

[0059] It should be noted that the third scenario includes scenarios where event pixels are used for imaging assistance. Because event pixels have higher resolution and are compatible with functional assistance, the third scenario also includes scenarios where event pixels are used for functional assistance. The fourth scenario is scenarios where multispectral pixels are used for functional assistance.

[0060] In some possible implementations, there are multiple event pixels. The electronic device is used to merge at least some of these event pixels into a single pixel unit, so that the pixel unit as a whole can capture light intensity changes and output an event signal. Here, merging multiple event pixels, i.e., forming a larger pixel through pixel binning, allows the binned pixel to work together to capture light intensity changes and output an event signal. It should be noted that the pixel unit formed by pixel binning is still an event pixel, only its size has changed. The event pixels capable of pixel binning are arranged adjacently, meaning that there are no other event pixels between them.

[0061] In this implementation, adjacent pixels in the event pixel are binning into larger pixels, thereby achieving a larger photosensitive area, which can collect more light and thus improve the signal-to-noise ratio and sensitivity of the acquired event data.

[0062] In some possible implementations, the second camera includes any one or more of the main camera, wide-angle camera, and telephoto camera of the electronic device.

[0063] In this implementation, the hybrid camera can assist one or more three-channel cameras in shooting, thereby improving the imaging effect of the three-channel cameras.

[0064] Thirdly, this application also provides an image processing method applied to an electronic device. The electronic device includes a first camera, which includes a hybrid image sensor and an optical lens. The hybrid image sensor is located on the image side of the optical lens and includes a first pixel and a second pixel. Each first pixel can independently acquire and output light intensity information, and the second pixel has a color channel number greater than or equal to 7. The method includes: the electronic device acquiring first data through the first camera, the first data including event data acquired by the first pixel and / or multispectral data acquired by the second pixel; and the electronic device outputting a first data stream through the first camera, the first data stream including an event data stream and / or a multispectral data stream.

[0065] In this application, a hybrid image sensor design is employed to combine the functions of an event camera and a multispectral camera. Since the hybrid camera requires only one lens assembly, AF and / or OIS motor assembly, it not only reduces the number of openings on the electronic device's casing, improving its appearance, but also reduces the size of the camera module and the area of ​​the corresponding readout circuitry, thus improving the overall size of the camera configuration and minimizing interference with the overall design of the electronic device. The multispectral pixels have at least 7 color channels, enabling the hybrid camera to achieve excellent color acquisition performance.

[0066] In some possible implementations, the electronic device also includes a second camera, which is a three-channel camera; the method also includes: the electronic device acquiring second data through the second camera and outputting a second data stream; the electronic device generating an image based on the first data stream and the second data stream.

[0067] In this implementation, a hybrid camera provides shooting assistance to a three-channel camera to improve its shooting performance. By designing a hybrid image sensor, the functions of an event camera and a multispectral camera are combined, allowing the hybrid image sensor to assist the three-channel camera in both dynamic shooting and color reproduction, thus solving the aforementioned shooting problems associated with the three-channel camera.

[0068] In some possible implementations, the light-incident ends of the first camera and the second camera are located on different surfaces of the electronic device.

[0069] In this implementation, a hybrid camera is used for detection to activate functions related to the three-channel camera, thereby assisting the three-channel camera in its operation.

[0070] For example, a hybrid camera can function as a front-facing camera, while a three-channel camera can function as a rear-facing camera. In some examples, the hybrid camera can activate the camera app via gesture detection or eye tracking to start the three-channel camera. In other examples, when the three-channel camera is on, shooting can be activated via gesture detection or eye tracking.

[0071] For example, the electronic device can be a foldable electronic device, and the hybrid camera and the three-channel camera can be located in different foldable parts so that when the electronic device is unfolded, the hybrid camera and the three-channel camera are located on the same side of the electronic device, thereby enabling the hybrid camera to assist the three-channel camera in taking pictures.

[0072] In some possible implementations, the light-incident ends of the first and second cameras are located on the same side of the electronic device, so that the hybrid camera can assist the three-channel camera in taking pictures.

[0073] In some possible implementations, there are multiple second cameras, and each of the multiple second cameras is set adjacent to the first camera.

[0074] In this implementation, by placing the hybrid camera adjacent to each of the three-channel cameras, the hybrid camera can provide shooting assistance to each of the three-channel cameras, which helps to balance the efficiency of shooting assistance to each three-channel camera. Furthermore, when providing spectral assistance, the images captured by the hybrid camera tend to be consistent with those captured by each of the three-channel cameras, thus balancing the spectral assistance effect of the hybrid camera on each of the three-channel cameras.

[0075] In some possible implementations, the distance between each of the multiple second cameras and the first camera is the same.

[0076] In this implementation, by setting equal intervals, the efficiency of the hybrid camera in assisting each three-channel camera in shooting, as well as the spectral assistance shooting effect of each three-channel camera, can be further balanced.

[0077] In some possible implementations, the first camera includes a hybrid image sensor, which includes event pixels and multispectral pixels, wherein the number of color channels of the multispectral pixels is greater than or equal to 7; the event pixels and multispectral pixels are arranged in a first arrangement or a second arrangement; in the first arrangement, the total area occupied by the event pixels and the total area occupied by the multispectral pixels have a first ratio, and in the second arrangement, the total area occupied by the event pixels and the total area occupied by the multispectral pixels have a second ratio, wherein the first ratio is less than the second ratio.

[0078] In some possible implementations, the first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the total area occupied by the second pixel.

[0079] In this implementation, in the first arrangement, the total area occupied by the event pixels is smaller than the total area occupied by the multispectral pixels, so that the event pixels can achieve auxiliary functions with a small number of pixel occupancy, leaving enough pixel occupancy for the multispectral pixels to have high resolution, so that the multispectral pixels can achieve imaging assistance.

[0080] In some other possible implementations, the first pixel and the second pixel have a second arrangement, wherein the total area occupied by the first pixel is greater than the total area occupied by the second pixel.

[0081] In this implementation, in the second arrangement, the total area occupied by the event pixels can be greater than the total area occupied by the multispectral pixels, so that the hybrid image sensor primarily uses the function of the event pixels and secondarily uses the function of the multispectral pixels.

[0082] In some possible implementations, in the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the second pixel, where M is a positive integer less than or equal to 5. For example, M can be, but is not limited to, 1, 2, 3, 4, or 5.

[0083] In this implementation, the pixel placement design described above ensures that the multispectral pixels have sufficient resolution to achieve the effect of prioritizing multispectral functionality while using event functionality as a secondary function. M should not be too large to avoid the total area occupied by the event pixels being too small, which would affect the working effect of the event pixels.

[0084] In some other possible implementations, in the second arrangement, the total area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5. For example, N can be, but is not limited to, 1, 2, 3, 4, or 5.

[0085] In this implementation, the pixel placement design described above ensures that the event pixels have sufficient resolution to achieve the effect of prioritizing event functionality while using multispectral functionality as a secondary function. N should not be too large to avoid the total area occupied by the event pixels becoming too small, thus affecting their performance.

[0086] In some possible implementations, the first camera has a first working mode, a second working mode, and a third working mode.

[0087] In the first working mode, the first pixel operates independently, while the hybrid camera is used to collect event data and output an event data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for gesture detection and eye tracking, outputting the detected gesture or eye position and providing it to the corresponding algorithm. The algorithm then calculates and activates the function of the electronic device corresponding to the gesture, or calculates the eye position and activates the function icon or application of the electronic device at the eye position.

[0088] In the second working mode, the second pixel operates independently, and the hybrid camera is used to acquire multispectral data and output a multispectral data stream. Different algorithms are applied based on different scenarios, and the data is then output. For example, the hybrid camera can be used for multispectral detection to capture a multispectral image of an object, which can then be output, for instance, stored in a gallery.

[0089] In the third working mode, both the first and second pixels are active. The hybrid camera collects event data and multispectral data, outputting event data streams and multispectral data streams respectively. The event data stream uses different algorithms depending on the application scenario, and the multispectral data stream uses different algorithms for the same scenario. Both calculation results are then output. For example, when dynamically acquiring color, event pixels are used to identify the dynamic object, while multispectral pixels are used to identify the color. Event pixels can help make the color image acquired by multispectral pixels clearer. As another example, when the hybrid camera uses multispectral pixels for spectral detection, event pixels can be used for gesture detection to activate different function options or enable shooting.

[0090] In some possible implementations, the first data stream includes an event data stream; the electronic device generates an image based on the first data stream and the second data stream, specifically including: the electronic device generates a first image based on the event data stream and the second data stream.

[0091] It should be noted that the second data stream is the data stream formed by the RGB data collected by the image sensor of the three-channel camera.

[0092] The hybrid camera collects event data through event pixels to form an event data stream. After processing by the ESP (Electronic Stability Program), different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the ISP (Image Signal Processor) for image preprocessing. The ISP combines the RGB data stream and the relevant results to perform image processing and generate an image. The event pixels provide functional assistance to the three-channel camera, and these scenarios can include: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and assisted OIS (Optical Image Interpretation).

[0093] The hybrid camera acquires event data through event pixels to form an event data stream. This stream is processed by the ESP (Electronic Stability Program) and then subjected to different algorithms depending on the scene to obtain the event image. The RGB data stream undergoes image preprocessing by the ISP (Image Signal Processor) to form the original RGB image. The ISP then uses an image fusion algorithm to fuse the event image and the original RGB image to generate the final image. The event pixels provide imaging assistance to the three-channel camera, and their applications can include: deblurring, video frame interpolation / slow motion, assisted EIS (Electronic Image Sensor), assisted OIS (Optical Image Sensor), depth detection, portrait blurring, and Livephoto.

[0094] In some possible implementations, the first data stream includes a multispectral data stream; the electronic device generates an image based on the first data stream and the second data stream, specifically including: the electronic device generates a second image based on the first multispectral data and the second data stream.

[0095] The hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After processing by the M-ISP, different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the ISP's image preprocessing algorithm. The ISP combines the RGB data stream and the relevant results to perform image preprocessing and generate an image. The multispectral pixels provide functional assistance to the three-channel camera, and their applications can include spectral inversion and high-dimensional white point estimation.

[0096] The hybrid camera uses multispectral pixels to acquire multispectral data, forming a multispectral data stream. This stream is then processed by an M-ISP (Multi-Spectral Image Processor), which applies different algorithms based on the scene to obtain relevant results. The RGB data stream undergoes pre-processing by the ISP to obtain the original RGB image. The ISP then uses post-processing algorithms to apply the relevant results to the original RGB image to generate the final image. The multispectral pixels provide functional assistance to the three-channel camera, and its applications include: ambient light spectrum estimation, true color estimation, and infrared and ultraviolet applications.

[0097] The hybrid camera uses multispectral pixels to acquire multispectral data, forming a multispectral data stream. This stream is then processed by an M-ISP (Multi-Spectral Image Processor), which applies different algorithms depending on the scene to obtain a multispectral image. The RGB data stream undergoes preprocessing by the ISP to produce the original RGB image. Finally, the ISP uses an image fusion algorithm to fuse the multispectral and original RGB images to generate the final image. The multispectral pixels provide imaging assistance to the three-channel camera, and its applications include: object spectral decomposition, color layering, tonal gradation, beautification and makeup effects, and creating interactive avatars.

[0098] In some possible implementations, the first data stream includes an event data stream and a multispectral data stream; the electronic device generates an image based on the first data stream and the second data stream, specifically including: the electronic device generates a third image based on the event data stream, the multispectral data stream and the second data stream.

[0099] In this implementation, the hybrid camera can acquire event data and form an event data stream through event pixels, and acquire multispectral pixels and form a multispectral data stream through multispectral pixels. The event data stream can be used for functional assistance or imaging assistance, and the multispectral data stream can be used for functional assistance or imaging assistance. According to the above embodiments, the application of event data stream and multispectral data can be combined to assist the three-channel camera in image preprocessing and / or image postprocessing, enriching the shooting scenarios of the three-channel camera and diversifying the generated images.

[0100] In some possible implementations, event pixels and multispectral pixels are arranged in a first configuration, the first event data is collected in a first scene, and the first multispectral data is collected in a second scene. The first scene includes any of the following: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and assisted OIS. The second scene includes any of the following: spectral inversion, high-dimensional white point estimation, ambient light source spectral estimation, True color estimation, infrared and ultraviolet applications, object spectral decomposition, color layering, tonal style, beautification and makeup, and shooting and creating interactive avatars.

[0101] In this implementation, the main purpose of the hybrid image sensor is to meet the needs of multispectral devices. Event pixels occupy a small amount of pixel area and mainly perform some auxiliary functions.

[0102] It should be noted that the first scenario is the scenario where event pixels are used to achieve functional assistance, and the second scenario includes the scenario where multispectral pixels can be used to achieve imaging assistance. Since multispectral pixels have higher resolution and can be compatible with functional assistance, the second scenario also includes the scenario where multispectral pixels can be used to achieve functional assistance.

[0103] In some possible implementations, event pixels and multispectral pixels are arranged in a second configuration. The first event data is collected in a third scenario, and the first multispectral data is collected in a fourth scenario. The third scenario includes any of the following: stroboscopic frequency and position detection, stroboscopic mask segmentation, motion region detection / optical flow, moving target detection, deblur, video frame interpolation / slow motion, assisted EIS, assisted OIS, detection depth, portrait blurring, and Livephoto. The fourth scenario includes any of the following: spectral inverse, high-dimensional white point estimation, ambient light source spectral estimation, True color estimation, and infrared and ultraviolet applications.

[0104] In this implementation, since the total area occupied by the event pixels is larger, the event pixels can be used to improve the image quality of the three-channel camera. Since the total area occupied by the multispectral pixels is smaller, they can be used as auxiliary devices to achieve some auxiliary functions.

[0105] It should be noted that the third scenario includes scenarios where event pixels are used for imaging assistance. Because event pixels have higher resolution and are compatible with functional assistance, the third scenario also includes scenarios where event pixels are used for functional assistance. The fourth scenario is scenarios where multispectral pixels are used for functional assistance.

[0106] In some possible implementations, the number of event pixels is multiple; the electronic device collects first data through the first camera, specifically including: the electronic device merging at least a portion of the multiple event pixels into a pixel unit, and collecting event data as a whole through the pixel unit.

[0107] In this implementation, adjacent pixels in the event pixel are binning to form a larger pixel, thereby achieving a larger photosensitive area, collecting more light, and thus improving the signal-to-noise ratio and sensitivity of the acquired event data. It should be noted that both the first and second pixel units are event pixels.

[0108] In some possible implementations, the second camera includes any one or more of the main camera, wide-angle camera, and telephoto camera of the electronic device.

[0109] In this implementation, the hybrid camera can assist one or more three-channel cameras in shooting, thereby improving the imaging effect of the three-channel cameras.

[0110] Fourthly, this application also provides a computer-readable storage medium storing a program or instructions that, when executed, implement any of the methods described above.

[0111] Fifthly, this application also provides a computer program product that stores a program or instructions that, when executed, implement any of the methods described above. Attached Figure Description

[0112] Figure 1A This is a schematic diagram of the spectral response curves of broadband multispectral transmission in some embodiments;

[0113] Figure 1B This is a schematic diagram of the spectral response curves of narrowband multispectral in some embodiments;

[0114] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0115] Figure 3 This is a software structure block diagram of an electronic device according to an embodiment of this application;

[0116] Figure 4A This is a schematic diagram showing the placement of the hybrid camera provided in this application as a front-facing camera in some embodiments;

[0117] Figure 4B This is a schematic diagram showing the placement of the hybrid camera provided in this application as a rear camera in some embodiments;

[0118] Figure 4CThis is a schematic diagram showing the placement of the hybrid camera as a front-facing camera in some other embodiments, as provided in this application.

[0119] Figure 5A This is a schematic diagram of the first arrangement of the hybrid sensors provided in this application in some embodiments;

[0120] Figure 5B This is a schematic diagram of the first arrangement of the hybrid sensor provided in this application in some other embodiments;

[0121] Figure 5C This is a schematic diagram of the first arrangement of the hybrid sensors provided in this application in some other embodiments;

[0122] Figure 6A This is a schematic diagram of the second arrangement of the hybrid sensors provided in this application in some embodiments;

[0123] Figure 6B This is a schematic diagram of the second arrangement of the hybrid sensor provided in this application in some other embodiments;

[0124] Figure 6C This is a schematic diagram of a second arrangement of the hybrid sensors provided in this application in some other embodiments;

[0125] Figure 6D This is a schematic diagram of a second arrangement of the hybrid sensors provided in this application in some other embodiments;

[0126] Figure 6E This is a schematic diagram of a second arrangement of the hybrid sensors provided in this application in some other embodiments;

[0127] Figure 7 This is a schematic diagram of the framework of the hybrid camera provided in this application as a functional auxiliary device in some embodiments;

[0128] Figure 8 This is a schematic diagram of the frame of the hybrid camera provided in this application as an imaging aid in some embodiments;

[0129] Figure 9 This is a flowchart illustrating some embodiments of the image processing method provided in this application;

[0130] Figure 10 This is a flowchart illustrating the image processing method provided in this application in some other embodiments. Detailed Implementation

[0131] The embodiments of this application are described below with reference to the accompanying drawings.

[0132] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, "connection" can be a detachable connection or a non-detachable connection; it can be a direct connection or an indirect connection through an intermediate medium. "Multiple" refers to at least two.

[0133] The directional terms mentioned in the embodiments of this application, such as "upper", "lower", "inner", "outer", "top", "bottom", "side", etc., are only for reference to the directions in the accompanying drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of this application, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0134] In the embodiments of this application, the relative positional relationships mentioned, such as parallel, perpendicular, and aligned, are defined in relation to the current technological level, rather than being absolutely strict. Slight deviations are permissible; approximations of parallelism, perpendicularity, or alignment are all acceptable. For example, "A and B are parallel" means that A and B are parallel or approximately parallel, and the angle between A and B can be between 0 and 10 degrees. Similarly, "A and B are perpendicular" means that A and B are perpendicular or approximately perpendicular, and the angle between A and B can be between 80 and 100 degrees.

[0135] In the embodiments of this application, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," and "fourth" may explicitly or implicitly include one or more of that feature.

[0136] To facilitate understanding, some related concepts involved in the embodiments of this application will be explained below.

[0137] 1. Event camera

[0138] An event camera, also known as an event camera, dynamic vision sensor (DVS), or neuromorphic vision sensor, is completely different from three-channel cameras such as complementary metal oxide semiconductor (CMOS) RGB cameras and charge coupled device (CCD) RGB cameras in terms of imaging mechanism, data format, and processing mechanism.

[0139] Unlike three-channel cameras that synchronously capture light intensity information from all pixels at a fixed frequency, event cameras asynchronously capture brightness changes in each pixel. This means each pixel in an event camera is independent, does not affect others, and outputs an event independently. Event cameras output high-resolution event data, achieving an accuracy of 1 microsecond (µs), with low latency, thus enabling the capture of fast-moving objects. Furthermore, because there is no concept of exposure time, event cameras do not suffer from motion blur. In addition, event cameras can operate normally in both very bright and very dark environments, exhibiting a high dynamic range. The pixels in the event camera, referred to as event pixels in this application, are individually capable of acquiring light intensity changes and outputting event signals.

[0140] Based on whether the brightness of the event pixel increases or decreases, "events" can be divided into positive polarity events and negative polarity events. When the brightness of the event pixel increases and the increase reaches a first threshold, a positive event signal is output, i.e., a positive polarity event; when the brightness of the event pixel decreases and the decrease reaches a second threshold, a negative event signal is output, i.e., a negative polarity event. The first and second thresholds can be set according to actual needs.

[0141] By reading the "events" from each event pixel of the event camera, an event data stream can be obtained. Event camera readout methods can be divided into asynchronous readout and synchronous readout.

[0142] In asynchronous readout mode, when the brightness value of an event pixel changes and the change exceeds a set threshold, an event signal is output. The Address Event Representation (AER) circuit assigns an address to the triggered event signal, namely the x and y coordinates of the event signal, as well as a timestamp, ultimately outputting a four-dimensional event data packet in the form of (x, y, p, t). Here, x and y represent the horizontal and vertical coordinates of the event pixel, respectively; p represents the polarity value of the event pixel, i.e., whether the brightness of the event pixel increases or decreases; and t represents the timestamp, i.e., the time when the brightness of the event pixel changed. Event cameras that use the above asynchronous readout method are also called asynchronous event cameras.

[0143] In synchronous readout mode, when the brightness value of an event pixel changes and the change exceeds a set threshold, an event signal is output. A timestamp is then added to the event signal, and the event data is read out in the form of event frames to obtain a synchronous event frame data stream. The event frame rate is very high, reaching 10,000 frames per second or higher. Each event frame can share one or more timestamps.

[0144] Unlike asynchronous readout, synchronous readout reads event data in the form of event frames. When reading each event frame, if the brightness change value of an event pixel reaches the set threshold of the ON comparator circuit, the value of that pixel in the event frame is +1; if the brightness change value of a pixel reaches the set threshold of the OFF comparator circuit, the value of that pixel in the event frame is -1; if the brightness change value of a pixel does not reach either the ON or OFF comparator circuit's set threshold, the value of that pixel in the event frame is 0. Therefore, the value of each event pixel in a synchronously read event frame is +1, -1, or 0. Since the entire event frame is read, it is no longer necessary to assign an address to each valid event signal, and the readout circuit for the event pixel may not include an AER circuit.

[0145] 2. Multispectral camera

[0146] A multispectral camera, also known as a hyperspectral camera, features multiple color channels (more than three) in each photosensitive unit. These channels can be created by using chemical dyes, coating interference, or metasurface micro / nano structures to create differentiated color filter arrays (CFAs). Therefore, a multispectral image sensor can extract spectral information from multiple channels to acquire images. Because multispectral image sensors have more color channels and stronger spectral sensing capabilities, the colors in the acquired multispectral images are more accurate. Multispectral image sensors acquire multispectral images through multiple filter channels and can transmit the acquired image information to a color reproduction module for analysis, achieving color imaging or color information extraction.

[0147] A multispectral image sensor can include a spectral modulation region and a photoelectric conversion region. The spectral modulation region uses various materials or optical structures to form multiple color channels to split the incident light. The photoelectric conversion region converts the split light signal into an electrical signal, and then outputs a digital signal or encoding through analog-to-digital conversion. A color restoration module (e.g., an image signal processor) is electrically connected to the multispectral image sensor. The color restoration module can calculate spectral or color information based on the light signal and pixel position information input from the multispectral image sensor. Based on the color transformation matrix from multiple channel signals to the color space, the color restoration module can also convert the multispectral signals acquired by the multispectral image sensor into the color space information of the image (e.g., RGB information, XYZ tristimulus values, or YUV information). Here, the color transformation matrix from multiple channel signals to the color space is established based on the visual response curve of the human eye (e.g., the spectral tristimulus value curve, including the human eye's response curve to red, green, and blue light) and the spectral response curve of the multispectral camera module (including the multispectral camera module's spectral response curve to different wavelengths). The color restoration module can also use the spectral information acquired by the multispectral camera module to perform color calibration on other cameras. For example, based on the transformation relationship between the spectral response curve of the multispectral camera module and the spectral response curves of other camera modules, it can map the color space information corresponding to the multispectral camera module onto other camera modules to correct their colors, thus enabling other cameras to achieve color calibration.

[0148] For a description of color channels, please refer to [link / reference needed]. Figure 1A and Figure 1B , Figure 1A This is a schematic diagram of the spectral response curves of broadband multispectral transmission in some embodiments; Figure 1B This is a schematic diagram of the spectral response curves of narrowband multispectral in some embodiments.

[0149] in, Figure 1A and Figure 1B In the diagram, the horizontal axis represents wavelength, and the vertical axis represents transmittance. Different color channels are represented by different lines. Broadband means that each color channel has a large wavelength response range, while narrowband means that each color channel has a small wavelength response range, such as multispectral channels with infrared or ultraviolet channels.

[0150] The number of color channels in a multispectral camera corresponds to the number of response curves. For example, Figure 1A The diagram shows nine different response curves, each corresponding to a color channel of nine. Figure 1B The diagram shows 16 different response curves, each corresponding to a color channel of 16. If the multispectral array has infrared or ultraviolet channels, then one or more color channels will have peak response curves in the range of less than 400 nm (ultraviolet) or greater than 700 nm (infrared).

[0151] 3. Three-channel camera

[0152] A three-channel camera, also known as an RGB camera or an RYB camera, typically operates as follows: During the exposure time, light passes through the lens to the photosensitive element; the photosensitive element senses the light signal and generates an electrical signal, which is then transmitted to the image signal processor (ISP); the ISP converts the electrical signal into image data and outputs the image data at a fixed frame rate.

[0153] In the aforementioned imaging process, the three-channel camera requires a sustained exposure period, allowing the photosensitive element to accumulate a certain number of photons. If the object being photographed moves at high speed relative to the camera during this exposure period, motion blur will occur. Furthermore, even high-performance three-channel cameras with high frame rates still experience millisecond-level latency; in low light or high brightness conditions, the three-channel camera acquires limited information, resulting in a lower dynamic range.

[0154] In other words, due to their imaging mechanisms and processing mechanisms, three-channel cameras typically suffer from problems such as motion blur, high latency (millisecond level), and low dynamic range, resulting in lower imaging capabilities for electronic devices and a poorer imaging experience for users.

[0155] Furthermore, in three-channel camera modules, each photosensitive unit in the image sensor includes three different color channels. The CFA in a three-channel camera is typically RGGB or RYYB, for example, red, green, and blue color channels, or red, yellow, and blue color channels. Therefore, limited by material capabilities, it can usually only capture color information from the red, green, and blue channels, resulting in relatively low spectral resolution and limited information. It mainly relies on the color and brightness information of the image for analysis, leading to relatively weak data acquisition and analysis capabilities. Its corresponding spectral response curve differs somewhat from the spectral response curve of the human eye, and the number of spectral bands that can be perceived is relatively small, resulting in weak color reproduction capabilities and the acquisition of images prone to color casts.

[0156] In other words, due to the limited number of color channels, three-channel cameras can only capture limited color information, resulting in weak data acquisition and analysis capabilities. This leads to lower shooting capabilities of electronic devices and a poorer user experience.

[0157] Currently, electronic devices with shooting capabilities are usually equipped with three-channel cameras. As a result, the imaging capabilities of these three-channel cameras are limited, leading to issues such as motion blur, high latency (millisecond level), low dynamic range, and color deviation in the images captured by these devices.

[0158] To address the aforementioned issues, an event camera can be introduced into the electronic device to assist the three-channel camera in capturing images. The event camera can overcome problems such as motion blur, high latency (millisecond level), and low dynamic range. Simultaneously, a multispectral camera can be introduced into the electronic device to assist the three-channel camera in capturing images, and the multispectral camera can overcome problems such as color deviation.

[0159] However, in electronic devices, taking the rear camera configuration of mobile phones as an example, current mainstream rear camera configurations include at least one main RGB or RYB camera, one wide-angle RGB or RYB camera, one telephoto RGB or RYB camera, and other auxiliary cameras or components, such as multispectral cameras, event cameras, flash, and time-of-flight (TOF) sensors. Both event cameras and multispectral cameras require complete lens assemblies, autofocus (AF) and / or optical image stabilization (OIS) motor assemblies. Therefore, if a mobile phone's camera needs to function as both an event camera and a multispectral camera, at least five lens holes need to be created on the back cover, thus affecting the phone's appearance design.

[0160] If an event camera and a three-channel camera from an electronic device are hybridized, the event pixels need to be incorporated into the image sensor of the three-channel camera. In this case, the space occupied by the event pixels reduces the pixel footprint of the image sensor in the three-channel camera, thus affecting the image quality captured by the three-channel camera. Similarly, hybridizing a multispectral camera and a three-channel camera will also affect the image quality captured by the three-channel camera.

[0161] To address the aforementioned issues, this application provides an image processing method, a hybrid camera, and an electronic device. The electronic device includes a hybrid camera and at least one three-channel camera. The hybrid camera includes a hybrid image sensor, which comprises event pixels and multispectral pixels. By designing the hybrid image sensor, the functions of the event camera and the multispectral camera are hybridized, enabling the hybrid image sensor to assist the three-channel camera in both dynamic shooting and color reproduction. This solves the aforementioned shooting problems of the three-channel camera. Furthermore, since the hybrid camera only requires one lens assembly, AF and / or OIS motor assembly, it not only reduces the number of openings on the electronic device's casing, improving the device's appearance, but also reduces the size of the camera module and the area of ​​the corresponding readout circuit, improving the overall size of the camera configuration and reducing interference with the overall size design of the electronic device.

[0162] It should be noted that a black and white color channel, an infrared channel, and an ultraviolet channel can also be added to a three-channel camera.

[0163] The image processing method provided in this application can be applied to electronic devices with shooting functions, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart home devices, and other electronic devices. This application does not impose any restrictions on the specific type of electronic device.

[0164] The following is combined Figure 2 The present application will now describe the structure of an electronic device 100 provided in an embodiment.

[0165] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application.

[0166] Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0167] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0168] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), event signal processor (ESP), multi-spectrum image signal processor (M-IPS), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0169] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0170] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0171] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0172] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0173] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0174] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0175] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0176] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0177] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0178] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0179] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0180] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0181] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0182] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0183] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0184] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. The one or N cameras 193 may include a hybrid camera, or a hybrid camera and at least one three-channel camera.

[0185] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0186] The Information Service Provider (ISP) is used to process data from a three-channel camera. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the three-channel camera's image sensor transmits this electrical signal to the ISP for processing, transforming it into a visible image. The ISP can also perform algorithmic optimizations on image noise, brightness, etc. Furthermore, the ISP can optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the three-channel camera.

[0187] A three-channel camera is used to capture still images or videos. An object passes through the lens, generating an optical image that is projected onto a photosensitive element. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Image Signal Processor) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for further processing. The DSP converts the digital image signal into standard image signals in formats such as RGB and YUV.

[0188] Electronic device 100 can also optimize the shooting of the three-channel camera through EPS and / or M-IPS and ISP.

[0189] EPS (Expanded Preprocessing) can be used to process event data output from event pixels in hybrid cameras. It preprocesses the event data using various filtering algorithms to remove noisy events, improving the accuracy and reliability of the event data. For example, it can determine whether an event is noise and filter it out based on characteristics such as the event's occurrence, spatial distribution, and polarity.

[0190] EPS can also extract features from event data. EPS can analyze event data to extract motion characteristics of objects, such as direction of motion, speed, and acceleration. This is crucial for subsequent motion tracking and target recognition. For example, by performing spatiotemporal analysis on consecutive events, the trajectory and velocity distribution of objects on the pixel plane can be calculated. EPS can also analyze event data to extract shape features of objects. When an object moves in a scene, a hybrid camera can capture brightness changes at the object's edges, forming a series of event signals. EPS can use these event signals to extract edge features of objects, such as their contours and boundaries. This aids in object recognition and classification. For example, by clustering event signals, events belonging to the same object can be grouped together, thus obtaining the approximate shape of the object.

[0191] EPS can also fuse event information acquired by hybrid cameras with information acquired by three-channel cameras, comprehensively utilizing the advantages of different sensors to improve scene understanding and analysis capabilities. For example, EPS can fuse motion information acquired by hybrid cameras with image information acquired by three-channel image sensors to deblur the captured images and improve their clarity.

[0192] M-ISP can separate and integrate spectra. Hybrid cameras can capture spectral information from multiple different bands. M-ISP is responsible for separating and extracting raw image data from the same spectral channel. For example, it can distinguish data from different spectral ranges such as visible light, near-infrared, and ultraviolet, providing a basis for subsequent independent processing. This separation operation allows information from each spectral band to be analyzed and processed separately, in order to better understand the characteristics of the target object under different spectra.

[0193] M-ISP can also enhance and correct multispectral images. Multispectral data acquired by hybrid cameras may contain various types of noise. M-ISP employs various noise removal algorithms to suppress and eliminate noise in multispectral images, improving the signal-to-noise ratio. For example, by comparing and analyzing adjacent pixels, it identifies and corrects abnormal pixel values, or uses filtering algorithms to smooth the image, reducing the impact of noise on multispectral image quality. Because different spectral bands have different response characteristics and sensitivities, images captured by hybrid cameras may exhibit color deviations or distortions. M-ISP uses color correction algorithms to adjust and calibrate the colors of the image, ensuring consistency and accuracy in color across different spectral bands. For example, based on known standard color samples or reference images, it matches and corrects the colors of multispectral images, ensuring that the image colors accurately reflect the spectral characteristics of the target object.

[0194] M-ISP can also perform feature extraction and analysis. Multispectral images contain rich spectral information, which can be used for target identification and classification. M-ISP utilizes image processing and pattern recognition techniques to extract and analyze the features of targets in multispectral images, identify the target's characteristic information, and classify and identify the target based on these features. For example, in geological exploration, different types of rocks and minerals can be identified by analyzing the spectral characteristics of rocks in multispectral images.

[0195] M-ISP can also fuse multispectral information acquired by a hybrid camera with information acquired by a three-channel camera. A three-channel camera can only capture information from red, green, and blue, or red, yellow, and blue channels. For objects with complex colors or special materials, it may not be able to accurately reproduce their true colors. A hybrid camera, on the other hand, can acquire spectral information from multiple different bands, including visible light and near-infrared. By fusing multispectral data, M-ISP can more accurately determine the color characteristics of an object and correct color deviations that may occur with three-channel cameras. For example, when photographing flowers, a hybrid camera can capture subtle color changes and textures, assisting a three-channel camera in better presenting the flower's true colors and details.

[0196] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0197] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0198] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0199] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0200] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0201] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0202] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0203] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0204] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0205] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0206] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0207] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0208] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0209] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0210] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an embedded SIM (eSIM) card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0211] It should be understood that the phone cards in the embodiments of this application include, but are not limited to, SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated circuit cards (UICC), etc.

[0212] The following is combined Figure 3 This application will now introduce the software structure of an electronic device 100 provided in an embodiment.

[0213] Please see Figure 3 , Figure 3 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0214] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0215] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer can include a series of application packages.

[0216] like Figure 3 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0217] In this embodiment, the camera application can be used to assist the three-channel camera in shooting simultaneously with a hybrid camera, thereby achieving better shooting results. After the camera application is launched, the electronic device can automatically identify the shooting scene through a recognition algorithm to control the hybrid camera for assisted shooting. For example, event pixels in the hybrid camera can improve the sharpness of the three-channel camera shots, while multispectral pixels can optimize the colors captured by the three-channel camera.

[0218] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0219] like Figure 3 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0220] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0221] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0222] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0223] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0224] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0225] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0226] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0227] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0228] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0229] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0230] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0231] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0232] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0233] A 2D graphics engine is a graphics engine for 2D drawing.

[0234] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0235] It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

[0236] The embodiments of this application will now be described in detail.

[0237] Example 1: Placement and shape of hybrid cameras.

[0238] Please refer to the following: Figures 4A to 4C , Figure 4A This is a schematic diagram showing the placement of the hybrid camera provided in this application as a front-facing camera in some embodiments; Figure 4B This is a schematic diagram showing the placement of the hybrid camera provided in this application as a rear camera in some embodiments; Figure 4C This is a schematic diagram showing the placement of the hybrid camera provided in this application as a front-facing camera in some other embodiments.

[0239] In some embodiments, the electronic device may include a hybrid camera and at least one three-channel camera.

[0240] In some examples, please refer to Figure 4A The light-input point of the hybrid camera can be located on a different surface of the electronic device than the light-input point of the three-channel camera. Detection is performed using the hybrid camera to activate functions associated with the three-channel camera, thereby assisting the three-channel camera in its operation.

[0241] For example, a hybrid camera can function as a front-facing camera, while a three-channel camera can function as a rear-facing camera. In some examples, the hybrid camera can activate the camera app via gesture detection or eye tracking to start the three-channel camera. In other examples, when the three-channel camera is on, shooting can be activated via gesture detection or eye tracking.

[0242] For example, the electronic device can be a foldable electronic device, and the hybrid camera and the three-channel camera can be located in different foldable parts so that when the electronic device is unfolded, the hybrid camera and the three-channel camera are located on the same side of the electronic device, thereby enabling the hybrid camera to assist the three-channel camera in taking pictures.

[0243] In other examples, please refer to Figure 4B and Figure 4C The light-input end of the hybrid camera can be located on the same side of the electronic device as the light-input end of the three-channel camera, so that the hybrid camera can assist the three-channel camera in taking pictures.

[0244] The hybrid camera can be placed adjacent to at least one three-channel camera. It should be noted that "adjacent" here means that there are no camera modules, flashes, sensors, or other imaging-related devices between the hybrid camera and the three-channel camera.

[0245] In this embodiment, by placing the hybrid camera and the three-channel camera adjacent to each other, the communication distance between them is shortened, thereby improving the communication efficiency of the hybrid camera assisting the three-channel camera in shooting. Furthermore, by reducing the distance between the hybrid camera and the three-channel camera, when the hybrid camera uses spectral detection for assisted shooting, the captured image from the hybrid camera tends to be consistent with the captured image from the three-channel camera, improving the accuracy of spectral-assisted shooting.

[0246] In some embodiments, there can be multiple three-channel cameras, and each of the multiple three-channel cameras can be arranged adjacent to the hybrid camera.

[0247] In this embodiment, by placing the hybrid camera adjacent to each of the three-channel cameras, it is convenient for the hybrid camera to provide shooting assistance to each of the three-channel cameras, which helps to balance the efficiency of shooting assistance to each three-channel camera. Furthermore, it is beneficial to ensure that when the hybrid camera provides spectral assistance, the captured images are more consistent with those captured by each of the three-channel cameras, thus balancing the spectral assistance effect of the hybrid camera on each of the three-channel cameras.

[0248] In this configuration, the distance between each camera in a hybrid camera system and multiple three-channel cameras can be equal.

[0249] In this embodiment, by setting the distance equally, the efficiency of the hybrid camera in assisting each three-channel camera in shooting, as well as the spectral assistance effect of each three-channel camera, can be further balanced.

[0250] It should be noted that in some other embodiments, the hybrid camera can be positioned closer to one of the three-channel cameras, depending on actual needs, in order to optimize the auxiliary shooting efficiency and spectral auxiliary shooting effect of the three-channel camera.

[0251] In some examples, please refer to Figure 4B The hybrid camera can be a rear-facing camera, and the number of three-channel cameras can be one or more. A three-channel camera can include any one or more of a main camera, a wide-angle camera, and a telephoto camera. In addition, other related shooting devices can be set, such as other cameras, flash, or TOF, etc.

[0252] In other examples, please refer to Figure 4C A hybrid camera can be a front-facing camera, while a three-channel camera is usually one camera, which can also be the main camera.

[0253] It should be noted that, Figure 4A and Figure 4B The illustrations only show the placement of hybrid cameras and three-channel cameras, the number of three-channel cameras, and the size of each camera; no specific limitations are imposed.

[0254] It should be noted that the hybrid camera in the electronic device can work independently, or the electronic device can include only the hybrid camera and not the three-channel camera.

[0255] Please refer to the following: Figures 5A to 5C , Figure 5A This is a schematic diagram of the first arrangement of the hybrid sensors provided in this application in some embodiments; Figure 5B This is a schematic diagram of the first arrangement of the hybrid sensor provided in this application in some other embodiments; Figure 5C This is a schematic diagram of the first arrangement of the hybrid sensor provided in this application in some other embodiments.

[0256] It should be noted that, Figures 5A to 5C In the diagram, pixels marked E are event pixels, and pixels marked M are multispectral pixels.

[0257] In some embodiments, the hybrid camera may include a hybrid image sensor and an optical lens. The hybrid image sensor may be located on the image side of the optical lens to sense light incident on the lens. The hybrid image sensor may include event pixels and multispectral pixels.

[0258] In this embodiment, the event camera function and the multispectral camera function are integrated by combining event pixels and multispectral pixels onto the same image sensor. By sharing an optical lens and image sensor with a single camera module, the functions of two cameras are realized, saving the size space of a single camera module and facilitating the thinner and lighter design of electronic devices.

[0259] Among them, the number of color channels of the multispectral pixels is greater than or equal to 7, so that the hybrid camera has a good color acquisition effect.

[0260] In this design, the number of color channels in a multispectral pixel can be less than or equal to 16 to ensure practicality. Having too many color channels in a multispectral pixel yields minimal additional benefits and presents significant challenges in design and fabrication due to consistency and process difficulties.

[0261] Specifically, the arrangement of event pixels and multispectral pixels in the hybrid image sensor is designed according to requirements.

[0262] In some embodiments, event pixels and multispectral pixels are arranged in a first configuration. In this first configuration, the hybrid image sensor primarily utilizes the function of multispectral pixels, with the function of event pixels playing a secondary role.

[0263] In this embodiment, the main purpose of the hybrid image sensor is to meet the needs of multispectral devices. Event pixels occupy a small amount of pixel area and mainly perform some auxiliary functions.

[0264] For example, in the first arrangement, the total area occupied by event pixels is smaller than the total area occupied by multispectral pixels. This allows event pixels to perform auxiliary functions with a small pixel footprint, while leaving sufficient pixel space for multispectral pixels to achieve high resolution and enable them to assist in imaging. Here, the total area occupied by event pixels refers to the sum of the areas occupied by all event pixels in the entire hybrid image sensor. Similarly, the total area occupied by multispectral pixels refers to the sum of the areas occupied by all multispectral pixels in the entire hybrid image sensor.

[0265] For example, in the first arrangement, the event pixels of the hybrid image sensor can be used for flicker detection, motion detection, etc. The detection results can assist the AE algorithm of the three-channel camera in adjusting some exposure strategies, including exposure control in flickering scenes, or exposure reduction strategies in moving scenes, or provide the AF algorithm with the detection box of the ROI or moving target to help the AF algorithm to focus quickly. The event pixels can have a certain resolution, such as, but not limited to, 302*204 or 640*480.

[0266] In this embodiment, during flicker frequency or mask detection, the event pixel can detect and output local frequencies, meaning different locations output different flicker frequencies. Furthermore, the detectable flicker frequency range is wider than that of a conventional flicker sensor, resulting in better flicker frequency or mask detection. Therefore, in this embodiment, by designing a hybrid image sensor, the flicker sensor in the electronic device can be replaced, saving on sensor structure while achieving better flicker frequency or mask detection.

[0267] In some embodiments, in a hybrid image sensor, an event pixel may include multiple pixels, which may be arranged at intervals, or some pixels may be arranged adjacently to binning into a larger pixel, thereby achieving a larger photosensitive area. In this case, the pixels after binning work together to jointly acquire changes in light intensity and output event signals.

[0268] It should be noted that the adjacent setting here means that there are no pixels of the same type between two adjacent pixels, that is, there are no other event pixels between two adjacent event pixels, but multispectral pixels can exist.

[0269] For example, the total area occupied by multispectral pixels can be M times the total area occupied by event pixels, where M is a positive integer less than or equal to 5, so that the multispectral pixels have sufficient resolution to achieve the effect of multispectral function as the primary function and event function as the secondary function. M should not be too large to avoid the total area occupied by event pixels being too small, which would affect the working effect of event pixels.

[0270] Please refer to Figure 5A The hybrid image sensor includes one event pixel and eight multispectral pixels with color channels. Each 3x3 pixel group can allocate eight pixels to sense different wavelengths of visible light, near-infrared light, or ultraviolet light, with one pixel designated as the event pixel. In this hybrid sensor, the filter on the event pixel can selectively filter out infrared light. When the filter on the event pixel does not filter out infrared light, the event pixel can be combined with a structured light emitter to act as a structured light receiver. This allows the hybrid image sensor to achieve more functions, eliminates the need for a Time-of-Flight (TOF) receiver, and facilitates the design of thinner and lighter electronic devices.

[0271] Please refer to Figure 5BThe hybrid image sensor includes an event pixel and 15 multispectral pixels with color channels. Each group of 4*4 pixels can be allocated 15 pixels to sense different wavelengths of visible light, near-infrared light, or ultraviolet light, and can be allocated 1 pixel as an event pixel.

[0272] Please refer to Figure 5C The hybrid image sensor includes event pixels and multispectral pixels with 12 color channels. Each group of 4*4 pixels can be allocated 12 pixels to sense different wavelengths of visible light, near-infrared light, or ultraviolet light. Four pixels can be binned into one event pixel, that is, the area occupied by one event pixel is equal to the area occupied by four multispectral pixels.

[0273] It should be noted that, Figures 5A to 5C The method does not limit the perceived color wavelength in multispectral pixels. Multispectral pixels can be implemented by changing the material of the color filter (CF), or by adding blue glass (BG) and coating the BG with an anti-reflective (AR) or infrared (IR) film. The multispectral spectrum corresponding to the color channel of the multispectral pixel can be narrowband or broadband multispectral; this is not limited here.

[0274] It should be noted that, Figures 5A to 5C In this diagram, the relative positions of event pixels are for illustrative purposes only. The actual positions can be adjusted as needed. Event pixels can be multiple scattered pixels or a large pixel formed by binning several pixels.

[0275] Please refer to the following: Figures 6A to 6E , Figure 6A This is a schematic diagram of the second arrangement of the hybrid sensors provided in this application in some embodiments; Figure 6B This is a schematic diagram of the second arrangement of the hybrid sensor provided in this application in some other embodiments; Figure 6C This is a schematic diagram of a second arrangement of the hybrid sensors provided in this application in some other embodiments; Figure 6D This is a schematic diagram of a second arrangement of the hybrid sensors provided in this application in some other embodiments; Figure 6E This is a schematic diagram of a second arrangement of the hybrid sensor provided in this application in some other embodiments.

[0276] In some embodiments, it should be noted that, Figures 6A to 6E In the diagram, pixels marked E are event pixels, and pixels marked M are multispectral pixels.

[0277] In some embodiments, the event pixels and multispectral pixels are arranged in a second configuration. In this second configuration, the ratio of the total area occupied by the event pixels to the total area occupied by the multispectral pixels is larger than that in the first configuration, enabling the event pixels to perform more functions.

[0278] For example, in the second arrangement, the total area occupied by the event pixels can be greater than the total area occupied by the multispectral pixels, so that the hybrid image sensor uses the function of the event pixels as the main function and the function of the multispectral pixels as the auxiliary function.

[0279] In this implementation, the event pixels occupy a larger total area, enabling them to be used to improve the image quality of the three-channel camera, such as for image deblurring and video frame interpolation. Since the multispectral pixels occupy a smaller total area, they can be used as auxiliary devices to perform auxiliary functions, such as estimating the ambient light spectrum and calculating high-dimensional white points.

[0280] For example, the total area occupied by the event pixels can be N times the total area occupied by the multispectral pixels. N can be a positive integer less than or equal to 5 to ensure the event pixels have sufficient resolution to achieve the effect of prioritizing event functionality while using multispectral functionality as a secondary function. N should not be too large to avoid the total area occupied by the event pixels being too small, thus affecting the working effect of the event pixels.

[0281] Please refer to Figure 6A The hybrid image sensor includes multispectral pixels with nine color channels. The footprint of each individual event pixel is relatively large, approximately 1 / 3 * 1 / 3 of the footprint of a single event pixel. Understandably, in some examples, the nine color channels of the multispectral pixel in the hybrid image sensor can be treated as a single large pixel. Figure 6A The pixels shown are arranged as a group of pixels, with multiple groups of pixels arranged in an array. In other examples, in hybrid image sensors, multispectral pixels with nine color channels can be distributed throughout the pixels of the hybrid image sensor to achieve a lower density of multispectral pixels.

[0282] Please refer to Figure 6B The hybrid image sensor includes 16 color-channel multispectral pixels. Each individual event pixel has a relatively large footprint, with each multispectral pixel occupying approximately 1 / 4 * 1 / 4 of the footprint of a single event pixel. Understandably, in some examples, in a hybrid image sensor, the 16 color-channel multispectral pixels can be treated as a single large pixel. Figure 6BThe pixels shown are arranged as a group of pixels, with multiple groups of pixels arranged in an array. In other examples, in hybrid image sensors, 16 color channels of multispectral pixels can be distributed throughout the pixels of the hybrid image sensor to achieve a lower density of multispectral pixels. Figure 6B and Figure 6A In comparison, the difference lies in, Figure 6B The multispectral pixels shown have smaller individual pixel areas, but more multispectral color channels.

[0283] Please refer to Figure 6C In a hybrid image sensor, a group of pixels can include multispectral pixels with 9 color channels and 4 event pixels. The event pixels are larger in size, and the area occupied by each multispectral pixel is 1 / 2 * 1 / 2 of the area occupied by one event pixel. The multispectral pixels are arranged between adjacent event pixels.

[0284] Please refer to Figure 6D In a hybrid image sensor, a group of pixels can include 10 multispectral pixels with color channels and 2 event pixels. The area occupied by each multispectral pixel is 1 / 2 * 1 / 2 of the area occupied by one event pixel, and the 10 multispectral pixels with color channels are arranged around the two event pixels.

[0285] Please refer to Figure 6E In a hybrid image sensor, a group of pixels can include a multispectral pixel with 7 color channels and an event pixel. The area occupied by each multispectral pixel is 1 / 2 * 1 / 2 of the area occupied by the event pixel. The multispectral pixels with 7 color channels are arranged on the two adjacent sides of the event pixel, which can be the right and left, or the top and right sides, without limitation.

[0286] It should be noted that, Figures 6A to 6E The method does not limit the color wavelength perceived in the multispectral pixel. The multispectral pixel can be implemented by changing the material of the CF, or by adding a BG and coating the BG with an AR or IR film.

[0287] It should be noted that, Figures 5A to 6E The pixel arrangement shown is for illustrative purposes only. In practical applications, it can be adapted to meet specific needs. The specific pixel arrangement structure is not limited here.

[0288] In some embodiments, the hybrid camera may have a first working membrane, a second working mode, and a third working mode.

[0289] For example, in the first working mode, the event pixels work independently, and the hybrid camera is used to collect event data and output an event data stream to perform different algorithms according to different scenarios. For example, the hybrid camera can be used for gesture detection, eye tracking, etc., outputting the detected gesture or eye position and providing it to the corresponding algorithm. The algorithm calculates and activates the function of the electronic device corresponding to the gesture, or calculates the eye position and activates the function icon or application of the electronic device corresponding to the eye position.

[0290] For example, in the second operating mode, multispectral pixels work independently, and the hybrid camera is used to acquire multispectral data and output a multispectral data stream to perform different algorithms according to different scenarios. For instance, multispectral detection is performed using the hybrid camera to capture a multispectral image of an object, and the multispectral image is output, for example, stored in a gallery.

[0291] For example, in the third working mode, both event pixels and multispectral pixels operate. The hybrid camera acquires event data and multispectral data, and outputs event data streams and multispectral data streams respectively. The event data stream uses different algorithms depending on the application scenario, and the multispectral data stream uses different algorithms depending on the application scenario. Both calculation results are then output. For instance, when dynamically acquiring color, event pixels are used to identify the dynamically acquired object, while multispectral pixels are used to identify the color. Event pixels can assist in making the color image acquired by multispectral pixels clearer. As another example, when the hybrid camera uses multispectral pixels for spectral detection, event pixels can be used for gesture detection to activate different function options or to enable shooting, etc.

[0292] Example 2: Hybrid camera as a functional auxiliary device.

[0293] Please see Figure 7 , Figure 7 This is a schematic diagram of the framework of the hybrid camera provided in this application as a functional auxiliary device in some embodiments.

[0294] In some embodiments, the hybrid camera can provide functional assistance to a three-channel camera, which can be any one or more of a main camera, a wide-angle camera, and a telephoto camera. The RGB data stream output by the three-channel camera flows through the normal image capture or video recording path, while the hybrid camera, as a functional auxiliary device, provides additional input to the image capture or video recording path.

[0295] It should be noted that functional auxiliary devices refer to devices that provide detection or statistical results to a three-channel camera to assist it in achieving its shooting function.

[0296] In this embodiment, since the hybrid camera is used as a functional auxiliary device, the pixel requirements of the hybrid image sensor are lower, which allows the size of the hybrid image sensor to be designed to be smaller. This is beneficial for the overall miniaturization of the hybrid camera, thereby reducing the size space occupied by the hybrid camera in the electronic device and thus facilitating the miniaturization design of the electronic device.

[0297] For example, the data stream of a hybrid camera may include an event data stream and a multispectral data stream, which occupy different Port Physics Layers (PHYs) or share a PHY output, but are distinguished by different visual channels (VCs) and data types (DTs), which are not limited here.

[0298] The separated event data stream undergoes preprocessing algorithms such as denoising through ESP, and then different algorithms are applied according to different scenarios to provide relevant results in real time. These results are then transmitted to the AE or AF algorithms of the RGB data stream output by the three-channel camera to assist the three-channel camera in exposure and focusing.

[0299] After ESP processing, the event data stream can be applied to scenarios including: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and assisted OIS. These applications can be launched along with the three-channel camera when the user opens the camera app.

[0300] In other embodiments, since the event pixels consume relatively little power, they can remain constantly on for use in non-camera-activated scenarios. For example, applicable scenarios could include moving object detection, gesture recognition, posture recognition, wake-up, and eye tracking, to enable user interaction or to activate specific functions.

[0301] For example, when the hybrid camera is the front-facing camera of an electronic device, the event pixels can detect the user's gaze and track the user's gaze point in real time as a user interaction method, such as eye-tracking unlock, eye-guided browsing, and eye-controlled photography. Furthermore, it can be applied to rear-camera shooting scenarios. For instance, by activating the front-facing hybrid camera and tracking the user's gaze, it can initiate shooting or recording when it detects the user's gaze is focused on the shutter button.

[0302] For example, after the multispectral data stream output by the hybrid image sensor undergoes preprocessing algorithms such as denoising by M-ISP, it can follow different algorithm branches depending on the different application scenarios.

[0303] In some examples, after the multispectral data stream is processed by M-ISP, different algorithms can be applied according to different scenarios to provide relevant results in real time. These results are then transmitted to the AWB algorithm of the RGB data stream to assist the three-channel camera in setting the automatic white balance.

[0304] Among them, after the multispectral data stream is processed by M-ISP, the scenarios in which the AWB algorithm assists the RGB data stream can include: spectral inverse solution and high-dimensional white point estimation.

[0305] In other examples, after multispectral data streams are processed by M-ISP, different algorithms can be applied according to different scenarios, providing auxiliary information for the post-processing algorithms of RGB data streams.

[0306] Among them, scenarios where multispectral data streams, after being processed by M-ISP, provide auxiliary information for post-processing algorithms of RGB data streams can include: ambient light source spectral estimation, True color estimation, and infrared and ultraviolet applications.

[0307] In scenarios involving ambient light spectrum estimation and True color estimation, a hybrid camera can assist a three-channel camera in color algorithms, which helps the three-channel camera capture richer and more realistic colors of objects.

[0308] In infrared and ultraviolet applications, hybrid cameras can assist three-channel cameras in functions such as skin health detection, face unlocking, and sun protection detection, thereby improving the user experience.

[0309] For example, when the color channels of a multispectral pixel include one or more ultraviolet or infrared channels, detection using multispectral pixels can achieve functions such as skin health detection and sun protection detection. When a hybrid camera is used as a front-facing camera, the infrared pixels in the multispectral pixels can be used for face unlocking.

[0310] It should be noted that the hybrid image camera can be used as a complete auxiliary device for a three-channel camera, or it can be used as a partial auxiliary device for a three-channel camera.

[0311] For example, when the hybrid image sensor is arranged in the first configuration, the total area occupied by the event pixels is small. The event pixels can be used as functional auxiliary devices for a three-channel camera to be applied to the following scenarios: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, and auxiliary OIS.

[0312] For example, when the hybrid image sensor is arranged in a second row, the total area occupied by the multispectral pixels is small. The multispectral pixels can be used as functional auxiliary devices for three-channel cameras to be applied to the following scenarios: spectral inversion, high-dimensional white point estimation, ambient light spectrum estimation, true color estimation, infrared and ultraviolet applications.

[0313] Example 3: Hybrid camera as an imaging aid.

[0314] Please see Figure 8 , Figure 8 This is a schematic diagram of the frame of the hybrid camera provided in this application as an imaging aid in some embodiments.

[0315] In some embodiments, a hybrid camera can provide imaging assistance to a three-channel camera, which can be any one or more of a main camera, a wide-angle camera, and a telephoto camera. The image acquired by the hybrid camera is post-processed and fused with the image acquired by the three-channel camera to optimize the shooting effect of the three-channel camera, thereby achieving imaging assistance for the three-channel camera.

[0316] It should be noted that imaging aids refer to devices that provide event images and / or multispectral images to a three-channel camera and fuse them with the original images captured by the three-channel camera to assist the three-channel camera in achieving imaging optimization.

[0317] In this embodiment, the hybrid camera has a high pixel count, approaching that of a three-channel camera, and can assist the three-channel camera in imaging, thereby improving the photo or video quality of the three-channel camera.

[0318] For example, the event data stream undergoes preprocessing algorithms such as denoising through ESP, and then different algorithms are applied according to different scenarios to obtain the corresponding event image. The event image or the data contained in the event image is then provided to the post-processing fusion algorithm of the RGB data stream, thereby optimizing the photo or video taken by the three-channel camera.

[0319] The event data stream undergoes preprocessing algorithms such as denoising through ESP, and then different algorithms are applied according to different scenarios to obtain the corresponding event images. Applicable scenarios include: deblur, video frame interpolation / slow motion, assisted EIS, assisted OIS, Livephoto, depth detection, portrait blurring, etc.

[0320] For example, high-resolution event data streams can assist in deblurring RGB images and frame interpolation in video processing to achieve image deblurring. Furthermore, event data streams can assist OIS algorithms for RGB images and EIS algorithms for videos, resulting in more stable images captured by a three-channel camera. Additionally, event pixels can be used to record continuous motion information in short videos captured by a three-channel camera, enabling frame rate enhancement, deblurring, and assisting the RGB data stream in temporal denoising, registration, and capturing the best moments, thereby helping the three-channel camera capture more dynamic live photos.

[0321] For example, when a hybrid camera is used as a front-facing camera, the event pixel can work with a three-channel camera to calculate binocular depth, or work with laser and TOF emitters to calculate depth map information. The depth map information can then be used to achieve applications such as blurring of portraits or videos.

[0322] For example, the multispectral data stream is processed by the M-ISP denoising preprocessing algorithm, and then different algorithms are applied according to different scenarios to obtain the corresponding multispectral image. The multispectral image or the data contained in the multispectral image is then provided to the RGB data stream postprocessing fusion algorithm to optimize the three-channel camera shooting or video.

[0323] After being processed by M-ISP, the multispectral data stream can be applied to scenarios such as: object spectral decomposition, color layering, tonal style, beautification and makeup, shooting and creating interactive avatars, etc.

[0324] For example, high-resolution multispectral data can be used to image the spectrum of an object, analyze its material and composition, or skin condition, etc. Multispectral images can also be fused with RGB images to achieve color layering or assist RGB images or videos in achieving different tonal styles.

[0325] For example, when a hybrid image sensor is used as a front-facing camera, the multispectral data stream can also be used for beautification or makeup applications. By utilizing the multispectral data stream, different colors of a face or makeup can be detected more accurately, recommending different makeup or beautification effects to the user. As another example, when shooting and creating interactive avatars, multispectral data can add various effects to the avatar, such as overlaying infrared and RGB images to make the avatar more visually appealing, and can also assist in switching the tone style of front-facing RGB images or videos.

[0326] It should be noted that when a hybrid camera is used as an imaging aid for a three-channel camera, because the hybrid image sensor has a high pixel resolution, the hybrid camera can also be used as a functional aid for the three-channel camera.

[0327] It should be noted that the hybrid image camera can be used as an imaging aid for a three-channel camera as a whole, or as a partial imaging aid for a three-channel camera.

[0328] For example, when the hybrid image sensor is arranged in the first row, the total area occupied by the multispectral pixels is relatively large. The multispectral pixels can be used as imaging auxiliary devices and functional auxiliary devices for three-channel cameras, and can be applied to the following scenarios: spectral inversion, high-dimensional white point estimation, ambient light spectrum estimation, true color estimation, infrared and ultraviolet applications, object spectral decomposition, color layering, tonal style, beautification and makeup, shooting and creating interactive avatars.

[0329] For example, when the hybrid image sensor is arranged in the second row, the total area occupied by the event pixels is relatively large. The event pixels can be used as imaging auxiliary devices or functional auxiliary devices for three-channel cameras, and can be applied to the following scenarios: strobe frequency and position detection, strobe mask segmentation, motion region detection / optical flow, moving target detection, deblur, video frame interpolation / slow motion, auxiliary EIS, auxiliary OIS, depth detection, portrait blurring, and Livephoto.

[0330] In this embodiment, the electronic device allows users to enter shooting mode by clicking on a camera app. In shooting mode, the electronic device can automatically identify the shooting scene using a recognition algorithm, enabling the hybrid camera to provide functional or imaging assistance to the three-channel camera. Understandably, the electronic device can also allow users to select a scene; for example, the scene can be selected via scene icons on the camera app's shooting page.

[0331] The following describes an image processing method provided by an embodiment of this application.

[0332] Please see Figure 9 , Figure 9 This is a flowchart illustrating some embodiments of the image processing method provided in this application.

[0333] In some embodiments, the image processing method can be applied to the electronic device provided in the above embodiments. Specifically, the image processing method may include:

[0334] S201, the electronic device acquires first data through a hybrid camera and second data through a three-channel camera.

[0335] The hybrid camera includes a hybrid image sensor, which comprises event pixels and multispectral pixels. Therefore, the first data acquired through hybridization may include event data acquired by the event pixels and / or multispectral data acquired by the multispectral pixels. Since a three-channel camera typically has three color channels (red, green, and blue) or three color channels (red, yellow, and blue), the second data acquired is RGB data or RYB data; RGB data is used as an example in this application.

[0336] Understandably, a hybrid camera may collect event data only, or only multispectral data, or both event data and multispectral data simultaneously.

[0337] S202, the electronic device outputs a first data stream through a hybrid image sensor and an RGB data stream through a three-channel image sensor.

[0338] The first data stream may include at least one of an event data stream and a multispectral data stream.

[0339] S203, the electronic device processes the event data stream in the first data stream through ESP, and the electronic device processes the multispectral data stream in the first data stream through M-ISP.

[0340] Among them, ESP mainly performs noise reduction and other processing on event data streams, while M-ISP mainly performs noise reduction and other processing on multispectral data streams.

[0341] S204, The electronic device performs algorithmic processing on the event data stream and / or multispectral data stream based on the application scenario to obtain relevant data and / or relevant images.

[0342] S205, the electronic device generates an output image based on relevant data and / or relevant images, RGB data streams.

[0343] You can refer to this article. Figure 7 and Figure 8 In some embodiments, the electronic device can generate images based on event data and RGB data.

[0344] In some examples, the hybrid camera acquires event data through event pixels to form an event data stream. After the event data stream is processed by the ESP, different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the image preprocessing algorithm of the ISP. The ISP combines the RGB data stream and the relevant results to perform image processing and generate an image.

[0345] Among them, the event pixel provides functional assistance to the three-channel camera, and its scenarios can include: strobe frequency and position detection, strobe mask segmentation, motion area detection / optical flow, moving target detection, and auxiliary OIS.

[0346] In other examples, the hybrid camera captures event data through event pixels to form an event data stream. After the event data stream is processed by the ESP, different algorithms are applied according to different scenarios to obtain the event image. The RGB data stream is preprocessed by the ISP to form the original RGB image. The ISP then uses an image fusion algorithm to fuse the event image and the original RGB image to generate an image.

[0347] Among them, the event pixel provides imaging assistance for the three-channel camera, and its application scenarios can include: deblur, video frame interpolation / slow motion, auxiliary EIS, auxiliary OIS, depth detection, portrait blurring, Livephoto, etc.

[0348] In other embodiments, the electronic device may generate images based on multispectral data and RGB data.

[0349] In some examples, a hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After the multispectral data stream is processed by the M-ISP, different algorithms are applied according to different scenarios to obtain relevant results. These results are then provided to the ISP's image preprocessing algorithm. The ISP combines the RGB data stream and the relevant results to perform image preprocessing and generate an image.

[0350] Among them, multispectral pixels provide functional assistance to three-channel cameras, and their application scenarios can include: spectral inversion, high-dimensional white point estimation, etc.

[0351] In other examples, a hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After the multispectral data stream is processed by an M-ISP, different algorithms are applied according to different scenarios to obtain relevant results. The RGB data stream is processed by the ISP to obtain the original RGB image after image preprocessing. The ISP uses post-processing algorithms to apply the relevant results to the original RGB image to generate an image.

[0352] Among them, multispectral pixels provide functional assistance to three-channel cameras, and their application scenarios can include: ambient light spectrum estimation, true color estimation, infrared and ultraviolet applications, etc.

[0353] In some other examples, the hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After the multispectral data stream is processed by the M-ISP, different algorithms are applied according to different scenarios to obtain a multispectral image. The RGB data stream is processed by the ISP to obtain the original RGB image. The ISP then uses an image fusion algorithm to fuse the multispectral image and the original RGB image to generate an image.

[0354] Among them, multispectral pixels provide imaging assistance for three-channel cameras, and their application scenarios can include: object spectral decomposition, color layering, tonal style, beautification and makeup, shooting and creating interactive avatars, etc.

[0355] In other embodiments, the hybrid camera can acquire event data and form an event data stream through event pixels, and acquire multispectral pixels and form a multispectral data stream through multispectral pixels. The event data stream can be used for functional assistance or imaging assistance, and the multispectral data stream can be used for functional assistance or imaging assistance. According to the above embodiments, the application of event data stream and multispectral data can be combined to assist the three-channel camera in image preprocessing and / or image postprocessing, enriching the shooting scenarios of the three-channel camera and diversifying the generated images.

[0356] In some embodiments, before the hybrid camera acquires event data, the electronic device can binning adjacent pixels in the event pixel to form a larger pixel, thereby achieving a larger photosensitive area, collecting more light, and thus improving the signal-to-noise ratio and sensitivity of the acquired event data. It should be noted that adjacent pixels in the event pixel are pixels of the same type, all capable of performing the function of an event pixel.

[0357] Please see Figure 10 , Figure 10 This is a flowchart illustrating the image processing method provided in this application in some other embodiments.

[0358] In some embodiments, the hybrid camera can perform image processing independently, without assisting the three-channel camera. Specifically, the image processing method may include:

[0359] S301, the electronic device acquires the first data through a hybrid camera.

[0360] The hybrid camera includes a hybrid image sensor, which comprises event pixels and multispectral pixels. Therefore, the first data acquired through hybridization may include event data acquired by the event pixels and / or multispectral data acquired by the multispectral pixels.

[0361] S302, the electronic device outputs a first data stream through a hybrid image sensor.

[0362] The first data stream may include at least one of an event data stream and a multispectral data stream.

[0363] S303, the electronic device processes the event data stream in the first data stream through ESP, and the electronic device processes the multispectral data stream in the first data stream through M-ISP.

[0364] Among them, ESP mainly performs noise reduction and other processing on event data streams, while M-ISP mainly performs noise reduction and other processing on multispectral data streams.

[0365] S304, The electronic device processes the event data stream and / or multispectral data stream based on the application scenario to obtain relevant data and / or relevant images.

[0366] In some examples, hybrid cameras collect event data through event pixels to form an event data stream. After the event data stream is processed by ESP, different algorithms are applied according to different scenarios to obtain relevant results. Electronic devices then respond based on these results.

[0367] Scenarios where event pixels work independently can include: moving target detection, gesture recognition, wake-up, etc.

[0368] In other examples, hybrid cameras capture event data through event pixels to form an event data stream, which is then processed by ESP to obtain an event image. This event image may contain information about the brightness distribution of objects.

[0369] In some other embodiments, the hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After processing by the M-ISP, the multispectral data stream is used with different algorithms according to different application scenarios to calculate relevant data. This relevant data can include multispectral data, object color information, etc.

[0370] In some other embodiments, the hybrid camera can acquire multispectral data through multispectral pixels to form a multispectral data stream. After the multispectral data stream is processed by M-ISP, a multispectral image can be obtained. The multispectral image may include spectral bands, spectral curve distribution, etc.

[0371] It should be noted that one or more of the modules or units described in this application can be implemented by software, hardware, or a combination of both. When any of the above modules or units are implemented by software, the software exists in the form of computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0372] When the modules or units described in this application are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of the software.

[0373] When the modules or units described in this application are implemented using software, they can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0374] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0375] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0376] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0377] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0378] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0379] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0380] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other, and any combination of features in different embodiments is also within the protection scope of this application. That is to say, the multiple embodiments described above can also be arbitrarily combined according to actual needs.

[0381] It should be noted that all the above figures are exemplary illustrations of this application and do not represent the actual size of the product. Furthermore, the dimensional proportions between the components in the figures are not intended to limit the actual product of this application.

[0382] The above are merely some embodiments and implementation methods of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A camera, characterized in that, The device includes a hybrid image sensor and an optical lens. The hybrid image sensor is located on the image side of the optical lens to receive light incident from the optical lens. The hybrid image sensor includes a first pixel and a second pixel. Each of the first pixels can independently acquire light intensity changes and output an event signal. The second pixel has a color channel number greater than or equal to 7.

2. The camera as described in claim 1, characterized in that, The first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the total area occupied by the second pixel; Alternatively, the first pixel and the second pixel may have a second arrangement, wherein the total area occupied by the first pixel is greater than the total area occupied by the second pixel.

3. The camera as described in claim 2, characterized in that, In the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the first pixel, where M is a positive integer less than or equal to 5; Alternatively, in the second arrangement, the area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5.

4. The camera as described in any one of claims 1 to 3, characterized in that, The camera has a first working mode, a second working mode and a third working mode; In the first operating mode, the first pixel operates independently; In the second working mode, the second pixel operates independently; In the third working mode, both the first pixel and the second pixel are working.

5. An electronic device, characterized in that, Includes the camera as described in any one of claims 1 to 4.

6. The electronic device as claimed in claim 5, characterized in that, The electronic device includes a first camera and a second camera. The first camera is the camera described above, and the second camera is a three-channel camera.

7. The electronic device as claimed in claim 5 or 6, characterized in that, The light-incident ends of the first camera and the second camera are located on different surfaces of the electronic device.

8. The electronic device as claimed in claim 5 or 6, characterized in that, The light-incident ends of the first camera and the second camera are located on the same side surface of the electronic device.

9. The electronic device as claimed in claim 8, characterized in that, There are multiple second cameras, and each of the second cameras is arranged adjacent to the first camera.

10. The electronic device as claimed in claim 9, characterized in that, The distance between each of the multiple second cameras and the first camera is the same.

11. The electronic device as claimed in any one of claims 5 to 10, characterized in that, The first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the total area occupied by the second pixel; Alternatively, the first pixel and the second pixel may have a second arrangement, wherein the total area occupied by the first pixel is greater than the total area occupied by the second pixel.

12. The electronic device as claimed in claim 11, characterized in that, In the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the first pixel, where M is a positive integer less than or equal to 5; Alternatively, in the second arrangement, the total area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5.

13. The electronic device as claimed in any one of claims 5 to 12, characterized in that, The first camera has a first working mode, a second working mode and a third working mode; In the first operating mode, the first pixel operates independently; In the second working mode, the second pixel operates independently; In the third working mode, both the first pixel and the second pixel are working.

14. The electronic device as claimed in any one of claims 11 to 13, characterized in that, The first camera is used to collect first event data, the second camera is used to collect first RGB data, and the electronic device is used to generate a first image based on the first event data and the first RGB data; And / or, the first camera is used to acquire first multispectral data, the second camera is used to acquire second RGB data, and the electronic device is used to generate a second image based on the first multispectral data and the second RGB data.

15. The electronic device as claimed in claim 14, characterized in that, The event pixels and the multispectral pixels are arranged in the first configuration, the first event data is collected in the first scene, and the first multispectral data is collected in the second scene. The first scenario includes any one of the following: flicker frequency and position detection, flicker mask segmentation, motion region detection / optical flow, moving target detection, and auxiliary OIS; The second scenario includes any of the following: spectral inversion, high-dimensional white point estimation, ambient light spectrum estimation, Truecolor estimation, infrared and ultraviolet applications, object spectral decomposition, color layering, tonal style, beautification and makeup, and shooting and creating interactive avatars.

16. The electronic device as claimed in claim 14, characterized in that, The event pixels and the multispectral pixels are arranged in the second configuration, the first event data is collected in the third scenario, and the first multispectral data is collected in the fourth scenario; The third scenario includes any one of the following: flicker frequency and position detection, flicker mask segmentation, motion region detection / optical flow, moving target detection, deblur, video frame interpolation / slow motion, assisted EIS, assisted OIS, depth detection, portrait blurring, and Livephoto; The fourth scenario includes any of the following: spectral inverse solution, high-dimensional white point estimation, ambient light source spectral estimation, Truecolor estimation, and infrared and ultraviolet applications.

17. The electronic device as claimed in any one of claims 5 to 16, characterized in that, The number of first pixels is multiple, and the electronic device is used to merge at least a portion of the multiple first pixels into a pixel unit, so that the pixel unit as a whole can acquire light intensity changes and output an event signal.

18. The electronic device as claimed in any one of claims 5 to 17, characterized in that, The second camera includes any one or more of the main camera, wide-angle camera, and telephoto camera of the electronic device.

19. An image processing method applied to an electronic device, characterized in that, The electronic device includes a first camera, which includes a hybrid image sensor and an optical lens. The hybrid image sensor is located on the image side of the optical lens. The hybrid image sensor includes a first pixel and a second pixel. Each of the first pixels can independently acquire and output light intensity information. The second pixel has a color channel number greater than or equal to 7. The method includes: The electronic device collects first data through the first camera, the first data including event data collected by the first pixel and / or multispectral data collected by the second pixel; The electronic device outputs a first data stream through the first camera, the first data stream including an event data stream and / or a multispectral data stream.

20. The method as described in claim 19, characterized in that, The electronic device further includes a second camera, which is a three-channel camera; the method further includes: The electronic device acquires second data through the second camera and outputs a second data stream; The electronic device generates images based on a first data stream and a second data stream.

21. The method as described in claim 20, characterized in that, The first pixel and the second pixel have a first arrangement, wherein the total area occupied by the first pixel is smaller than the area occupied by the second pixel; Alternatively, the first pixel and the second pixel may have a second arrangement, wherein the total area occupied by the first pixel is greater than the area occupied by the second pixel.

22. The method as described in claim 21, characterized in that, In the first arrangement, the total area occupied by the second pixel is M times the total area occupied by the second pixel, where M is a positive integer less than or equal to 5; Alternatively, in the second arrangement, the total area occupied by the first pixel is N times the total area occupied by the second pixel, where N is a positive integer less than or equal to 5.

23. The method according to any one of claims 19 to 22, characterized in that, The first camera has a first working mode, a second working mode and a third working mode; In the first operating mode, the first pixel operates independently; In the second working mode, the second pixel operates independently; In the third working mode, both the first pixel and the second pixel are working.

24. The method according to any one of claims 21 to 23, characterized in that, The first data stream includes the event data stream; The electronic device generates images based on a first data stream and a second data stream, specifically including: The electronic device generates a first image based on the event data stream and the second data stream.

25. The method according to any one of claims 21 to 23, characterized in that, The first data stream includes the multispectral data stream; The electronic device generates images based on a first data stream and a second data stream, specifically including: The electronic device generates a second image based on the multispectral data stream and the second data stream.

26. The method according to any one of claims 21 to 23, characterized in that, The first data stream includes the event data stream and the multispectral data stream; The electronic device generates images based on a first data stream and a second data stream, specifically including: The electronic device generates a third image based on the event data stream, the multispectral data stream, and the second data stream.

27. The method according to any one of claims 24 to 26, characterized in that, The first pixel and the second pixel are arranged in the first configuration, the event data is collected in the first scene, and the multispectral data is collected in the second scene. The first scenario includes any one of the following: flicker frequency and position detection, flicker mask segmentation, motion region detection / optical flow, moving target detection, and auxiliary OIS; The second scenario includes any of the following: spectral inversion, high-dimensional white point estimation, ambient light spectrum estimation, Truecolor estimation, infrared and ultraviolet applications, object spectral decomposition, color layering, tonal style, beautification and makeup, and shooting and creating interactive avatars.

28. The method according to any one of claims 24 to 26, characterized in that, The first pixel and the second pixel are arranged in the second configuration, the event data is collected in the third scene, and the multispectral data is collected in the fourth scene. The third scenario includes any one of the following: flicker frequency and position detection, flicker mask segmentation, motion region detection / optical flow, moving target detection, deblur, video frame interpolation / slow motion, assisted EIS, assisted OIS, depth detection, portrait blurring, and Livephoto; The fourth scenario includes any of the following: spectral inverse solution, high-dimensional white point estimation, ambient light source spectral estimation, Truecolor estimation, and infrared and ultraviolet applications.

29. The method according to any one of claims 19 to 28, characterized in that, The number of the first pixels is multiple, and the electronic device collects first data through the first camera, specifically including: The electronic device merges a portion of the multiple first pixels into a pixel unit, and collects event data as a whole through the pixel unit.

30. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 19 to 29.

31. A computer program product, characterized in that, The computer program product stores a program or instructions that, when executed, implement the method as described in any one of claims 19 to 29.

Images