Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and relates to the technical field of terminal equipment, and the method can optimize the image quality of image data collected by a first pixel unit related to an image sensor through brightness change data collected by a second pixel unit related to an event camera in a camera, thereby improving the image quality. And the quality of the shot image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of terminal devices, and in particular to an image processing method and electronic device. Background Art

[0002] With the development of communication technology, the computing power and hardware capabilities of electronic devices have been improved, and the shooting functions of electronic devices have become more and more powerful.

[0003] However, when there is a physical barrier (such as a display screen, stains, or fog, etc.) in the lighting direction of the camera of an electronic device, the image captured by the electronic device through the camera may have distortion problems such as diffraction, glare, blur, low contrast, and high noise in dark scenes, resulting in poor image quality. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an image processing method and electronic device. In the method, the image quality of the image data collected by the first pixel unit related to the image sensor can be optimized through the brightness change data collected by the second pixel unit related to the event camera in the camera, so that the quality of the captured image is improved.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes at least one camera, and the at least one camera includes a first pixel unit related to an image sensor and a second pixel unit related to an event camera; wherein the first pixel unit is used to collect first image data, and the second pixel unit is used to collect first brightness change data related to the first image data. The method includes: in response to a received first user operation, obtaining, through the at least one camera, second image data related to the first image data, and second brightness change data related to the second image data, wherein the first user operation indicates taking an image or video; and fusing the second image data and the second brightness change data to obtain a target image or target video.

[0006] Wherein, the image sensor is a color image sensor.

[0007] For example, the image sensor may be a color image sensor with RGGB as a color filter array, and the first pixel unit included in the at least one camera may be an RGB pixel unit.

[0008] The second pixel unit may be an event camera (EVS) pixel unit.

[0009] The first brightness change data collected by the second pixel unit may be used to reflect the brightness change of the image data collected by one or more pixel units.

[0010] For example, the first image data may be an RGB image, and the first brightness change data may be expressed in the form of an event stream, which is not specifically limited.

[0011] In an embodiment of the present application, two types of pixel units are integrated in at least one camera, namely a first pixel unit associated with an image sensor, and a second pixel unit associated with an event camera. The first pixel unit can collect image data, and the second pixel unit can collect brightness change data related to the image data. Then, the image data and the brightness change data related to the image data can be acquired through the first pixel unit and the second pixel unit on the at least one camera, and the two types of data can be fused. The brightness change data can be used to improve distortion problems of the image data such as diffraction, glare, blur, low contrast, and high noise in dark scenes, so that the image quality of the fused image or video is higher.

[0012] According to the first aspect, the electronic device further includes a display screen, the at least one camera is located below the display screen, and the light received by the at least one camera is the light transmitted through the display screen.

[0013] The present application can be applied to the under-screen camera scenario, and the at least one camera can be arranged below the display screen of the electronic device. In this way, there is a physical barrier in the lighting direction of at least one camera, which is the display screen here, thereby solving the many distortion problems such as diffraction, glare, blur, low contrast, and high noise in dark scenes existing in the images taken by the under-screen camera (only having RGB pixel units, not having EVS pixel units) in the related technology. The images taken by the under-screen camera of the present application do not have the above-mentioned distortion problems due to the obstruction of the display screen. The distortion problem can be effectively improved and the image quality is better.

[0014] According to the first aspect, or any implementation manner of the first aspect above, the at least one camera includes a composite sensor, the composite sensor includes the first pixel unit and the second pixel unit, and the obtaining of second image data related to the first image data and second brightness change data related to the second image data through the at least one camera includes: obtaining mixed data collected by the at least one camera through the composite sensor, wherein the mixed data includes the first image data and the first brightness change data; extracting the first image data and the first brightness change data from the mixed data respectively; generating second image data based on the first image data; wherein the second brightness change data is related to the first brightness change data.

[0015] In this embodiment, the first pixel unit and the second pixel unit can be integrated into one sensor to obtain the composite sensor. By extracting two types of data from the mixed data (e.g., mixed data stream) collected by the camera using the composite sensor, the first image data collected by the first pixel unit and the first brightness change data collected by the second pixel unit can be obtained. Optionally, the second pixel unit may occupy the layout position originally belonging to one or more first pixel units in the composite sensor. In this way, the above-mentioned first image data is not complete image data, and some pixels are missing. Therefore, the electronic device can generate complete second image data based on the first image data. The second brightness change data can be the same as the first brightness change data, or the second brightness change data is the brightness change data obtained by interpolating the first brightness change data.

[0016] By setting a composite sensor in the front camera, wherein the composite sensor may include not only a first pixel unit to collect image data, but also an EVS pixel unit (e.g., an event unit) to collect pixel-level brightness change data, two types of data are collected on one hardware (the composite sensor here), and not only the brightness change data can be used to optimize the image data collected by the first pixel unit to improve the image quality of the obtained target image or target video, but also the layout space of the camera sensor can be saved in hardware, and the cost can be saved.

[0017] According to the first aspect, or any implementation manner of the first aspect, the second pixel units arranged on the composite sensor are sparser than the first pixel units arranged on the composite sensor.

[0018] For example, the size of the array formed by the second pixel units may be smaller than the size of the array formed by the first pixel units, so that the layout of the second pixel units on the composite sensor is sparser than the layout of the first pixel units on the composite sensor. This can improve the image quality while reducing the number of pixel units related to the event camera on the composite sensor to save costs.

[0019] According to the first aspect, or any implementation of the first aspect, the first pixel unit is arranged around the second pixel unit.

[0020] In this embodiment, by arranging the first pixel unit around the second pixel unit, the brightness change data collected by the second pixel unit can reflect the brightness change of the image data collected by the first pixel units located around it, so as to improve the image quality of the image collected by the surrounding first pixel units.

[0021] According to the first aspect, or any implementation manner of the first aspect above, the at least one camera includes a first camera and a second camera, wherein the first camera includes the image sensor, the image sensor includes a plurality of the first pixel units, and the second camera includes the event camera, the event camera includes a plurality of the second pixel units; the acquiring of the second image data related to the first image data and the second brightness change data related to the second image data through the at least one camera includes: collecting a plurality of first image data through the plurality of first pixel units of the first camera to obtain the second image data; collecting a plurality of the first brightness change data through the plurality of second pixel units of the second camera to obtain the second brightness change data (for example, an event stream), wherein a single first brightness change data may be event data.

[0022] In this embodiment, two sensors, namely an image sensor and an event camera (also a sensor), are independently set in the two cameras of the electronic device. The two sensors are two independent hardware and are physically isolated. Therefore, the data collected by each is independent of each other and there is no interference with each other. The image data collected by the image sensor is affected by the setting position under the screen and has the problem of lighting interference. However, the event camera and the image sensor are set and operated independently of each other, so that the brightness change data output by the event camera is not affected by the image sensor, so that it can have accurate and stable pixel-level brightness change data output. Then the electronic device can use the accurate and stable pixel-level brightness change data output by the event camera to fuse with the image data collected by the image sensor, so that the image quality of the fused target image or target video can be better.

[0023] According to the first aspect, or any implementation of the first aspect above, the size of the array formed by the first pixel units on the sensor to which they belong is related to the resolution of the target image or the video frame in the target video.

[0024] In one implementation, the sensor to which the first pixel unit belongs may be a hybrid sensor or an image sensor. The resolution of the target image or video frame in the target video captured by the electronic device of the present application through the at least one camera may be related to the size of the array formed by the first pixel unit on the sensor to which it belongs.

[0025] According to the first aspect, or any implementation of the first aspect above, before responding to the received first user operation, the method also includes: in response to the received second user operation, starting a switch for enhancing the image quality of the image or video captured by the at least one camera based on the second pixel unit.

[0026] Among them, the electronic device can be set with a switch for whether to increase the image quality of the image or video taken by the electronic device. When the switch is in the activated state through user operation, for example, there are stains in the lighting direction of the camera, or there is fog in the shooting environment, the user can operate to turn on the switch of the electronic device, so that the event camera can be used to perform lighting enhancement and other processing on the image through the above method, so that the captured image is not affected by physical barriers such as stains or fog on the camera, and also has higher image quality.

[0027] According to the first aspect, or any implementation method of the first aspect above, the second image data and the second brightness change data are fused to obtain a target image or a target video, including: extracting a first image feature from the second image data; extracting a first brightness feature from the second brightness change data; and fusing the first image feature and the first brightness feature to obtain a target image or a target video.

[0028] In an embodiment of the present application, the electronic device can perform feature-level fusion on the image data and the brightness change data, where the first brightness feature is a feature representing the brightness change, thereby capturing a higher quality image or video.

[0029] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: adjusting at least one pixel point in the second image data according to the second brightness change data to obtain the target image or the target video.

[0030] In an embodiment of the present application, the electronic device can perform pixel-level fusion of image data and brightness change data to adjust parameters such as the color value of at least one pixel in the image data in combination with the brightness change data, thereby improving image contrast, noise, etc.

[0031] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: aligning the second image data and the second brightness change data; fusing the aligned second image data and the second brightness change data to obtain a target image or a target video.

[0032] In this embodiment, when fusing image data and brightness change data, the electronic device may align the two types of data according to timestamps and perform image registration to achieve alignment of the two types of data, and then fuse the aligned data to improve the accuracy of image processing.

[0033] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: based on the second brightness change data, denoising the second image data to obtain a target image or a target video.

[0034] In this embodiment, when the image data and the brightness change data are merged, the electronic device may perform denoising on the image data based on the brightness change data, so as to reduce noise points in the captured image or video.

[0035] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: based on the second brightness change data, dehazing the second image data to obtain a target image or a target video.

[0036] In this embodiment, when the image data and the brightness change data are merged, the electronic device may perform dehazing processing on the image data based on the brightness change data to reduce the haziness of the captured image or video.

[0037] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: based on the second brightness change data, deblurring the second image data to obtain a target image or a target video.

[0038] In this embodiment, when the image data and the brightness change data are merged, the electronic device may perform a deblurring process on the image data based on the brightness change data to improve the clarity of the captured image or video.

[0039] According to the first aspect, or any implementation method of the first aspect above, the fusing of the second image data and the second brightness change data to obtain a target image or a target video includes: based on the second brightness change data, performing contrast enhancement processing on the second image data to obtain a target image or a target video.

[0040] In this embodiment, when the image data and the brightness change data are merged, the electronic device may perform contrast enhancement processing on the image data based on the brightness change data to improve the contrast of the captured image or video.

[0041] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: at least one camera, the at least one camera includes a first pixel unit associated with an image sensor and a second pixel unit associated with an event camera; wherein the first pixel unit is used to collect first image data, and the second pixel unit is used to collect first brightness change data related to the first image data, and the electronic device includes: an acquisition module, which is used to respond to a received first user operation, and obtain second image data related to the first image data and second brightness change data related to the second image data through the at least one camera, wherein the first user operation indicates to take an image or video, wherein the first user operation indicates to take an image or video; a fusion module, which is used to fuse the second image data and the second brightness change data to obtain a target image or target video.

[0042] According to the second aspect, the electronic device further includes a display screen, the at least one camera is located below the display screen, and the light received by the at least one camera is the light transmitted through the display screen.

[0043] According to the second aspect, or any implementation manner of the second aspect above, the at least one camera includes a composite sensor, the composite sensor includes the first pixel unit and the second pixel unit, and the acquisition module is specifically used to: obtain mixed data collected by the at least one camera through the composite sensor, wherein the mixed data includes the first image data and the first brightness change data; extract the first image data and the first brightness change data from the mixed data respectively; generate second image data based on the first image data; wherein the second brightness change data is related to the first brightness change data.

[0044] According to the second aspect, or any implementation of the second aspect, the second pixel units arranged on the composite sensor are sparser than the first pixel units arranged on the composite sensor.

[0045] According to the second aspect, or any implementation of the second aspect, the first pixel unit is arranged around the second pixel unit.

[0046] According to the second aspect, or any implementation of the second aspect above, the at least one camera includes a first camera and a second camera, wherein the first camera includes the image sensor, the image sensor includes a plurality of the first pixel units, and the second camera includes the event camera, the event camera includes a plurality of the second pixel units; the acquisition module is specifically used to: collect a plurality of first image data through the plurality of first pixel units of the first camera to obtain second image data; collect a plurality of first brightness change data through the plurality of second pixel units of the second camera to obtain second brightness change data.

[0047] According to the second aspect, or any implementation of the second aspect above, the size of the array formed by the first pixel units on the sensor to which they belong is related to the resolution of the target image or the video frame in the target video.

[0048] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a starting module, used to start a switch for enhancing the image quality of the image or video captured by the at least one camera based on the second pixel unit in response to a received second user operation.

[0049] According to the second aspect, or any implementation method of the second aspect above, the fusion module is specifically used to: extract the first image feature from the second image data; extract the first brightness feature from the second brightness change data; and fuse the first image feature with the first brightness feature to obtain a target image or a target video.

[0050] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to: adjust at least one pixel point in the second image data according to the second brightness change data to obtain a target image or a target video.

[0051] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to: align the second image data and the second brightness change data; and fuse the aligned second image data and the second brightness change data to obtain a target image or a target video.

[0052] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to denoise the second image data based on the second brightness change data to obtain a target image or a target video.

[0053] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to dehaze the second image data based on the second brightness change data to obtain a target image or a target video.

[0054] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to deblur the second image data based on the second brightness change data to obtain a target image or a target video.

[0055] According to the second aspect, or any implementation of the second aspect above, the fusion module is specifically used to perform contrast enhancement processing on the second image data based on the second brightness change data to obtain a target image or a target video.

[0056] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0057] In a third aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method in the first aspect and any one of the implementations of the first aspect.

[0058] The technical effects corresponding to the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable medium for storing a computer program, which, when executed on an electronic device, enables the electronic device to execute the method in the first aspect and any one of the implementations of the first aspect.

[0060] The technical effects corresponding to the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0061] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the method in the first aspect and any one of the implementations of the first aspect.

[0062] The technical effects corresponding to the fifth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0063] In a sixth aspect, an embodiment of the present application provides a chip. The chip includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a first electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the first electronic device executes the method in the first aspect and any one of the implementations of the first aspect.

[0064] The technical effects corresponding to the sixth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 One of the structural schematic diagrams of an electronic device is shown as an example;

[0067] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;

[0068] Figure 3 A schematic diagram of a mobile phone in the prior art is shown as an example;

[0069] Figure 4a A schematic diagram of a mobile phone is shown as an example;

[0070] Figure 4b A schematic diagram of a mobile phone is shown as an example;

[0071] Figure 5a A schematic diagram of a sensor shown as an example;

[0072] Figure 5b A schematic diagram of a sensor shown as an example;

[0073] Figure 5c A schematic diagram of a sensor shown as an example;

[0074] Figure 5d is a schematic diagram of an exemplary sensor;

[0075] Figure 6a is a schematic diagram of an image processing process shown as an example;

[0076] Figure 6b is a schematic diagram of an image processing process shown as an example;

[0077] Figure 7a is a schematic diagram of an exemplary sensor;

[0078] Figure 7b is a schematic diagram of an exemplary sensor;

[0079] Figure 8 is a schematic diagram of an image processing process shown as an example;

[0080] Fig. 9 A schematic diagram of the structure of the device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0083] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0084] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0085] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0086] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. It should be understood that Figure 1 The electronic device 100 shown is only an example of an electronic device. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a cellular phone, a tablet computer (pad), a wearable device, or an IoT device, etc., which has a front camera, and this application does not limit this. It should be noted that the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0087] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0088] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0089] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0090] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0091] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0092] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0093] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0094] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0095] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0096] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0097] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0098] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0099] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0100] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0101] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0102] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0103] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0104] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0105] The camera 193 is used to capture still images or videos. The object is projected onto the photosensitive element by generating an optical image through the lens.

[0106] The camera 193 may be located below the display screen 194 to serve as an under-screen camera, so that the display screen 194 of the electronic device 100 is implemented as a full screen.

[0107] In some embodiments, the camera 193 may include an event sensor (EVS), also referred to as an Event-based Camera, or an Event Camera, and a red, green, and blue sensor (also referred to as a traditional camera, or an RGB sensor). The EVS includes an EVS pixel unit, and the RGB sensor includes an RGB pixel unit.

[0108] In some embodiments, the camera 193 may include a composite image sensor, which may include not only EVS pixel units but also RGB pixel units.

[0109] In some embodiments, the image signal processor (ISP) in the processor 110 can process the data collected by the camera 103 (which will be described in detail in the following embodiments) to obtain a target image with brightness compensated. The target image does not have blurring or distortion problems due to the camera 193 being an under-screen camera, and the image quality is good.

[0110] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0111] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 enables the electronic device 100 to execute the method in the embodiment of the present application by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0112] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0113] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0114] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0115] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0116] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0117] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0118] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0119] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0120] The layered architecture of the electronic device 100 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0121] The application layer can include a series of application packages.

[0122] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0123] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0124] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0125] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0126] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0127] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0128] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0129] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0130] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0131] The system library and runtime layer include the system library and Android Runtime. The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The 3D graphics library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the function functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0132] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0133] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0134] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0135] A 2D graphics engine is a drawing engine for 2D drawings.

[0136] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0137] Understandably, Figure 2 The components included in the system framework layer, system library and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently.

[0138] Under Display Camera (UDC) is an emerging camera technology that aims to hide the front camera under the display screen of electronic devices such as mobile phones, realizing the full-screen design of electronic devices without bangs, water drops, holes and other design elements that affect the integrity of the screen.

[0139] Traditional front cameras are usually located at the top center or edge of the display, so a certain amount of space needs to be left on the screen to accommodate the camera module, thus forming a notch screen, a water drop screen or a hole screen, etc. The under-screen camera mainly uses the light transmittance of the OLED screen as a display to achieve the function of front-facing shooting.

[0140] Figure 3 The figure is a schematic diagram of a mobile phone using an under-screen camera.

[0141] Please refer to Figure 3 ,like Figure 3 As shown in (1), the display screen 201 of the mobile phone 200 is an OLED screen, and a front camera 2030 is arranged below the display screen 201. Since the front camera 2030, as an under-screen camera, is invisible to the user on the display screen 201 of the mobile phone 200, in order to show the position of the front camera 2030 below the display screen 201, the front camera 2030 is shown as a dotted circle. Figure 3 (2) Figure 3 (1) is a cross-sectional schematic diagram of a mobile phone, the front camera 2030 may include Figure 3 (2) RGB image sensor (RGB Sensor) 203, as shown in Figure 3 As shown in (2), the display screen 201 is composed of a plurality of light-emitting elements 202 , and the RGB image sensor 203 is located below the light-emitting elements 202 of the display screen 201 .

[0142] Since there is a light-emitting element 202 in the full-screen display screen 201, and the light-emitting element 202 is placed above the RGB image sensor 203, the RGB image sensor of the under-screen camera has a natural physical barrier, which affects the lighting of the RGB image sensor 203 when shooting with the front camera 2030, resulting in poor light transmittance and poor imaging quality; in addition, complex ambient light and the light emitted by the light-emitting element 202 of the display screen 201 itself will also interfere with the light capture of the RGB image sensor 203, thereby causing diffraction and glare problems.

[0143] Therefore, compared with traditional front cameras installed on non-full-screen displays such as notch screens and water drop screens, the image quality of images or videos taken by the under-screen camera will be attenuated to a certain extent. For example, there will be distortion problems such as diffraction, glare, blur, low contrast, and high noise in dark scenes, which reduce the image shooting quality.

[0144] To this end, the present application provides an electronic device and an image processing method, which can assist the front camera in imaging through an event sensor (EVS) to improve the imaging effect of the under-screen camera.

[0145] Of course, in some embodiments, the front camera may not be an under-screen camera. When the sensor of the camera of the electronic device (not limited to the front camera) is slightly contaminated or has obstructions such as dust or water vapor, the method of the present application is also applicable.

[0146] An event camera is a special type of visual sensor, also referred to as an EVS image sensor below. It does not output image frames at a constant rate, but rather outputs a data stream related to brightness changes at the pixel level. Because it is good at capturing brightness changes, it can also output valid data in dark and bright scenes, making it have the advantages of high frame rate, low latency, high dynamics, low power consumption, less data redundancy, and low transmission bandwidth. For example, its frame rate can be greater than 5KFPS (frames per second).

[0147] The following takes the electronic device as a mobile phone as an example, combined with Example 1 and Example 2, to introduce the technical solution of the present application. When the electronic device is other types of electronic devices, such as a tablet computer or a personal computer, the implementation principle is the same and will not be repeated here.

[0148] Example 1

[0149] In this example 1, we will combine Figure 4a , Figure 5a , Figure 6a , Figure 7a as well as Figure 8 To introduce the technical solution of this application.

[0150] Figure 4a The front schematic diagram and the cross-sectional schematic diagram of the mobile phone of the present application are exemplarily shown.

[0151] like Figure 4a As shown in (1), a front camera is arranged below the display screen 301 (full screen) of the mobile phone 300. The front camera may include an RGB image sensor 303 and an EVS image sensor 304. Figure 4a In (1), the positions of the RGB image sensor 303 and the EVS image sensor 304 located below the display screen 301 are indicated by dotted circles. In actual applications, the RGB image sensor 303 and the EVS image sensor 304 cannot be seen from the display screen 301 of the mobile phone 300.

[0152] In other embodiments, the number of the RGB image sensor 303 and the number of the EVS image sensor 304 in the front camera may also be multiple. In addition, the present application does not limit whether the number of the two image sensors is the same.

[0153] Figure 4a (2) Figure 4a (1) is a schematic cross-sectional view of the mobile phone 300 in the width direction indicated by the double arrows.

[0154] like Figure 4a As shown in (2), the display screen 301 is composed of a plurality of light-emitting elements 302, and the RGB image sensor 303 and the EVS image sensor 304 are both located below the light-emitting elements 302 of the display screen 301, so that the display screen 301 can be used as a full screen, and the RGB image sensor 303 and the EVS image sensor 304 are both used as under-screen cameras.

[0155] In other embodiments, the RGB image sensor 303 in Example 1 may also be replaced by other types of color image sensors capable of capturing color information of an image.

[0156] For example, the RGB image sensor 303 is described by taking RGGB as the color filter array. In other embodiments, the RGB image sensor 303 can also be replaced by a color image sensor including other color filter arrays. For example, the color filter array on the color image sensor can be a color filter array (CFA), etc., and the principle is the same, which will not be repeated here.

[0157] Figure 5a (1) Figure 5a (2) respectively show Figure 4a The structural schematic diagram of the RGB image sensor 303 and the EVS image sensor 304 is shown.

[0158] like Figure 5a As shown in (1), the RGB image sensor 303 may be composed of a plurality of RGB pixel units (also referred to as RGB sensing pixel points), where Figure 5a As shown in (1), the multiple RGB pixel units of the RGB image sensor 303 may include a pixel unit of a red channel (represented by R) 3031, a pixel unit of a green channel (represented by G) 3032, and a pixel unit of a blue channel (represented by B) 3033.

[0159] like Figure 5a As shown in (1), the multiple RGB pixel units are arranged above the RGB image sensor 303 in a Bayer array (also expressed as an RGGB array).

[0160] The Bayer array consists of pixel units for three color channels: red (R), green (G), and blue (B), of which there are two green channels. Each pixel unit corresponds to information of only one color channel, and the Bayer array is usually composed of a 4x4 cell arrangement similar to the following pattern:

[0161]

[0162] In this pattern, the red photosensitive element (R) is located at the red channel position, the blue photosensitive element (B) is located at the blue channel position, and the green photosensitive element (G) is located at the green channel position. The purpose of this arrangement is to approximate the sensitivity of the human eye to different colors to provide a color image.

[0163] like Figure 5a As shown in (1), the multiple RGB pixel units of the RGB image sensor 303 are a Bayer array of M1*N1, wherein M1=N1, M1 represents the number of pixel units in the width direction of the array, which is 8 here, and N1 represents the number of pixel units in the height direction of the array, which is 8 here.

[0164] In other embodiments, the RGB image sensor 303 may also arrange pixel units in other arrays, for example, the array may also be an RGBW (red, green, blue, white), RGBE (with enhanced channels) or other arrays, and the present application does not impose any limitation on this.

[0165] When the array of pixel units of the RGB image sensor 303 is changed, the size of the RGB image sensor 303 may also be different, and the present application is not limited to M1=N1.

[0166] The RGB image sensor 303 whose pixel units form an array of size M1*N1 can be used to collect RGB images with a resolution of M1*N1, so that the resolution of the image output by the camera having the RGB image sensor 303 is M1*N1.

[0167] However, since the RGB image sensor 303 is an under-screen camera, using it alone for shooting may cause the light of the captured image to be affected. Therefore, the under-screen camera of the present application also includes the following: Figure 4a The EVS image sensor 304 shown is used to optimize the illumination of the image captured by the RGB image sensor 303 to reduce the use of the RGB image sensor 303 (eg Figure 3 The image quality is optimized by using a solution that can capture images without causing image diffraction, glare, blur, low contrast, and high noise in dark scenes.

[0168] like Figure 5a As shown in (2), the EVS image sensor 304 may be composed of a plurality of EVS pixel units (also referred to as EVS sensing pixel points), wherein: Figure 5aAs shown in (2), the multiple EVS pixel units of the EVS image sensor 304 are multiple event units 3041, each of which may correspond to multiple RGB pixel units in the RGB image sensor 303, or in other words, one event unit may correspond to multiple pixel points (corresponding to the above-mentioned multiple RGB pixel units) in the image captured by the front camera to reflect the brightness changes of the multiple pixel points. The multiple event units 3041 are arranged above the EVS image sensor 304 in an array of M2*N2.

[0169] Here, M2=N2=4, and the present application does not limit whether the width (M2) and height (N2) of the array are the same.

[0170] In some embodiments, compared to Figure 5a (1) and Figure 5a (2) The layout of the RGB pixel units on the RGB image sensor 303 is relatively dense, while the layout of the array composed of the EVS pixel units on the EVS image sensor 304 is relatively sparse.

[0171] For example, Figure 5a The event data output by the event unit 3041 in the first row and first column shown in (2) can be used to reflect the following: Figure 5a (1) shows the brightness change of the image area (resolution is 2*2) corresponding to the 4 RGB pixel units in the upper left corner. Figure 5a The event data output by other event units shown in (2) can also reflect Figure 5a (1) The brightness changes of the corresponding image area.

[0172] Among them, the 4 RGB pixel units in the upper left corner are Figure 5a The RGB image sensor 303 shown in (1) has four RGB pixel units arranged in the first row and first column, the first row and second column, the second row and first column, and the second row and first column.

[0173] Although the size of the array of RGB pixel units on the RGB image sensor 303 is different from the size of the array of EVS pixel units (also referred to as event units) on the EVS image sensor 304, they are used to reflect the color data and brightness change data of the same image area under the same shooting scene. In this way, the RGB image sensor 303 can capture an RGB image with a resolution of M1*N1 (a frame of image in the first RGB image stream described below), such as Figure 5a The event data output by all event units of the EVS image sensor 304 shown in (2) can jointly reflect the brightness change data of each pixel area in the RGB image with a resolution of M1*N1.

[0174] In addition, the present application does not restrict whether the size of the array formed by the RGB pixel units of the RGB image sensor 303 (M1*N1) is the same as the size of the array formed by the EVS pixel units of the EVS image sensor 304 (M2*N2). Generally, the two are different.

[0175] In addition, the present application does not limit the array structure of the multiple event units 3041.

[0176] also, Figure 5a The shape and layout of each event unit 304 in the EVS image sensor 304 shown in (2) are not used to limit the EVS image sensor 304, and the shape and layout can be flexibly configured as needed.

[0177] The EVS image sensor 304 (i.e., event camera) of the present application is a special type of visual sensor, which is different from a traditional frame camera, for example, the frame camera is an RGB image sensor 303. The EVS pixel unit of the event camera is not based on Figure 5a (1) is arranged in a pixel array, but Figure 5a (2) are arranged in the form of event units 3041. In some embodiments, these event units 3041 can generally adopt pixel-level event sensing, and each event unit 3041 can independently sense photon events.

[0178] When an event unit (corresponding to one or more pixels in the image taken by the mobile phone 300) detects a photon event, it will generate a corresponding event to record the location where the event occurred (ie, the coordinates of the event unit) and the timestamp.

[0179] Specifically, when the light intensity received by the event unit 3041 changes, the event unit 3041 detects a photon event to generate and output a corresponding event.

[0180] For example, when the brightness received by the event unit 3041 increases and the increased brightness exceeds the specified threshold, it means that the event unit 3041 detects a photon event and outputs a positive event (indicated by "+"); on the contrary, when the brightness received by the event unit 3041 decreases and the decreased brightness exceeds the specified threshold, it means that the event unit 3041 detects a photon event, and the event unit 3041 can output a negative event (indicated by "-").

[0181] Regardless of whether it is a positive event or a negative event, the event output by the event unit 3041 may include the following information: coordinates (x, y), timestamp, and polarity.

[0182] The coordinate (x, y) is the position of one or more pixels represented by the event unit 3041;

[0183] The timestamp is the time when the event unit 3041 detects the photon event, that is, the time when the brightness change is detected;

[0184] The polarity is positive or negative, where a positive polarity indicates an increase in brightness and a negative polarity indicates a decrease in brightness.

[0185] Therefore, the output result of the EVS image sensor 304 is a stream containing events, and each event in the stream contains the coordinate position, time stamp, and polarity information of the event unit.

[0186] The EVS image sensor 304 does not capture the image of the entire scene at fixed time intervals, but instead outputs pixel-level event data (including timestamp, coordinates, and polarity) to record brightness change information of a scene, so that its output result can express information about when and at which position the pixel undergoes a brightness change, whether the brightness increases or decreases.

[0187] Therefore, the event camera is highly sensitive to rapid scene changes and light changes. In addition, the event camera has low latency and can capture details in high-speed motion without motion blur. Figure 4a and Figure 5a The EVS image sensor 304 shown captures the brightness change information of the entire shooting scene to optimize the RGB image whose light is affected and collected by the RGB image sensor 303.

[0188] Combine the following Figure 6a , Figure 7a and Figure 8 To introduce the mobile phone 300 of this application, Figure 4a and Figure 5a The RGB image sensor 303 and the EVS image sensor 304 are used to perform processing in an image or video shooting scenario.

[0189] like Figure 6a As shown, the process may include the following steps:

[0190] S101a, the RGB image sensor 303 of the mobile phone 300 collects a first RGB image stream.

[0191] For example, the first RGB image stream may include multiple frames of RGB images continuously captured by the RGB image sensor 303 for the shooting scene, the resolution of each frame of RGB image is M1*N1, and each frame of RGB image may have a timestamp to indicate the shooting time. The pixel points of a frame of RGB image may include RGB values.

[0192] The present application does not limit the specific implementation of S101a, and the image acquisition process of the RGB image sensor in the prior art can be adopted, which will not be described in detail here.

[0193] S101b, the EVS image sensor 304 of the mobile phone 300 collects a first data stream related to brightness changes of pixels.

[0194] For each pixel point (or a pixel area composed of multiple pixel points) of each frame of RGB image in the first RGB image stream in S101a, the EVS image sensor 304 can reflect the brightness change of the pixel point through the event data output by the event unit corresponding to the pixel point (or the corresponding pixel area).

[0195] In this embodiment, Figure 5a (2) The event data output by an event unit 3041 can be used to reflect Figure 5a The RGB image sensor 303 shown in (1) collects brightness changes of a 2*2 pixel area at a corresponding position (related to the coordinates of the event unit 3041) in a frame of RGB image in the first RGB image stream.

[0196] It should be understood that due to the different layouts of the event units on the composite image sensor 403 and the different layouts of the RGB pixel units, the size of the pixel area corresponding to the brightness change reflected by the event data output by a single event unit may be different, and is not limited to the 2*2 pixel area in the above example.

[0197] The first data stream may be an event stream output by the EVS image sensor 304 regarding the captured scene. The event stream may include two independent event streams: a positive event stream and a negative event stream, so as to record the occurrence of photon events.

[0198] The positive event flow corresponds to an event with increased brightness, which can also be understood as the arrival of a photon event causing an increase in brightness. The event unit that outputs a positive event is usually represented by a "+" symbol in the EVS image sensor 304.

[0199] The negative event flow corresponds to the event of reduced brightness, which can be understood as the departure of the photon event causing the brightness reduction. The event unit outputting the negative event is usually represented by a "-" symbol in the EVS image sensor 304.

[0200] The EVS image sensor 304 can capture the spatiotemporal information of brightness changes in the scene by recording the occurrence of positive events and negative events. The first data stream output by the EVS image sensor 304 is a continuous, asynchronous event stream, rather than the discrete image frames output by the RGB image sensor 303. The EVS image sensor 304 has the advantages of high temporal resolution and low latency.

[0201] Figure 7a The schematic diagram of the EVS image sensor 304 at three time points (from early to late, time point t0, time point t1, and time point t2) is exemplarily shown.

[0202] like Figure 7a (1) At time t0, the mobile phone 300 receives an instruction from the user to use the front camera of the mobile phone 300 to take a photo. Therefore, at time t0, none of the event units in the EVS image sensor 304 in the front camera outputs an event.

[0203] Then, at time t1, if Figure 7a As shown in (2), some event units 3041 detect photon events and output corresponding events. Figure 7a (2) The event unit represented by "+" in the above example detects that the brightness of the corresponding pixel point has increased compared with the brightness of the previous moment (for example, moment t0), and the increase is greater than the specified threshold, thereby outputting a positive event; and a part of the event units (in the above example) Figure 7a The event unit represented by “-” in (2) detects that the brightness of the corresponding pixel point is reduced compared with the brightness of the previous moment (for example, moment t0), and the brightness reduction is greater than the specified threshold, thereby outputting a negative event; of course, there are also some time units whose corresponding pixel points do not undergo brightness changes, so they do not output events.

[0204] Next, similar to the above description about time t1, at time t2, if Figure 7a As shown in (3), some event units 3041 detect photon events and output corresponding events. Figure 7a (3)) detects that the brightness of its corresponding pixel point has increased compared with the brightness of the previous moment (for example, moment t1), and the increase is greater than the specified threshold, thereby outputting a positive event; and some event units (in Figure 7a(3) The event unit represented by “-” in (3) detects that the brightness of the corresponding pixel point is reduced compared with the brightness of the previous moment (for example, moment t1), and the brightness reduction is greater than the specified threshold, thereby outputting a negative event; of course, there are also some time units whose corresponding pixel points have not changed in brightness compared with the previous moment (for example, moment t1), so they do not output events.

[0205] Among them, the time t0, the time t1, and the time t2 are not necessarily continuous times. Whether the three times are continuous depends on the brightness change of each pixel point corresponding to the shooting scene.

[0206] Through this process, the EVS image sensor 304 can output a positive event stream and a negative event stream (both are Figure 6a The event streams are transmitted to the fusion module 103. These event streams can record the rapidly changing brightness information in the captured scene with extremely high temporal resolution and extremely low latency.

[0207] Among them, S101a and S101b are parallel steps. After the user operates the front camera of the mobile phone 300 to take a picture, the mobile phone can trigger the execution of the above S101a and S101b in response to the user operation.

[0208] S103, the fusion module of the mobile phone 300 can fuse the first RGB image stream from the RGB image sensor 303 and the first data stream to obtain a first image or a first video.

[0209] For example, the user triggers the operation of taking a photo with the front camera to execute this application. Figure 6a The first image can be obtained by following the process;

[0210] For example, the user triggers the operation of using the front camera to record a video to execute this application Figure 6a The first video can be obtained by following the process.

[0211] In some embodiments, the function of the fusion module can be implemented by any one of the RGB image sensor 303 and the EVS image sensor 304 .

[0212] In some embodiments, the function of the fusion module may also be implemented by an image signal processor (ISP) of the mobile phone 300 .

[0213] The fusion module can be implemented as a module in hardware or software, which is not limited here.

[0214] Figure 8 The internal modules of the fusion module are shown as an example. Figure 8 As shown, the fusion module may include but is not limited to: an alignment module, a denoising module, a defogging module, a deblurring module, a contrast processing module, etc.

[0215] In some embodiments, the internal modules of the fusion module can be merged, split, increased, or decreased, and can be flexibly configured according to actual fusion requirements, which is not limited in this application.

[0216] In some embodiments, although the direct output result of the EVS image sensor 304 is not an image frame, but a first data stream related to the brightness change of the pixel point (such as the above-mentioned positive event stream and negative event stream), when the mobile phone 300 executes S103, in order to better implement the fusion operation, the mobile phone can process and analyze the first data stream (event stream) to convert the event stream into a visual result, such as generating an event image or other forms of expression. These conversion processes usually involve processing steps such as accumulation, filtering and integration of the event stream to obtain more intuitive brightness change information or visualization results (such as images, expressed as image streams of event cameras). Then when the fusion module fuses the two data streams, it can fuse the converted image stream of the event camera and the first RGB image stream.

[0217] For example, the fusion module may aggregate (e.g., accumulate or average) event streams with timestamps within a period of time to generate an image frame representing brightness changes (also referred to as an image of an event camera), the resolution of which is independent of the resolution of each RGB image frame in the first RGB image stream (e.g., M1*N1), and the resolution of the image of the event camera is related to the size (M2*N2) of the array formed by the EVS pixel units on the EVS image sensor 304. However, it should be noted that each pixel point in the image of the event camera representing brightness changes generated by the fusion module reflects the brightness changes of one or more pixels in the first image or the first video of the shooting scene captured by the front camera, rather than the absolute brightness of the corresponding pixels in the first image or the first video of the shooting scene. The image of the event camera generated in this way can present the dynamic changes in brightness and brightness intensity information in the shooting scene.

[0218] Then, when the mobile phone 300 performs a fusion operation, the fusion module may fuse the sequence of images of the event camera representing brightness changes with the first RGB image stream to obtain a first image or a first video.

[0219] The specific process of the fusion module processing and analyzing the first data stream to obtain the above-mentioned target information is not limited here and can be achieved through known related technologies or related technologies to be developed in the future.

[0220] In addition, in some embodiments, although the RGB image sensor 303 outputs an image stream of RGB images, before fusion, the mobile phone 300 may also convert the first RGB image stream into an image stream of other image formats by hardware or software, and then pass it to the fusion module to be fused with the output result of the EVS image sensor 304.

[0221] For example, the raw data collected by the RGB image sensor 303 is usually captured in RGB format, but for storage and transmission, the collected first RGB image stream can also be converted into an image stream in other image formats, such as YUV format, by software or hardware, etc., where YUV is a color space mainly used for video compression and transmission. In this color space, "Y" represents luminance, and "U" and "V" represent chrominance.

[0222] Event cameras have shown great potential in many applications, especially in scenes where the shooting light of traditional cameras may be limited, such as low light, high dynamic range, or fast-moving scenes. Using the output of an event camera to enhance RGB images may require complex algorithms and techniques. The following lists some possible implementation methods. In practical applications, the implementation methods listed in this application can be optimized and adjusted according to needs and future technological developments. The optimized and adjusted solutions are still within the scope of protection of this application. This application can use event cameras to optimize images captured by traditional RGB image sensors, so that the optimized Figure 6a The noise reduction, blur reduction, and glare reduction of the video frames in the first image or the first video shown can improve the quality of the first image or the first video.

[0223] Figure 8 For an exemplary embodiment of a possible implementation method, to achieve the fusion of two sets of image streams to improve the quality of the first image or the first video, please refer to Figure 8 The specific fusion process performed by the fusion module may include but is not limited to the following steps:

[0224] S201, an alignment module may align a first RGB image stream and a first data stream related to brightness changes of pixels.

[0225] The alignment operation of the alignment module may include, but is not limited to, timestamp alignment and image registration operations.

[0226] Regarding the alignment operation: Since the working principles and acquisition frequencies of the RGB image sensor 303 and the EVS image sensor 304 are different, there is a time difference between the data collected by each. Therefore, the alignment module can align the RGB image in the first RGB image stream with the event stream in the first data stream according to the timestamp so that they can be fused at the same time point.

[0227] Optionally, as described above, the first data stream output by the EVS image sensor 304 can also be converted into an image stream about brightness changes, and the pixel value of a pixel point in each frame of the image stream is: the brightness change value of the pixel area corresponding to the pixel point in the aligned RGB image.

[0228] Please refer to Figure 5a , the first data stream output by the EVS image sensor 304 can be converted into an image stream about brightness changes, and the resolution of each frame of the image stream (also described as an image of the event camera) is M2*N2 (related to the size of the array formed by the event units of the EVS image sensor 304). Here, M2=N2=4.

[0229] Similarly, the resolution of each frame of RGB image in the first RGB image stream output by the RGB image sensor 303 is Figure 5a (1) shows M1*N1, where M1=N1=8.

[0230] Therefore, there is the following corresponding relationship between the aligned event camera image and the RGB image: the brightness change value expressed by the pixel value of a pixel point in the first row and first column of the event camera image is: the brightness change of the image area in the upper left corner of the RGB image. The resolution of the image area in the upper left corner is 2*2, where 2=8 / 4. The image area in the upper left corner is an image area composed of 4 pixels in the first row and first column, the first row and second column, the second row and first column, and the second row and second column of the RGB image. The pixel values ​​of other pixels in the event camera image can reflect the brightness change of the corresponding 2*2 image area in the RGB image.

[0231] In some embodiments, when aligning, the alignment module may also align the first RGB image stream and the image stream regarding brightness changes according to timestamps.

[0232] About the registration operation: Image registration refers to the process of matching and superimposing two or more images acquired at different times, with different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.).

[0233] The alignment module may perform image registration on the RGB image in the first RGB image stream and the image in the image stream regarding brightness changes (hereinafter referred to as the image of the event camera) to ensure that they are in the same coordinate system.

[0234] When performing image registration, the alignment module can usually use feature matching techniques to find corresponding points between the two image streams.

[0235] The feature matching technology may be, for example, a scale-invariant feature transform (SIFT) algorithm, an ORB (Oriented FAST and Rotated BRIEF) algorithm, a Speeded Up Robust Features (SURF), etc., which are not limited here. Among them, ORB is an algorithm for feature point descriptors in computer vision.

[0236] For example, the alignment module can find corresponding points between two image streams by following the steps below:

[0237] S1, feature extraction and matching.

[0238] The alignment module can use a feature extraction algorithm (such as SIFT, ORB, FAST, etc.) to obtain feature points and corresponding descriptors for the first RGB image stream and the image stream of the event camera.

[0239] The alignment module can use feature matching algorithms (such as optical flow method, descriptor-based matching, etc.) to match the extracted feature points between consecutive frames and find the same feature point pairs.

[0240] S2, pose estimation and image alignment.

[0241] The alignment module can use a camera motion estimation algorithm (such as a homography matrix, a fundamental matrix, etc.) in combination with feature point pairs to estimate the motion posture between cameras.

[0242] The alignment module uses the estimated motion pose to rectify and align the RGB image so that it geometrically corresponds to the image from the event camera.

[0243] The above examples of the alignment module are not intended to limit the present application. The present application may also implement other implementations to implement the timestamp alignment and image registration of the first RGB image stream and the first data stream.

[0244] S202, a denoising module may denoise an RGB image according to an image (or event stream) of an event camera.

[0245] Denoising is to reduce noise in the image and improve image quality. In some embodiments, the denoising module can use a classic image denoising method (such as Gaussian filtering, median filtering, etc.) or a deep learning model to denoise the RGB image.

[0246] In one possible implementation, the denoising module can use the high temporal resolution property of event data to determine the difference between noise and real content. Specifically, for each event in the event stream output by the event camera (i.e., the EVS image sensor 304), the denoising module can check the events of event units adjacent to the pixel position. If a large number of adjacent pixels trigger an event at the same time in a short period of time, the triggered event may be noise. On the contrary, an isolated event may represent a real edge or texture.

[0247] For example, the front camera of Example 1 (specifically including the RGB image sensor 303 and the EVS image sensor 304) is located under the display screen. When the front camera is used for shooting, the RGB image output by the RGB image sensor 303 may be very noisy due to the influence of the lighting, such as insufficient lighting. However, the event data captured by the EVS image sensor 304 can help determine the real structure and random noise in the RGB image, and the denoising module can use this as a basis to denoise the RGB image.

[0248] It should be understood that the above only exemplarily describes a denoising process that may be implemented by a denoising module. In other embodiments, the denoising module may also use other known or future-developed technologies to denoise the RGB image based on the image (or event stream) of the event camera to reduce the noise of the RGB image caused by lighting shooting under the display screen.

[0249] S203, the defogging module may defog the RGB image according to the image (or event stream) of the event camera.

[0250] Defogging is used to reduce the haze (hazy feeling) in the image to improve the image clarity. In some embodiments, the defogging module can use a single image defogging algorithm (such as Dark Channel Prior) or a deep learning model to perform defogging to improve the image visibility of the obtained first image or first video.

[0251] The RGB image sensor 303 is located under the display screen 301. The complex ambient light and the light emitted by the light-emitting element 302 of the display screen 301 itself will also interfere with the light capture of the RGB image sensor 303, thereby causing diffraction and glare problems. The event camera (i.e., the EVS image sensor 304) can help determine the intensity of diffraction and glare. When shooting, the EVS image sensor 304 can capture the brightness changes caused by scattered light or diffracted light, etc., so the defogging module can estimate the thickness of the fog by comparing the event data in the event stream and the RGB image, and perform defogging accordingly.

[0252] For example, in this example 1, the front camera (specifically including the RGB image sensor 303 and the EVS image sensor 304) are both located under the display screen. When the front camera is used to take pictures, the lighting of the RGB image sensor 303 is affected. In this way, when there is water vapor or the like in the image acquisition direction of the front camera, the image clarity of the first image or the first video taken can be enhanced through this method.

[0253] It should be understood that the above only exemplarily describes a dehazing process that may be implemented by a dehazing module. In other embodiments, the dehazing module may also use other known or future-developed technologies to dehaze the RGB image based on the image (or event stream) of the event camera to reduce the haziness of the captured RGB image caused by the front camera shooting under the display screen.

[0254] S204 , a deblurring module may deblur the RGB image according to the image (or event stream) of the event camera.

[0255] Deblurring is to reduce image blur caused by camera shake or object motion. In some embodiments, the deblurring module can use a single or multiple image deblurring algorithm, such as blind deblurring or a deep learning model, to deblur the RGB image.

[0256] The RGB image sensor 303 is located under the display screen 301. The complex ambient light and the light emission of the light-emitting element 302 of the display screen 301 itself will also interfere with the light capture of the RGB image sensor 303, thereby causing the problem of blurred RGB images. The deblurring module can use the high temporal resolution of the event stream of the event camera (i.e., the EVS image sensor 304) to estimate motion and camera shake. Specifically, the deblurring module can estimate the motion trajectory at the pixel level by analyzing the continuous event data in the event stream. These trajectories can be used to guide the estimation of the blur kernel, thereby achieving deblurring of the RGB image.

[0257] It should be understood that the above only exemplarily describes a deblurring process that may be implemented by a deblurring module. In other embodiments, the deblurring module may also use other known or future-developed technologies to deblur the RGB image based on the image (or event stream) of the event camera to reduce the blur of the RGB image caused by the front camera shooting under the display screen.

[0258] S205, the contrast processing module may perform contrast enhancement processing on the RGB image according to the image (or event stream) of the event camera.

[0259] Contrast processing is used to enhance details and contrast in the RGB image, making the processed RGB image more visually appealing. In some embodiments, the contrast processing module can enhance the contrast of the RGB image based on the image (or event stream) of the event camera by contrast enhancement techniques such as histogram equalization or limited contrast adaptive histogram equalization (CLAHE).

[0260] The RGB image sensor 303 is located below the display screen 301. The complex ambient light and the light emission of the light-emitting element 302 of the display screen 301 itself will also interfere with the light capture of the RGB image sensor 303, resulting in the problem of low contrast and unclear image details of the captured RGB image.

[0261] The contrast processing module can enhance the dynamic range of the RGB image by using the brightness changes captured in the event stream of the event camera (i.e., the EVS image sensor 304). The event camera is very sensitive to brightness changes, and the contrast processing module can use the event stream acquired by the event camera as a form of local histogram equalization to achieve the effect of enhancing the contrast of the RGB image.

[0262] For example, in a scene with alternating light and dark, the event data in the event stream can help the contrast processing module determine which areas should be brightened and which areas should be darkened, and adjust the contrast of the RGB image accordingly.

[0263] It should be understood that Figure 8 The processes shown and the related introductions are only some implementations of the fusion module using the output of the event camera to enhance the RGB image. In practical applications, new processing operations can be added or some processing operations can be reduced as needed. In addition, the specific processing process of the fusion module can also be coupled to the process of image processing of the RGB image stream collected by the traditional RGB image sensor in the prior art, which is not limited here.

[0264] in addition, Figure 8The execution order of the modules shown is only an example. In other embodiments, the order may be adjusted according to the desired image effect.

[0265] It is clear that the image effect of the first image or the first video processed by the fusion module of the present application can be improved in terms of image clarity, contrast, etc., compared with the image or video obtained after processing the first RGB image stream captured by the RGB image sensor 303 in a single channel, so as to reduce the impact of the display screen 301 on the image captured by the RGB image sensor 303.

[0266] Figure 8 The specific operation process of the fusion module of the present application when performing various types of fusion operations (denoising, defogging, deblurring, and contrast processing) is shown.

[0267] In some embodiments, Figure 6a As shown, the fusion module is in progress Figure 8 When any type of fusion operation is performed, it can be achieved through a pixel-level fusion method, or through a feature-level fusion method.

[0268] 1. Pixel-level fusion: For each pixel in the first image or the first video, the fusion module can adjust the color value (e.g., RGB value) of the corresponding pixel (e.g., the pixel area reflected by the coordinates of the event information) in the first RGB image stream output by the RGB image sensor 303 according to the event information in the event stream provided by the event camera (EVS image sensor 304).

[0269] For example, the RGB image can be weighted and adjusted according to the intensity (i.e., the amount of brightness increase, the amount of brightness decrease) and polarity of the event to highlight the importance of the event. This method can retain the details of the RGB image and enhance it in combination with event information.

[0270] For example, computer vision libraries such as OpenCV can be used to process and align images.

[0271] In a specific example, the method is applied to the field of autonomous driving. RGB images can provide visual information of the road, but the RGB image sensor 303 is used as an under-screen camera and may be blurred or lost in fast motion or low light conditions. The event camera (i.e., the EVS image sensor 304) can capture motion or fast-changing events in real time to provide information with high temporal resolution.

[0272] In this case, the fusion module can use a pixel-level fusion method to process the event data of the event camera to obtain event intensity and polarity information about each pixel or each pixel area in the RGB image. Then, the fusion module adjusts the pixel color of the RGB image according to this event information to improve the visibility of moving objects on the road. In this way, the driving system can enhance its perception of dynamic changes on the road and improve safety and robustness.

[0273] Specific plan:

[0274] S11, the fusion module obtains the RGB image stream and the image stream of the event camera.

[0275] First, the fusion module can obtain the image streams corresponding to the RGB image sensor 303 and the EVS image sensor 304 respectively through the camera interface of OpenCV.

[0276] S12, the fusion module processes the RGB image stream and the image stream of the event camera.

[0277] The fusion module can use OpenCV functions and tools to preprocess the RGB image stream and the event camera image stream. The preprocessing may include but is not limited to: image enhancement, filtering, resizing, etc. to ensure that the two image streams have similar features.

[0278] S13, a fusion module fuses the images.

[0279] The fusion module can use the addWeighted function of OpenCV to fuse the RGB image and the image of the event camera at the pixel level. The weights of the fusion can be flexibly adjusted as needed to make the contribution of the two images meet the requirements.

[0280] In this way, the fusion module can assign different weights to the RGB image and the image of the event camera through a pixel-level fusion method to obtain a fused first image or a first video.

[0281] 2. Feature-level fusion: In addition to pixel-level adjustment, the fusion module can also perform feature extraction on the RGB image and the event stream output by the event camera (or the image stream of the event camera), respectively, to extract image features from the RGB image and brightness features from the event stream. The brightness features represent the features of brightness changes, and the extracted features are fused together to obtain a first image or a first video.

[0282] For example, the fusion module can use a convolutional neural network (CNN) to extract features from the RGB image, use the event data from the event camera as additional information, and then fuse the two features. This method can combine the appearance information of the RGB image and the temporal dynamic information of the event camera for fusion. For example, the fusion module can use a deep learning framework such as PyTorch or TensorFlow to implement feature extraction and fusion operations.

[0283] In a specific example, the method is also applied to the field of autonomous driving.

[0284] Specific plan:

[0285] S21, Data collection and preparation:

[0286] First, it is necessary to collect image data and event data collected by the RGB image sensor 303 and the EVS image sensor 304 respectively, and set label data (for example, object position, speed, etc.) to obtain a training set.

[0287] S22, build a deep learning model:

[0288] Build a deep neural network model that takes the RGB image and the image from the event camera as input and produces a fused feature map as output.

[0289] For example, you can use pre-trained convolutional neural networks (such as ResNet, VGG) as feature extractors and customize fusion modules, such as fusion convolutional layers or attention mechanisms.

[0290] S23, training model:

[0291] The training set with labeled data obtained in S21 is used to train the model constructed in S22. The training target can be the label of tasks such as object detection and lane tracking, wherein the model can be trained using an appropriate loss function and optimizer.

[0292] S24, applied to autonomous driving:

[0293] In the autonomous driving scenario, the mobile phone 300 can use the trained converged model to perform real-time feature fusion on the image data and event data (or the converted event camera images) collected by the RGB image sensor 303 and the EVS image sensor 304 to obtain a fused feature map, for example Figure 6a The first image or the image in the first video is shown.

[0294] In this way, the fusion module can extract features from the RGB image and the image of the event camera through a feature-level fusion method and fuse the extracted features to obtain a fused first image or a first video.

[0295] It should be understood that the above is merely an example in which the fusion module of the present application adopts a feature-level fusion method or a pixel-level fusion method to fuse the RGB image stream and the event stream. In practical applications, other fusion algorithms can be flexibly adopted according to the fusion requirements and the image requirements for the first image or the first video, and the fusion process of the example in this embodiment can be adjusted and optimized, which is not limited here.

[0296] In addition, when fusing the RGB image stream and the event stream, the fusion module of the present application is not limited to the feature-level fusion method or the pixel-level fusion method, but may also adopt other known or future-developed fusion methods and technologies to achieve the fusion of the RGB image stream and the event stream, so that the resulting fused image or image sequence has improved noise, diffraction, glare, blur, contrast and other problems compared to the image or image sequence obtained after processing a single RGB image stream.

[0297] In this example 1, two image sensors are independently set in the front camera, namely, the RGB image sensor 303 and the EVS image sensor 304. The two image sensors are two independent hardwares and are physically isolated. Therefore, the data collected by each is independent of each other and there is no interference with each other. The RGB image stream collected by the RGB image sensor 303 is affected by the setting position under the screen and has the problem of lighting interference. However, the EVS image sensor 304 is set and operated independently of the RGB image sensor, so that the event stream output by the EVS image sensor 304 is not affected by the RGB image sensor, so that it can have accurate and stable pixel-level brightness change data output. Then the mobile phone 300 uses the accurate and stable pixel-level brightness change data output by the EVS image sensor 304 and the RGB image stream collected by the RGB image sensor 303 to fuse, so that the image quality of the fused first image or first video can be better.

[0298] Example 2

[0299] The implementation process of Example 2 is mostly the same as that of Example 1, except that the front camera of the mobile phone is equipped with only one composite image sensor instead of two types of image sensors (RGB image sensor and EVS image sensor). Two types of pixel units are integrated in the composite image sensor, namely, RGB pixel units and EVS pixel units.

[0300] Combine the following Figure 4b , Figure 5b , Figure 6b , Figure 7b , 7c, and 7d are used to mainly introduce the differences between Example 2 and Example 1. The similarities can be referred to the introduction of Example 1 and will not be repeated here.

[0301] Figure 4b The front schematic diagram and the cross-sectional schematic diagram of the mobile phone of the present application are exemplarily shown.

[0302] like Figure 4b As shown in (1), a front camera is arranged below the display screen 401 (full screen) of the mobile phone 400, and the front camera may include a composite image sensor 403. Figure 4b In (1), the position of the composite image sensor 403 located below the display screen 401 is indicated by a dotted circle. In actual application, the composite image sensor 403 cannot be seen from the display screen 401 of the mobile phone 400.

[0303] In other embodiments, the number of the composite image sensors 403 in the front camera may be multiple. In addition, the front camera may also include other types of image sensors, which are not limited here.

[0304] Figure 4b (2) Figure 4b (1) is a schematic cross-sectional view of the mobile phone 400 in the width direction indicated by the double arrows.

[0305] like Figure 4b As shown in (2), the display screen 401 is composed of a plurality of light-emitting elements 402, and the composite image sensor 403 is located below the light-emitting elements 402 of the display screen 401, so that the display screen 401 can be used as a full screen, and the composite image sensor 403 is used as an under-screen camera.

[0306] The composite image sensor 403 may include an RGB pixel unit 403a and an EVS pixel unit 403b, so that the composite image sensor 403 can not only capture an RGB image, but also capture information on brightness changes associated with pixels.

[0307] Figure 5b The structural diagram of the composite image sensor 403 is exemplarily shown.

[0308] like Figure 5b As shown, the array size formed by the pixel units on the composite image sensor 403 is M3*N3, where N8=N3=8.

[0309] A plurality of RGB pixel units are arranged on the composite image sensor 403, and the plurality of RGB pixel units may include a pixel unit of a red channel (indicated by R) 3031, a pixel unit of a green channel (indicated by G) 3032, and a pixel unit of a blue channel (indicated by B) 3033. The plurality of RGB pixel units are arranged on the composite image sensor 403 in a Bayer array. Figure 5b As shown, the size of the array formed by the multiple RGB pixel units is the same as the size of the array corresponding to the composite image sensor 403, both of which are M3*N3.

[0310] In addition, a plurality of EVS pixel units are arranged on the composite image sensor 403. The plurality of EVS pixel units are a plurality of event units, respectively: Figure 5b Shown are event unit 304a, event unit 304b, event unit 304c and event unit 304d.

[0311] exist Figure 5b In the embodiment, each event unit occupies the position of a pixel unit belonging to a green channel in an RGB pixel unit arranged in a Bayer array, so that the event units are arranged in an M4*N4 (here 2*2) array on the composite image sensor 403.

[0312] in, Figure 5b In an embodiment, the size of the array formed by the event cells on the composite image sensor 403 is smaller than M3*N3.

[0313] In other embodiments, the event cells on the composite image sensor 403 may also occupy the positions of pixel cells of other color channels in the array composed of RGB pixel cells, such as pixel cells of a blue channel and / or pixel cells of a red channel.

[0314] The array size of the composite image sensor 403 is M3*N3 (here 8), so Figure 5b In an embodiment, the event data output by each event unit in the event units arranged in a 2*2 array may be used to reflect the brightness change of an image area with a resolution of 4*4 in the RGB image output by the composite image sensor 403 .

[0315] For example, the pixel area corresponding to the coordinate position of the event unit 304a is the 4*4 pixel area in the upper left corner of the M3*N3 array, specifically including the pixel area composed of the pixel points from the first column to the fourth column of the first row, the first column to the fourth column of the second row, the first column to the fourth column of the third row, and the first column to the fourth column of the fourth row. Therefore, the event data output by the event unit 304a can reflect the brightness change of the 4*4 pixel area at the upper left corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; similarly, the event data output by the event unit 304b can reflect the brightness change of the 4*4 pixel area at the lower left corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; the event data output by the event unit 304c can reflect the brightness change of the 4*4 pixel area at the upper right corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; and the event data output by the event unit 304d can reflect the brightness change of the 4*4 pixel area at the lower right corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403. In this way, the brightness change of the image area of ​​the entire RGB image with a resolution of M3*N3 output by the composite image sensor 403 is fully reflected.

[0316] It should be understood that the present application does not limit the layout positions of the EVS pixel units arranged on the composite image sensor 403, and the positional relationship between them and the RGB pixel units.

[0317] In practical applications, the more EVS pixel units are arranged on the composite image sensor 403, the better the brightness improvement of the image captured by the camera including the composite image sensor 403, but the cost is also higher. Therefore, according to factors such as cost, the area size of the composite image sensor 403, and the intensity of improvement required for the captured image, the EVS pixel units of a reasonable number and reasonable position can be arranged on the composite image sensor, and this application does not limit this.

[0318] In addition, the present application does not limit the shape of the components of the event unit on the composite image sensor 403 and the size of the space occupied. Figure 5b In the example, one event unit occupies the layout space of one RGB pixel unit. In other embodiments, one event unit may also occupy the layout space of multiple RGB pixel units.

[0319] In some embodiments, an event unit can reflect the brightness change of the pixel area represented by multiple RGB pixel units. Therefore, the layout of multiple event units on the composite image sensor 403 can be more sparse than the layout of RGB pixel units. For example, the size of the array composed of event units can generally be smaller than the size of the array composed of RGB pixel units. Of course, in some embodiments, the sizes of the two arrays can also be the same.

[0320] In other embodiments, the layout of the event units on the composite image sensor 403 may also be denser than the layout of the RGB pixel units, so that the size of the array formed by the event units is larger than the size of the array formed by the RGB pixel units. The present application does not limit the size relationship between the two arrays, which can be set according to needs.

[0321] The multiple pixel units (including RGB pixel units and EVS pixel units) of the composite image sensor 403 are overall arranged as an array of M3*N3, wherein M3=N3, M3 represents the number of pixel units in the width direction of the array, which is 8 here, and N3 represents the number of pixel units in the height direction of the array, which is 8 here.

[0322] In other embodiments, the multiple RGB pixel units on the composite image sensor 403 may also be arranged in other arrays, which is not limited here. Similarly, the multiple EVS pixel units on the composite image sensor 403 may also be arranged in other arrays, which is also not limited. In addition, the present application is not limited to M3 and N3 being the same value, and they may also be different.

[0323] exist Figure 5b In the embodiment, the event unit on the composite image sensor 403 occupies a position that can accommodate an RGB pixel unit, so that the number of RGB pixel units is partially missing in its array structure. Figure 5c As shown, Figure 5c A schematic diagram of the structure of a pixel unit on another composite image sensor 403 is shown.

[0324] Figure 5c Examples and Figure 5b Most of the contents of the embodiments are the same, except that Figure 5c The event unit arranged on the composite image sensor 403 shown does not occupy the position belonging to the RGB pixel unit in the Bayer array composed of the RGB pixel units, so that even if the EVS pixel unit is arranged on the composite image sensor 403, it does not affect the array structure of the RGB pixel unit.

[0325] like Figure 5cAs shown, the size of the array formed by all pixel units (including RGB pixel units and EVS pixel units) on the composite image sensor 403 is M3*N3+M4*N4.

[0326] The RGB pixel unit on the composite image sensor 403 still adopts the Bayer array in Example 1, and the size of the array is M3*N3 (here 8*8), while the size of the array composed of the event unit 304a, event unit 304b, event unit 304c and event unit 304d on the composite image sensor 403 is M4*N4 (here 2*2). The sizes of the two arrays are independent of each other and can be flexibly set according to needs, and are not limited here.

[0327] exist Figure 5c In the embodiment, the event unit 304a is taken as an example. There are 4 groups of RGB pixel units arranged around the event unit 304a. Each group of RGB pixel units forms a 2*2 Bayer array, so that the event data output by the event unit 304a can be used to reflect the brightness change of the image area (resolution 4*4) in the upper left corner of the RGB image (resolution M3*N3) output by the composite image sensor 403. The other event units are similar. Figure 5b The same embodiment, Figure 5c Although the layout position of the event unit in the embodiment is Figure 5b Although the layout positions shown are different, the event data output by the event units denoted by the same reference numerals reflect the same size and position of the image area.

[0328] For example, the pixel area corresponding to the coordinate position of the event unit 304a is the 4*4 pixel area in the upper left corner of the M3*N3 array, specifically including the pixel area composed of the pixel points from the first column to the fourth column of the first row, the first column to the fourth column of the second row, the first column to the fourth column of the third row, and the first column to the fourth column of the fourth row. Therefore, the event data output by the event unit 304a can reflect the brightness change of the 4*4 pixel area at the upper left corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; similarly, the event data output by the event unit 304b can reflect the brightness change of the 4*4 pixel area at the lower left corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; the event data output by the event unit 304c can reflect the brightness change of the 4*4 pixel area at the upper right corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403; and the event data output by the event unit 304d can reflect the brightness change of the 4*4 pixel area at the lower right corner of the RGB image with a resolution of M3*N3 output by the composite image sensor 403. In this way, the brightness change of the image area of ​​the entire RGB image with a resolution of M3*N3 output by the composite image sensor 403 is fully reflected.

[0329] In some embodiments, the present application does not limit the size of the array formed by the event units arranged on the composite image sensor 403. Figure 5d In the embodiment, Figure 5d Examples and Figure 5c The difference between the embodiments is that Figure 5d In the embodiment, the EVS pixel units (ie, event units) arranged on the composite image sensor 403 are denser, and the improvement effect on the RGB image is better.

[0330] like Figure 5d As shown, the size of the array formed by all pixel units (including RGB pixel units and EVS pixel units) on the composite image sensor 403 is M3*N3+M5*N5.

[0331] The RGB pixel unit on the composite image sensor 403 still adopts the Bayer array in Example 1, and the size of the array is M3*N3 (here 8*8), while the size of the array formed by the event unit 3041 on the composite image sensor 403 is M5*N5 (here 4*4). The sizes of the two arrays are independent of each other and can be flexibly set according to needs, and are not limited here.

[0332] and Figure 5c The difference between the embodiments is that Figure 5dEach event unit 3041 shown can reflect the brightness change of an image area of ​​2*2 in an image with a resolution of M3*N3 (here 8*8) taken by the front camera of the mobile phone 400.

[0333] For example Figure 5d In the 4*4 array of event units on the composite image sensor 403 shown in the figure, the event data output by the event unit 3041 located in the first row and the first column can be used to reflect the event data of the composite image sensor 403. Figure 5d The brightness change of the image area in the upper left corner of the image with a resolution of M3*N3 (here 8*8) captured by the RGB pixel unit in the composite image sensor 403 is shown.

[0334] The image area in the upper left corner includes 4 RGB pixel units, which are Figure 5d In the array of 8*8 RGB pixel units arranged on the composite image sensor 403 shown, four RGB pixel units are located in the first row and first column, the first row and second column, the second row and first column, and the second row and first column.

[0335] Above Figure 5a to Figure 5c Some structural schematic diagrams of the composite image sensor 403 of the present application are only illustrated by way of example. In other embodiments, the structure of the composite image sensor 403 of the present application may vary. In other words, the present application does not limit the size and shape of the array formed by the RGB pixel units on the composite image sensor 403, and does not limit the size and shape of the array formed by the EVS pixel units on the composite image sensor 403.

[0336] Combine the following Figure 6b , Figure 7b and Figure 8 To introduce the mobile phone 400 of this application, Figure 4b and Figure 5b The composite image sensor 403 shown in the figure is used to perform image or video capture in a processing scenario. Figure 5c or Figure 5d Or other composite structures, the implementation principle of the method is the same and will not be repeated here.

[0337] S301 , the composite image sensor 403 of the mobile phone 400 collects a mixed data stream.

[0338] Among them, Figure 5b As shown, the composite image sensor 403 with an array size of M3*N3 can be used to collect a mixed data stream, which may include RGB information collected by RGB pixel units and an event stream collected by EVS pixel units.

[0339] The event units on the composite image sensor 403 of the present application may adopt pixel-level event sensing, and each event unit may independently sense photon events to output pixel-level event data (including timestamp, coordinates, and polarity).

[0340] The difference from Example 1 is that Figure 5b As shown, event unit 304a, event unit 304b, event unit 304c and event unit 304d on the composite image sensor 403 are arranged interspersed between the RGB pixel units, so that each event unit can sense the brightness change of the 4*4 pixel area in the RGB image corresponding to its coordinates.

[0341] Therefore, the mixed data corresponding to one timestamp in the mixed data stream may include RGB image data output by the RGB pixel unit on the compound image sensor 403 and event data output by the event unit on the compound image sensor 403 .

[0342] Figure 7b The schematic diagram of the composite image sensor 403 at three time points (from earliest to latest, time point t0, time point t1, and time point t2) is exemplarily shown.

[0343] like Figure 7b (1) At time t0, the mobile phone 400 receives an instruction from the user to use the front camera of the mobile phone 400 to take a picture. Therefore, at time t0, none of the event units in the composite image sensor 403 in the front camera outputs an event, and the pixel units of the red channel, the green channel, and the blue channel do not output a color value.

[0344] Then, at time t1, if Figure 7b As shown in (2), some event units detect photon events and output corresponding events. Figure 7b (2) The event units represented by "+", such as event unit 304a and event unit 304c, detect that the brightness of the corresponding 4*4 pixel area has increased compared with the brightness of the previous moment (such as moment t0), and the increase is greater than the specified threshold, thereby outputting a positive event; and a part of the event units (in Figure 7a The event unit represented by “-” in (2), such as event unit 304d), detects that the brightness of the corresponding 4*4 pixel area is reduced compared with the brightness of the previous moment (such as moment t0), and the brightness reduction is greater than the specified threshold, thus outputting a negative event; of course, there are also some time units (such as Figure 7b The event unit 304b) shown in (2) has a 4*4 pixel area corresponding to it with no brightness change, so it does not output an event.

[0345] At the same time, at time t1, if Figure 7b The RGB pixel unit shown in (2) also outputs RGB image data, such as the coordinate position of the RGB pixel unit and the collected RGB value (such as color or brightness). However, since there are no RGB pixel units arranged at the positions of event unit 304a, event unit 304b, event unit 304c and event unit 304d in the composite image sensor 403, these four positions do not output RGB values.

[0346] Next, similar to the above description about time t1, at time t2, if Figure 7b As shown in (3), some event units detect photon events and output corresponding events. Figure 7b The event units represented by "+" in (3), such as event units 304c and 304d, detect that the brightness of the corresponding 4*4 pixel area has increased compared with the brightness at the previous moment (such as moment t1), and the increase is greater than the specified threshold, thereby outputting a positive event; and a part of the event units (in Figure 7b The event unit represented by “-” in (3), such as event unit 304b), detects that the brightness of the corresponding 4*4 pixel area is reduced compared with the brightness of the previous moment (such as moment t1), and the brightness reduction is greater than the specified threshold, thus outputting a negative event; of course, there are also some time units (such as Figure 7b The brightness of the 4*4 pixel area corresponding to the event unit 304a) shown in (3) has not changed compared with the previous moment (for example, moment t1), so they do not output events.

[0347] At the same time, at time t2, if Figure 7b The RGB pixel unit shown in (3) also outputs RGB image data, such as the coordinate position of the RGB pixel unit and the collected RGB value (such as color or brightness). However, since there are no RGB pixel units arranged at the positions of event unit 304a, event unit 304b, event unit 304c and event unit 304d in the composite image sensor 403, these four positions do not output RGB values.

[0348] Among them, time t0, time t1, and time t2 are continuous moments, such as Figure 7b Each RGB pixel unit in the composite image sensor 403 shown in FIG. 1 outputs RGB image data at each moment. However, Figure 7b Each event unit in the composite image sensor 403 shown does not necessarily output event data at every consecutive moment, because the brightness of some pixels may not change.

[0349] therefore, Figure 7bThis is only used as an example to help readers understand the working principle of the composite image sensor 403 of the present application, but is not used to limit the output result of the composite image sensor 403 at each moment.

[0350] Through this process, the composite image sensor 403 can output a mixed data stream, which may include an RGB image stream output by the RGB pixel unit on the composite image sensor 403 at each moment (not the data of a complete 8*8 RGB image, as there are missing RGB values ​​of four pixels), and an event stream output by the event unit on the composite image sensor 403 at each moment.

[0351] S302, the extraction and generation module of the mobile phone 400 can extract an incomplete second RGB image stream from the mixed data stream, and extract a second data stream related to the brightness change of the pixel points; optionally, the extraction and generation module can also generate a complete third RGB image stream for the incomplete second RGB image stream, and optionally, the extraction and generation module can generate a third data stream related to the brightness change of the pixel points for the second data stream.

[0352] The second RGB image stream is a data stream composed of RGB image data output by RGB pixel units arranged on the composite image sensor 403 in the mixed data stream.

[0353] Among them, Figure 5b As shown, there are missing pixel units of four G channels on the composite image sensor 403. Therefore, there are missing RGB values ​​of the four G channels in each RGB image frame in the second RGB image stream. Then, the extraction and generation module can perform interpolation and other processing (the specific processing method is not limited and can be flexibly selected according to needs) on each RGB image frame extracted in the second RGB image stream to supplement the color values ​​of the missing channels in each RGB image frame to obtain a complete frame of M3*N3=8 RGB images to obtain a complete third RGB image stream.

[0354] Taking the extraction and generation module performing interpolation processing on a frame of RGB image in the second RGB image stream as an example, the extraction and generation module can perform interpolation processing on a frame of RGB image according to the information about Figure 5b The RGB values ​​around the position of the event unit 304a shown (i.e., the coordinate position where the G channel information is missing) are used to supplement the color value of this position; according to this principle, the color values ​​of the missing 4 G channels can be supplemented to obtain a complete frame of RGB image, so as to obtain a complete third RGB image stream.

[0355] In addition, in this embodiment, if Figure 5bThe event data output by each of the four event units (event unit 304a, event unit 304b, event unit 304c and event unit 304d) on the composite image sensor 403 shown can reflect the brightness changes of each corresponding pixel area in the third RGB image stream. Therefore, the second data stream extracted from the mixed data stream by the extraction generation module can be directly used as the third data stream for processing by the fusion module.

[0356] In other embodiments, for example, Figure 5b The composite image sensor 403 shown is not provided with the event unit 304d, so that the 4*4 image area at the lower right corner of each frame image in the third RGB image stream does not have event data in the second data stream that can reflect its brightness change. Similar to the principle of generating the third RGB image stream from the second RGB image stream, the extraction and generation module can generate the incomplete second data stream into a complete third data stream that can reflect the brightness changes of all image areas of the entire RGB image of M3*N3 through interpolation and other processing methods, which will not be repeated here.

[0357] After S302 , the mobile phone 400 may execute S103 .

[0358] S103, the fusion module of the mobile phone 300 can fuse the third RGB image stream and the third data stream to obtain a second image or a second video.

[0359] The fusion processing method may be a pixel-level fusion method or a feature-level fusion method, etc., which will not be elaborated here.

[0360] Among them, the third RGB image stream in Example 2 is similar to the first RGB image stream in Example 1, and the third data stream in Example 2 is similar to the first data stream in Example 1. The principle and process of the processing process of fusing the third RGB image stream and the above-mentioned third data stream in Example 2 are the same as the principle and process of the processing process of fusing the first RGB image stream and the first data stream in Example 1. For details, please refer to the relevant introduction of Example 1, which will not be repeated here.

[0361] In addition, in Example 2, Figure 6b The modules in the extraction generation module and the fusion module shown can be merged or split, and some modules can be added or reduced, which is not limited here.

[0362] And, if Figure 6b The extraction generation module and the fusion module shown can be implemented as hardware or software. In some embodiments, the extraction generation module and the fusion module can also be implemented by the ISP of the mobile phone 400.

[0363] In this example 2, a composite image sensor 403 is provided in the front camera, wherein the composite image sensor 403 may include not only RGB pixel units to collect RGB image data, but also EVS pixel units (i.e., event units) to collect pixel-level brightness change information. By collecting two types of data on one hardware (the composite image sensor 403 here), not only can the brightness change information be used to optimize the RGB image to improve the image quality of the obtained second image or second video, but also the layout space of the image sensor can be saved on the hardware, and the cost can be saved.

[0364] The above Examples 1 and 2 take the application of the sensor structure of the present application to the front camera as an example to illustrate that in some embodiments, the above method and the structure of the image sensor can also be applied to the rear camera of the electronic device and other camera sensors that are slightly contaminated or blocked by dust or water vapor, and the method of the present application is equally applicable.

[0365] In some embodiments, the mobile phone can set a switch option on whether to turn on the image optimization through the event camera. When the user operates the switch option to turn on the switch, the mobile phone can execute the method of Example 1 or Example 2 above to optimize the RGB image through the event data collected by the event camera or EVS pixel unit, so that the images or videos taken by the mobile phone are not affected by the pollution, dust, and water vapor of the camera sensor, and can also have higher image quality.

[0366] In a possible implementation, an embodiment of the present application provides an electronic device. The electronic device includes: at least one camera, the at least one camera includes a first pixel unit associated with an image sensor and a second pixel unit associated with an event camera; wherein the first pixel unit is used to collect first image data, and the second pixel unit is used to collect first brightness change data related to the first image data, and the electronic device includes: an acquisition module, which is used to respond to a received first user operation, and obtain second image data related to the first image data and second brightness change data related to the second image data through the at least one camera, wherein the first user operation indicates to take an image or video, wherein the first user operation indicates to take an image or video; a fusion module, which is used to fuse the second image data and the second brightness change data to obtain a target image or target video.

[0367] In a possible implementation, the electronic device further includes a display screen, the at least one camera is located below the display screen, and the light received by the at least one camera is light transmitted through the display screen.

[0368] In a possible implementation, the at least one camera includes a composite sensor, the composite sensor includes the first pixel unit and the second pixel unit, and the acquisition module is specifically used to: obtain mixed data collected by the at least one camera through the composite sensor, wherein the mixed data includes the first image data and the first brightness change data; extract the first image data and the first brightness change data from the mixed data respectively; generate second image data based on the first image data; wherein the second brightness change data is related to the first brightness change data.

[0369] In a possible implementation manner, the second pixel units arranged on the composite sensor are sparser than the first pixel units arranged on the composite sensor.

[0370] In a possible implementation manner, the first pixel unit is arranged around the second pixel unit.

[0371] In a possible implementation, the at least one camera includes a first camera and a second camera, wherein the first camera includes the image sensor, the image sensor includes a plurality of the first pixel units, and the second camera includes the event camera, the event camera includes a plurality of the second pixel units; the acquisition module is specifically used to: collect a plurality of first image data through the plurality of first pixel units of the first camera to obtain second image data; collect a plurality of first brightness change data through the plurality of second pixel units of the second camera to obtain second brightness change data.

[0372] In a possible implementation manner, the size of the array formed by the first pixel units on the corresponding sensor is related to the resolution of the target image or the video frame in the target video.

[0373] In a possible implementation, the electronic device further includes: a start module, configured to start a switch for enhancing the image quality of an image or video captured by the at least one camera based on the second pixel unit in response to a received second user operation.

[0374] In a possible implementation, the fusion module is specifically used to: extract the first image feature from the second image data; extract the first brightness feature from the second brightness change data; and fuse the first image feature with the first brightness feature to obtain a target image or a target video.

[0375] In a possible implementation, the fusion module is specifically configured to adjust at least one pixel point in the second image data according to the second brightness change data to obtain a target image or a target video.

[0376] In a possible implementation, the fusion module is specifically used to: align the second image data and the second brightness change data; and fuse the aligned second image data and the second brightness change data to obtain a target image or a target video.

[0377] In a possible implementation, the fusion module is specifically configured to denoise the second image data based on the second brightness change data to obtain a target image or a target video.

[0378] In a possible implementation, the fusion module is specifically configured to dehaze the second image data based on the second brightness change data to obtain a target image or a target video.

[0379] In a possible implementation, the fusion module is specifically configured to deblur the second image data based on the second brightness change data to obtain a target image or a target video.

[0380] In a possible implementation, the fusion module is specifically configured to perform contrast enhancement processing on the second image data based on the second brightness change data to obtain a target image or a target video.

[0381] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0382] In one example, Fig. 9 A schematic block diagram of a device 300 according to an embodiment of the present application is shown. The device 300 may include: a processor 301 and a transceiver / transceiver pin 302 , and optionally, a memory 303 .

[0383] The components of the device 300 are coupled together via a bus 304, wherein the bus 304 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 304 in the figure.

[0384] Optionally, the memory 303 may be used for instructions in the aforementioned method embodiment. The processor 301 may be used to execute instructions in the memory 303, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0385] The apparatus 300 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0386] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0387] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0388] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0389] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0390] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0391] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0392] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0393] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0394] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0395] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0396] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0397] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0398] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device. Of course, the processor and the storage medium can also be present in a network device as discrete components.

[0399] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0400] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes at least one camera, the at least one camera includes a first pixel unit associated with an image sensor and a second pixel unit associated with an event camera; wherein the first pixel unit is used to collect first image data, and the second pixel unit is used to collect first brightness change data associated with the first image data, and the method includes: In response to a received first user operation, acquiring, through the at least one camera, second image data related to the first image data and second brightness change data related to the second image data, wherein the first user operation indicates capturing an image or a video; The second image data and the second brightness change data are fused to obtain a target image or a target video.

2. The method according to claim 1, characterized in that The electronic device also includes a display screen, the at least one camera is located below the display screen, and the light received by the at least one camera is the light transmitted through the display screen.

3. The method according to claim 1 or 2, characterized in that: The at least one camera includes a composite sensor, the composite sensor includes the first pixel unit and the second pixel unit, and the acquiring, through the at least one camera, second image data related to the first image data and second brightness change data related to the second image data includes: Acquire mixed data collected by the at least one camera through the composite sensor, wherein the mixed data includes the first image data and the first brightness change data; respectively extracting the first image data and the first brightness change data from the mixed data; generating second image data based on the first image data; The second brightness change data is related to the first brightness change data.

4. The method according to claim 3, characterized in that The second pixel units arranged on the composite sensor are sparser than the first pixel units arranged on the composite sensor.

5. The method according to claim 3 or 4, characterized in that: The first pixel unit is arranged around the second pixel unit.

6. The method according to claim 1 or 2, characterized in that: The at least one camera comprises a first camera and a second camera, wherein the first camera comprises the image sensor, the image sensor comprises a plurality of the first pixel units, and the second camera comprises the event camera, the event camera comprises a plurality of the second pixel units; The acquiring, through the at least one camera, second image data related to the first image data and second brightness change data related to the second image data comprises: Collecting a plurality of first image data through the plurality of first pixel units of the first camera to obtain second image data; A plurality of the first brightness change data are collected through the plurality of second pixel units of the second camera to obtain second brightness change data.

7. The method according to any one of claims 1 to 6, characterized in that The size of the array formed by the first pixel units on the corresponding sensor is related to the resolution of the target image or the video frame in the target video.

8. The method according to any one of claims 1 to 7, characterized in that Before responding to the received first user operation, the method further includes: In response to the received second user operation, a switch for enhancing the image quality of the image or video captured by the at least one camera based on the second pixel unit is activated.

9. The method according to any one of claims 1 to 8, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: extracting first image features from the second image data; extracting a first brightness feature from the second brightness change data; The first image feature and the first brightness feature are fused to obtain a target image or a target video.

10. The method according to any one of claims 1 to 8, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: At least one pixel point in the second image data is adjusted according to the second brightness change data to obtain a target image or a target video.

11. The method according to any one of claims 1 to 10, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: aligning the second image data and the second brightness change data; The aligned second image data and the second brightness change data are fused to obtain a target image or a target video.

12. The method according to any one of claims 1 to 11, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: Based on the second brightness change data, the second image data is denoised to obtain a target image or a target video.

13. The method according to any one of claims 1 to 12, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: Based on the second brightness change data, the second image data is dehazed to obtain a target image or a target video.

14. The method according to any one of claims 1 to 13, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: Based on the second brightness change data, the second image data is deblurred to obtain a target image or a target video.

15. The method according to any one of claims 1 to 14, characterized in that The fusing the second image data and the second brightness change data to obtain a target image or a target video includes: Based on the second brightness change data, contrast enhancement processing is performed on the second image data to obtain a target image or a target video.

16. An electronic device, characterized in that: The electronic device includes at least one camera, the at least one camera includes a first pixel unit associated with an image sensor and a second pixel unit associated with an event camera; wherein the first pixel unit is used to collect first image data, and the second pixel unit is used to collect first brightness change data associated with the first image data, and the electronic device includes: an acquisition module, configured to acquire, in response to a received first user operation, second image data related to the first image data and second brightness change data related to the second image data through the at least one camera, wherein the first user operation indicates capturing an image or a video, wherein the first user operation indicates capturing an image or a video; A fusion module is used to fuse the second image data and the second brightness change data to obtain a target image or a target video.

17. An electronic device, characterized in that: include: a memory and a processor, the memory and the processor being coupled; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the image processing method according to any one of claims 1 to 15.

19. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 15.

20. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method described in any one of claims 1 to 15.

Images