Image sensor, image processing method, electronic device, storage medium and product

Abstract

The present application provides an image sensor, an image processing method, an electronic device, a storage medium and a product, which belong to the field of image processing. The method comprises: in response to an exposure instruction, a pixel array module generates a first image and a second image, the first image comprises a plurality of pixel units, each pixel unit is composed of a plurality of pixels of the same color, the resolution of the first image is higher than the resolution of the second image, and the pixels in the second image are arranged in a Bayer manner; the first processing module processes the first image to obtain a third image, the number of pixels in the third image is the same as that in the first image, and the pixels in the third image are arranged in a Bayer manner; the second processing module processes the second image to obtain a fourth image, and the resolution of the fourth image is the same as that of the third image; the image fusion module fuses the third image and the fourth image to obtain a fused image. The present application can improve image quality and reduce image noise.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image sensor, an image processing method, an electronic device, a storage medium and a product. Background Art

[0002] With the development of technology, more and more terminals have camera functions. When users use terminals to shoot or preview images, they have higher requirements for image quality, hoping that the images have higher clarity and the image noise is as small as possible. In order to meet the needs of users, how to process images has become a problem that technicians in this field are more concerned about. Summary of the invention

[0003] The embodiments of the present application provide an image sensor, an image processing method, an electronic device, a storage medium and a product, which can improve the clarity of an image and reduce the noise of the image. The technical solution is as follows:

[0004] In a first aspect, an image sensor is provided, the image sensor comprising a pixel array module, a first processing module, a second processing module and an image fusion module, wherein a first output end of the pixel array module is connected to an input end of the first processing module, a second output end of the pixel array module is connected to an input end of the second processing module, and an output end of the first processing module and an output end of the second processing module are both connected to an input end of the image fusion module;

[0005] The pixel array module is used to generate a first image and a second image in response to an exposure instruction, send the first image to the first processing module through the first output terminal, and send the second image to the second processing module through the second output terminal, the first image includes a plurality of pixel units, each pixel unit is composed of a plurality of pixel points of the same color, the resolution of the first image is higher than the resolution of the second image, and the pixel points in the second image are arranged in a Bayer manner;

[0006] The first processing module is used to process the first image to obtain a third image, and send the third image to the image fusion module, the third image has the same number of pixels as the first image, and the pixels in the third image are arranged in a Bayer manner;

[0007] The second processing module is used to process the second image to obtain a fourth image, and send the fourth image to the image fusion module, wherein the resolution of the fourth image is the same as that of the third image;

[0008] The image fusion module is used to fuse the third image and the fourth image to obtain a fused image.

[0009] In a first possible implementation of the first aspect, the pixel array module is used to generate charges in response to the exposure instruction, obtain the first image by reading the generated charges, and merge the multiple pixel points included in each pixel unit in the first image to obtain the second image.

[0010] In a second possible implementation manner of the first aspect, the first processing module is used to rearrange pixels of the same color in each pixel unit by using a Remosaic algorithm to obtain the third image.

[0011] In a third possible implementation manner of the first aspect, the second processing module is configured to perform interpolation processing on the second image by using a preset interpolation algorithm to obtain the fourth image.

[0012] In a fourth possible implementation manner of the first aspect, the image fusion module is used to calculate the fused pixel value of the pixel at each position based on the pixel values ​​of the pixel at the same position of the third image and the fourth image; and generate the fused image based on the fused pixel value of the pixel at each position.

[0013] In a fifth possible implementation manner of the first aspect, the image fusion module is used to calculate an average value of pixel values ​​of pixels at the same position of the third image and the fourth image to obtain a fused pixel value of the pixel at each position; or

[0014] The image fusion module is used to perform interpolation calculation on the pixel values ​​of the pixel points at the same position of the third image and the fourth image to obtain the fused pixel value of the pixel points at each position.

[0015] In a sixth possible implementation manner of the first aspect, the image sensor further includes a downsampling module, and an input end of the downsampling module is connected to an output end of the image fusion module;

[0016] The downsampling module is used to perform downsampling processing on the fused image to obtain a downsampled fused image.

[0017] In a seventh possible implementation manner of the first aspect, the image sensor further includes an output module, and an output end of the downsampling module is connected to an input end of the output module;

[0018] The output module is used to output the downsampled fused image.

[0019] In a second aspect, an image processing method is provided, the method being applied to the image sensor according to the first aspect, the method comprising:

[0020] In response to an exposure instruction, a first image and a second image are generated, wherein the first image includes a plurality of pixel units, each pixel unit is composed of a plurality of pixel points of the same color, a resolution of the first image is higher than a resolution of the second image, and the pixel points in the second image are arranged in a Bayer manner;

[0021] Processing the first image to obtain a third image, wherein the third image has the same number of pixels as the first image, and the pixels in the third image are arranged in a Bayer manner;

[0022] Processing the second image to obtain a fourth image, wherein a resolution of the fourth image is the same as a resolution of the third image;

[0023] The third image and the fourth image are fused to obtain a fused image.

[0024] In a first possible implementation manner of the second aspect, generating the first image and the second image in response to the exposure instruction includes:

[0025] In response to the exposure instruction, charges are generated, the first image is obtained by reading the generated charges, and a plurality of pixel points included in each pixel unit in the first image are merged to obtain the second image.

[0026] In a second possible implementation manner of the second aspect, processing the first image to obtain a third image includes:

[0027] The remosaic algorithm is used to rearrange the pixels of the same color in each pixel unit to obtain the third image.

[0028] In a third possible implementation manner of the second aspect, the processing the second image to obtain a fourth image includes:

[0029] The second image is interpolated using a preset interpolation algorithm to obtain the fourth image.

[0030] In a fourth possible implementation manner of the second aspect, fusing the third image and the fourth image to obtain a fused image includes:

[0031] Calculating a fused pixel value of a pixel at each position based on the pixel values ​​of the pixel at the same position of the third image and the fourth image;

[0032] The fused image is generated based on the fused pixel values ​​of the pixels at each position.

[0033] In a fifth possible implementation manner of the second aspect, calculating, based on the pixel values ​​of the pixels at the same positions of the third image and the fourth image, the fused pixel value of the pixel at each position includes:

[0034] Calculate the average value of pixel values ​​of pixels at the same position of the third image and the fourth image to obtain a fused pixel value of the pixel at each position; or,

[0035] Interpolation calculation is performed on the pixel values ​​of the pixel points at the same position of the third image and the fourth image to obtain a fused pixel value of the pixel point at each position.

[0036] In a sixth possible implementation manner of the second aspect, the method further includes:

[0037] Downsampling is performed on the fused image to obtain a downsampled fused image.

[0038] In a seventh possible implementation manner of the second aspect, the method further includes:

[0039] The downsampled fused image is output.

[0040] In a third aspect, an electronic device is provided, comprising a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the image processing method as described in the second aspect.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and when the at least one computer program is executed by a processor, the second image processing method can be implemented.

[0042] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and the computer program, when executed by a processor, can implement the image processing method as described in the second aspect.

[0043] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0044] The embodiment of the present application provides an image sensor, the pixel array module of which can generate a high-resolution first image and a low-resolution second image in one exposure. For the generated first image, since the pixels in the first image are not arranged in a Bayer manner, it is necessary to use a first processing module to reorder the pixels of each pixel unit in the generated first image to obtain a third image in which the pixels are arranged in a Bayer manner. For the generated second image, since the resolution of the second image is lower than that of the first image, it is necessary to use a second processing module to perform interpolation processing to obtain a fourth image, and the resolution of the fourth image is the same as that of the third image. By using the image fusion module of the image sensor to fuse the third image and the fourth image, a fused image can be obtained. The fused image has the advantages of high clarity of a high-resolution image and low noise of a low-resolution image, and can meet the user's image quality requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0047] Figure 2 is a structural schematic diagram of an image sensor provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of a first image before being processed by a Remosaic algorithm and a third image after being processed, provided in an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of a third image provided in an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of an interpolated image obtained by interpolating an original image using different interpolation algorithms in an embodiment of the present application;

[0051] Figure 6 is a schematic diagram of an image obtained by interpolation calculation of an image in Binning mode outputted by an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of a fused image output by an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of the structure of another image sensor provided in an embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of an image before and after downsampling processing in an embodiment of the present application;

[0055] Fig.10 is a schematic diagram of the structure of another image sensor provided in an embodiment of the present application;

[0056] Fig.11 is a flow chart of an image processing method provided by an embodiment of the present application;

[0057] Fig.12 It is a block diagram of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0059] It can be understood that the terms "each", "multiple", and "any" used in the embodiments of the present application include two or more, each refers to each of the corresponding multiple, and any refers to any one of the corresponding multiple. For example, the multiple words include 10 words, and each word refers to each of the 10 words, and any word refers to any one of the 10 words.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0061] Figure 1 The structure diagram of the electronic device 100 is shown. The electronic device 100 may be a mobile phone, a personal computer (PC), a tablet computer, an AR (Augmented Reality) device, a VR (Virtual Reality) device, a vehicle-mounted computer, a wearable device, a smart home device, etc.

[0062] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, an image sensor 180M, etc.

[0063] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0064] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0065] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0066] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0067] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a User Identity Module interface, and / or a Universal Serial Bus (USB) interface, etc.

[0068] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to realize the touch function of the electronic device 100.

[0069] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0070] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0071] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0072] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0073] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0074] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, can also be used to transmit data between the electronic device 100 and a peripheral device, and can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0075] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0076] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0077] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0078] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.

[0079] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0080] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0081] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and may be set in the same device as the mobile communication module 150 or other functional modules.

[0082] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0083] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GNASS), a Beidou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and / or a Satellite Based Augmentation System (SBAS).

[0084] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0085] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0086] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0087] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0088] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0089] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0090] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0091] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0092] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0093] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, for example, at least one disk storage device, a flash memory device, a universal flash storage (Universal Flash Storage, UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0094] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0095] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0096] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0097] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0098] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0099] The earphone interface 170D is used to connect a wired earphone and can be the USB interface 130 or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0100] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0101] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (for example, taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example, time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0102] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0103] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0104] In order to improve image clarity and reduce image noise, an embodiment of the present application provides an image sensor. Figure 1 The image sensor 180M in the electronic device shown may be a CMOS sensor. Figure 2 The image sensor includes a pixel array module 201, a first processing module 202, a second processing module 203 and an image fusion module 204. The first output end of the pixel array module 201 is connected to the input end of the first processing module 202, the second output end of the pixel array module 201 is connected to the input end of the second processing module 203, and the output end of the first processing module 202 and the output end of the second processing module 203 are both connected to the input end of the image fusion module 204.

[0105] In the application embodiment, the pixel array module 201 can provide two image output modes, one is the full-size mode and the other is the Binning mode. The images generated by the full-size mode and the Binning mode have their own advantages and disadvantages, which are specifically manifested in the following aspects: first, the image generated by the full-size mode has a higher resolution, while the image generated by the Binning mode has a lower resolution; second, the image generated by the full-size mode has poor quality and relatively more noise, while the image generated by the Binning mode has better quality and relatively less noise; third, the pixels in the image generated by the full-size mode are not arranged in the Bayer manner, and the arrangement order of the pixels in the image needs to be adjusted, while the pixels in the image generated by the Binning mode are arranged in the Bayer manner, and the arrangement order of the pixels in the image does not need to be adjusted.

[0106] In a photo shooting scene, a preview scene or a video scene, when an exposure instruction sent by an electronic device is received, in response to the exposure instruction, the pixel array module 201 in the image sensor can generate a first image and a second image. The first image is an image generated by the pixel array module 201 in full-size mode. The first image includes a plurality of pixel units, each of which is composed of a plurality of pixel points of the same color. For example, each pixel unit may include 4 pixel points of the same color, 9 pixel points of the same color, or 16 pixel points of the same color. The number of pixel points of the same color included in each pixel unit can be set according to demand. Since the first image is an image generated in full-size mode, the pixel points in the first image are not arranged in a Bayer manner. The second image is an image generated by the pixel array module 201 in Binning mode, and the pixel points of the same color in the second image are dispersedly arranged. And the resolution of the first image is higher than the resolution of the second image. For example, the resolution of the first image can be 4800, and the resolution of the second image can be 1200. Since the resolution of the first image is higher than that of the second image, the clarity of the first image is higher than that of the second image. Accordingly, the signal-to-noise ratio of the first image is lower than that of the second image, and the noise of the first image is greater than that of the second image.

[0107] Specifically, in response to the exposure instruction, the process of the pixel array module 201 generating the first image and the second image can adopt the following method: in response to the exposure instruction, the pixel array module 201 generates a photoelectric effect, generates a charge, and then reads the generated charge using a preset scanning method, which can be row-by-row reading, column-by-column reading, etc. By reading the generated charge, the first image can be obtained. Then the pixel array module 201 merges the multiple pixel points included in each pixel unit in the first image to obtain the second image. If each pixel unit includes 4 pixel points, the 4 pixel points are merged into one pixel point; if each pixel unit includes 9 pixel points, the 9 pixel points are merged into one pixel point. Since each pixel point in the second image is a combination of multiple pixel points in the first image, the brightness of each pixel point in the second image is greater.

[0108] Further, after generating the first image and the second image, the pixel array module 201 sends the first image to the first processing module 202 through the first output terminal, and sends the second image to the second processing module 203 through the second output terminal.

[0109] In the embodiment of the present application, the first processing module 202 may be a Remosaic module, which may be a hardware module. The first processing module 202 may process the first image to obtain a third image, wherein the arrangement of pixels in the third image is different from that in the first image, the pixels in the first image are not arranged in a Bayer manner, and are not a RAW domain image in an RGB format, while the pixels in the third image are arranged in a Bayer manner, and are a RAW domain image in an RGB format.

[0110] Specifically, when the first processing module 202 processes the first image to obtain the third image, the following method can be used: the first processing module 202 uses the Remosaic algorithm to rearrange the pixels of the same color in each pixel unit to obtain the third image. The Remosaic algorithm is an algorithm that restores multiple pixels of the same color in a pixel unit to a Bayer arrangement. Figure 3 A schematic diagram of a first image before and after being processed by the Remosaic algorithm is shown, see Figure 3 , Figure 3 The left picture in the figure is the first image. The first image includes four pixel units. The first pixel unit includes 4 R pixels, the second pixel unit includes 4 G pixels, the third pixel unit includes 4 G pixels, and the fourth pixel unit includes 4 B pixels. After the first image is processed by the Remosaic algorithm, we can get Figure 3 In the third image shown in the right figure in FIG, pixels of the same color in the third image are no longer arranged in combination, but are dispersedly arranged in a Bayer manner.

[0111] In the embodiment of the present application, the number of pixels in the third image is the same as that in the first image, that is, the resolution of the third image is the same as that of the first image, and accordingly, the quality of the third image is also the same as that of the first image, and both may have some artifacts. Figure 4 The third image is generated by processing the first image using the first processing module 202, see Figure 4 The third image has high definition, but there are some artifacts in the image, and the image quality is poor.

[0112] Further, after obtaining the third image, the first processing module 202 sends the third image to the image fusion module 204 .

[0113] In the embodiment of the present application, the second processing module 203 may be an upsampling module (Upscaler module), and the second processing module 203 may process the second image to obtain a fourth image. The pixels in the fourth image are arranged in a Bayer manner, that is, the fourth image is a RAW domain image in RGB format, and the resolution of the fourth image is the same as the resolution of the third image. Since the resolution of the fourth image is the same as the resolution of the third image, when the third image and the fourth image are fused, the amount of fusion calculation can be reduced.

[0114] Specifically, the second processing module 203 may use a preset interpolation algorithm to perform interpolation processing on the second image to obtain the fourth image. The preset interpolation algorithm includes a nearest neighbor interpolation algorithm, a bilinear difference algorithm, a quadratic cubic interpolation algorithm, and the like. Figure 5 The interpolated images obtained by interpolating the original images using different interpolation algorithms are shown. Figure 5 It can be seen that images with the same resolution can be obtained by using different interpolation algorithms. Figure 6 FIG. 2 is a schematic diagram showing a fourth image obtained by the second processing module 203 using any interpolation algorithm to interpolate the second image output by the pixel array module 201, and FIG. Figure 4 Compared with the images in Figure 6 The image in the image is less clear, but Figure 6 The image has fewer artifacts and better image quality.

[0115] Further, after obtaining the fourth image, the second processing module 203 sends the fourth image to the image fusion module 204 .

[0116] In an embodiment of the present application, the image fusion module 204 can fuse the third image and the fourth image to obtain a fused image. Since the third image and the fourth image have the same resolution, that is, the third image and the fourth image include the same number of pixels, when the image fusion module 204 fuses the third image and the fourth image, it can fuse based on the pixels at the same position of the third image and the fourth image. Specifically, the image fusion module 204 can calculate the fused pixel value of the pixel at each position based on the pixel value of the pixel at the same position of the third image and the fourth image, and then generate a fused image based on the fused pixel value of the pixel at each position. When the image fusion module 204 calculates the fused pixel value of the pixel at each position based on the pixel value of the pixel at the same position of the third image and the fourth image, it can adopt the following method:

[0117] In the first method, the image fusion module calculates the average pixel value of the pixel points at the same position of the third image and the fourth image to obtain the fused pixel value of the pixel points at each position. For example, if the pixel value of a pixel point at any position of the third image is a, and the pixel value of the pixel point at the same position of the fourth image is b, then the fused pixel value of the pixel point at this position is (a+b) / 2.

[0118] In the second method, the image fusion module interpolates the pixel values ​​of the pixels at the same position of the third image and the fourth image to obtain the fused pixel value of the pixel at each position. For example, the pixel value of a pixel at any position of the third image is a, and the pixel value of the pixel at the same position of the fourth image is b. The pixel value a and the pixel value b are interpolated to obtain a pixel value c, and the fused pixel value of the pixel at this position is c.

[0119] The algorithms of the above two calculation methods are relatively simple and can meet the real-time requirements of image sensors for image processing.

[0120] By fusing the third image and the fourth image, the fused image has the clarity advantage of a high-resolution image and the low noise advantage of a low-resolution image, thereby meeting the image quality requirements of users. Figure 7 Shows the Figure 4 and Figure 6 The fused image obtained by fusing the images in Figure 4 Compared with the images in Figure 7 The image in Figure 4 The clarity of the image in , but with fewer artifacts than in 4, Figure 6 Compared with the images in Figure 7 The image in Figure 6 The advantage of less artifacts is that Figure 6 The clarity of the image should be high.

[0121] The image sensor provided in the embodiment of the present application performs image fusion in the RAW domain. Since the RAW domain image itself contains more information, the image quality is better, and the image fusion occurs on the chip. The back end can directly output the image without using heavy algorithms for fusion, thereby reducing the processing pressure on the back end.

[0122] Since the fused image has a high resolution, large data volume, long transmission time, and consumes more resources, in order to shorten the transmission time of the fused image and save the consumption of resources during the transmission process, see Figure 8 The image sensor further includes a downsampling module 205, the input end of which is connected to the output end of the image fusion module 204, and the downsampling module 205 is used to downsample the fused image to obtain a downsampled fused image. When downsampling the fused image, the downsampling module 205 may use a downsampling algorithm to reduce the resolution of the fused image to a preset resolution. The downsampling algorithm may be a bilinear interpolation algorithm, a neighboring interpolation algorithm, etc. The preset resolution may be set according to the user's clarity requirements. Fig. 9 2 is an image before and after being processed by the downsampling module 205, wherein Fig. 9 The left picture in the figure is the fused image before downsampling. Fig. 9 The right picture in the figure is the fused image after downsampling. Fig. 9 It can be seen from the left and right figures in that the size of the fused image after downsampling is smaller than that of the fused image before downsampling. The transmission time of the fused image after downsampling will be shorter and the transmission resource consumption will be less.

[0123] See also Fig.10 The image sensor further includes an output module 206, and an input end of the output module 206 is connected to an output end of the downsampling module 205. When a downsampled fused image is obtained, the downsampling module 205 sends the downsampled fused image to the output module 206. After receiving the downsampled fused image, the output module 206 can output the downsampled fused image so that the electronic device can display the downsampled fused image.

[0124] Of course, in addition to Figure 2 or Figure 8 or Fig.10 In addition to the functional modules shown, the image sensor may also include other modules, which will not be described one by one here.

[0125] The embodiment of the present application provides an image sensor, the pixel array module of which can generate a high-resolution first image and a low-resolution second image in one exposure. For the generated first image, the pixels in the first image are not arranged in a Bayer manner, and the first processing module is required to reorder the pixels of each pixel unit in the generated first image to obtain a third image in which the pixels are arranged in a Bayer manner. For the generated second image, since the resolution of the second image is lower than that of the first image, the second processing module is required to perform interpolation processing to obtain a fourth image, and the resolution of the fourth image is the same as that of the third image. By using the image fusion module of the image sensor to fuse the third image and the fourth image, a fused image can be obtained. The fused image has the advantages of high clarity of a high-resolution image and low noise of a low-resolution image, and can meet the user's image quality requirements.

[0126] The image sensor provided in the embodiment of the present application can be set in any electronic device with a camera function to improve the clarity of the image of the electronic device in the camera scene, preview scene and video scene, and reduce the noise of the image.

[0127] The present application provides an image processing method for Figure 2 or Figure 8 or Fig.10 The image sensor shown is shown in Fig.11 , the method flow provided in the embodiment of the present application includes:

[0128] 1101. In response to an exposure instruction, generate a first image and a second image.

[0129] In an embodiment of the present application, when an exposure instruction is received from an electronic device, the pixel array module of the image sensor outputs the image in full-size mode to obtain a first image, and outputs the image in Binning mode to obtain a second image. The first image includes a plurality of pixel units, each of which is composed of a plurality of pixel points of the same color. The resolution of the first image is higher than that of the second image, and the pixel points in the first image are not arranged in a Bayer manner, while the pixel points in the second image are arranged in a Bayer manner.

[0130] 1102. Process the first image to obtain a third image.

[0131] The first processing module of the image sensor processes the first image using a Remosaic algorithm to obtain a third image, wherein the resolution of the third image is the same as that of the first image, but the arrangement of pixels in the third image and the first image is different, the pixels in the first image are not arranged in a Bayer manner, and the pixels in the third image are arranged in a Bayer manner. By processing the first image using the first processing module, the first image can be converted into a RAW domain image in RGB format.

[0132] 1103. Process the second image to obtain a fourth image.

[0133] The second processing module of the image sensor processes the second image using a preset interpolation algorithm to obtain a fourth image. The resolution of the fourth image is the same as that of the third image, and pixels in the fourth image are arranged in a Bayer manner.

[0134] It should be noted that, although the first processing module processes the first image to obtain the third image as step 1002, and the second processing module processes the second image to obtain the fourth image as step 1003, the above steps 1002 and 1003 can be executed successively or simultaneously.

[0135] 1104. Fuse the third image and the fourth image to obtain a fused image.

[0136] The image fusion module of the image sensor calculates the fused pixel value of the pixel at each position based on the pixel values ​​of the pixel at the same position in the third image and the fourth image, and then generates a fused image based on the fused pixel value of the pixel at each position. The fused image has the clarity advantage of a high-resolution image and the low noise advantage of a low-resolution image, and can meet users' requirements for image quality.

[0137] Furthermore, after obtaining the fused image, the fusion module of the image sensor can input the fused image into the downsampling module, and the downsampling module downsamples the fused image to obtain a downsampled fused image, and then send the downsampled fused image to the output module, and the output module outputs the downsampled fused image.

[0138] Fig.12 A flowchart of the image processing method provided by the embodiment of the present application is shown. Fig.12 , the process consists of four steps:

[0139] Step 1: In response to an exposure instruction, the pixel array module outputs images in full-size mode to obtain a full-size image (ie, a first image), and outputs images in Binning mode to obtain a Binning image (ie, a second image).

[0140] Step 2: The first processing module processes the full-size image using a remosaic algorithm to obtain a remosaic processed image. The second processing module processes the binning image using an interpolation algorithm to obtain an interpolated image.

[0141] Step 3: The image fusion module fuses the remosaic processed image and the interpolated image to obtain a fused full-size image (ie, a fused image).

[0142] Step 4: The downsampling module downsamples the fused full-size image to obtain a downsampled image (ie, the downsampled fused image), and then outputs the downsampled image.

[0143] An embodiment of the present application provides a computer-readable storage medium, in which at least one computer program is stored. When the at least one computer program is executed by a processor, the image processing method described in the above embodiment can be implemented.

[0144] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image processing method described in the above embodiment can be implemented.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image sensor, characterized in that: The image sensor comprises a pixel array module, a first processing module, a second processing module, an image fusion module and a downsampling module, wherein a first output end of the pixel array module is connected to an input end of the first processing module, a second output end of the pixel array module is connected to an input end of the second processing module, an output end of the first processing module and an output end of the second processing module are both connected to an input end of the image fusion module, and an input end of the downsampling module is connected to an output end of the image fusion module; The pixel array module is used to generate a first image and a second image in response to an exposure instruction, send the first image to the first processing module through the first output terminal, and send the second image to the second processing module through the second output terminal, the first image includes a plurality of pixel units, each pixel unit is composed of a plurality of pixel points of the same color, the resolution of the first image is higher than the resolution of the second image, and the pixel points in the second image are arranged in a Bayer manner; The first processing module is used to process the first image to obtain a third image, and send the third image to the image fusion module, the third image has the same number of pixels as the first image, and the pixels in the third image are arranged in a Bayer manner; The second processing module is used to process the second image to obtain a fourth image, and send the fourth image to the image fusion module, wherein the resolution of the fourth image is the same as that of the third image; The image fusion module is used to calculate the average value of the pixel values ​​of the pixel points at the same position of the third image and the fourth image, obtain the fused pixel value of the pixel points at each position, and generate a fused image based on the fused pixel value of the pixel points at each position; or perform interpolation calculation on the pixel values ​​of the pixel points at the same position of the third image and the fourth image, obtain the fused pixel value of the pixel points at each position, and generate a fused image based on the fused pixel value of the pixel points at each position; The downsampling module is used to perform downsampling processing on the fused image to obtain a downsampled fused image.

2. The image sensor according to claim 1, characterized in that The pixel array module is used to generate charges in response to the exposure instruction, obtain the first image by reading the generated charges, and merge multiple pixel points included in each pixel unit in the first image to obtain the second image.

3. The image sensor according to claim 1, characterized in that The first processing module is used to rearrange the pixels of the same color in each pixel unit by using a Remosaic algorithm to obtain the third image.

4. The image sensor according to claim 1, characterized in that The second processing module is used to perform interpolation processing on the second image by using a preset interpolation algorithm to obtain the fourth image.

5. The image sensor according to claim 1, characterized in that The image sensor further comprises an output module, and the output end of the downsampling module is connected to the input end of the output module; The output module is used to output the downsampled fused image.

6. An image processing method, characterized in that: The method is applied to the image sensor according to any one of claims 1 to 5, and the method comprises: In response to an exposure instruction, a first image and a second image are generated, wherein the first image includes a plurality of pixel units, each pixel unit is composed of a plurality of pixel points of the same color, a resolution of the first image is higher than a resolution of the second image, and the pixel points in the second image are arranged in a Bayer manner; Processing the first image to obtain a third image, wherein the third image has the same number of pixels as the first image, and the pixels in the third image are arranged in a Bayer manner; Processing the second image to obtain a fourth image, wherein a resolution of the fourth image is the same as a resolution of the third image; Calculating a fused pixel value of a pixel at each position based on the pixel values ​​of the pixel at the same position of the third image and the fourth image; Generate the fused image based on the fused pixel values ​​of the pixels at each position; Downsampling the fused image to obtain a downsampled fused image; The calculating, based on the pixel values ​​of the pixels at the same position of the third image and the fourth image, the fused pixel value of the pixel at each position comprises: Calculate the average value of pixel values ​​of pixels at the same position of the third image and the fourth image to obtain a fused pixel value of the pixel at each position; or, Interpolation calculation is performed on the pixel values ​​of the pixel points at the same position of the third image and the fourth image to obtain a fused pixel value of the pixel point at each position.

7. The method according to claim 6, characterized in that The step of generating the first image and the second image in response to the exposure instruction comprises: In response to the exposure instruction, charges are generated, the first image is obtained by reading the generated charges, and a plurality of pixel points included in each pixel unit in the first image are merged to obtain the second image.

8. The method according to claim 6, characterized in that The step of processing the first image to obtain a third image includes: The remosaic algorithm is used to rearrange the pixels of the same color in each pixel unit to obtain the third image.

9. The method according to claim 6, characterized in that The processing of the second image to obtain a fourth image includes: The second image is interpolated using a preset interpolation algorithm to obtain the fourth image.

10. The method according to claim 6, characterized in that The method further comprises: The downsampled fused image is output.

11. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the image processing method as described in any one of claims 6 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, it can implement the image processing method according to any one of claims 6 to 10.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 6 to 10 can be implemented.

Images