Image fusion method, chip, electronic device and storage medium

Abstract

Embodiments of the present application relate to the technical field of image processing, and particularly relate to an image fusion method, a chip, an electronic device and a storage medium. In the method, in response to a shooting operation, a camera of the electronic device collects a first raw image and a second raw image; the first raw image and the second raw image are collected by the camera through one exposure process. That is, the camera can obtain the first raw image and the second raw image through a single-exposure multiple-reading mode. Next, the electronic device calculates an image light ratio according to pixel values of pixel points of the first raw image and pixel values of pixel points of the second raw image. Then, the electronic device performs image fusion on the first raw image and the second raw image according to the image light ratio to obtain a fused raw image. Finally, the electronic device displays the fused raw image. Through the method, the signal-to-noise ratio of the obtained fused image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image fusion method, chip, electronic device and storage medium. Background Technology

[0002] Currently, electronic devices can fuse multiple images to obtain a fused image.

[0003] At present, how to improve the quality of the fused image obtained by electronic devices through image fusion is a problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image fusion method, a chip, an electronic device, and a storage medium, which can improve the quality of the fused image obtained by image fusion.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, an image fusion method is provided, applied to an electronic device including a camera. The method includes: in response to a shooting operation, the camera acquires a first raw image and a second raw image; the first and second raw images are acquired by the camera through a single exposure process. That is, the camera can obtain the first and second raw images through a single-exposure, multiple-read mode. Next, the electronic device calculates the image light ratio using the pixel values ​​of the first and second raw images. Then, the electronic device performs image fusion on the first and second raw images based on the image light ratio to obtain a fused raw image. Finally, the electronic device displays the fused raw image.

[0007] In the above method, the image light ratio is calculated by the electronic device using the pixel values ​​of the pixels in the images to be fused. Therefore, the image light ratio is more consistent with the actual shooting scene. By using a light ratio that better reflects the actual shooting scene, the exposure alignment effect of the images to be fused can be improved during the image fusion process. This method can improve the quality of the fused image, such as increasing its signal-to-noise ratio.

[0008] In one possible design of the first aspect, the brightness of the first raw image is higher than the brightness of the second raw image. The electronic device calculates the image light ratio using the pixel values ​​of the pixels in the first raw image and the pixel values ​​of the pixels in the second raw image, including: the electronic device filters the pixels of the first raw image to obtain target pixels. Then, the electronic device calculates the ratio of each target pixel to its corresponding pixel in the second raw image to obtain the light ratio of each target pixel. Finally, the electronic device uses the average of the light ratios of each target pixel as the image light ratio.

[0009] In this design, the electronic device filters the first raw image, removing pixels that are unsuitable for calculating the image's light ratio. This further improves the match between the image's light ratio and the actual shooting scene, and further enhances the quality of the fused image.

[0010] In another possible design of the first aspect, the electronic device filters the pixels of the first raw image to obtain target pixels, including: the electronic device calculates the light ratio of the pixels of the first raw image. Next, the electronic device removes pixels from the first raw image whose light ratio is an outlier to obtain the target pixels.

[0011] In this design, the electronic device removes pixels with outlier light ratios, which can further improve the consistency between the image light ratio and the actual shooting scene; it can also further improve the quality of the fused image obtained by image fusion.

[0012] In another possible design of the first aspect, the method further includes: the electronic device removing pixels with pixel values ​​greater than overexposure values ​​from the pixels of the first raw image to obtain non-overexposure pixels of the first raw image. The electronic device calculating the light ratio of the pixels of the first raw image includes: the electronic device calculating the light ratio of the non-overexposure pixels. The electronic device removing pixels with outlier light ratios from the pixels of the first raw image to obtain target pixels includes: the electronic device removing pixels with outlier light ratios from the non-overexposure pixels to obtain target pixels.

[0013] In this design, the electronic device removes overexposed pixels, which can further improve the consistency between the image's light ratio and the actual shooting scene; it can also further improve the quality of the fused image obtained from image fusion.

[0014] In another possible design of the first aspect, the electronic device removes pixels with outliers in the light ratio from the non-overexposed pixels to obtain the target pixel, including: the electronic device removes pixels with outliers in the light ratio from the non-overexposed pixels by using the standard deviation of the light ratio of the non-overexposed pixels to obtain the target pixel.

[0015] In this design, electronic devices can efficiently remove pixels with outliers in terms of light ratio by using standard deviation.

[0016] In another possible design of the first aspect, the aforementioned electronic device obtains the target pixel by removing pixels with outliers in their light ratios using the standard deviation of the light ratios of non-overexposed pixels. This includes: the electronic device constructing an inner point set, which includes each non-overexposed pixel; the electronic device then iteratively performing a standard deviation filtering process on the inner point set until the number of times the standard deviation filtering process is performed equals a preset number, or until the inner point set converges; and after the electronic device finishes iteratively performing the standard deviation filtering process on the inner point set, the electronic device uses the pixels included in the inner point set as the target pixel. The standard deviation filtering process includes: the electronic device calculating the light ratio of each pixel included in the inner point set; then, the electronic device calculating the mean of the light ratios of the pixels included in the inner point set; then, the electronic device calculating the standard deviation of the light ratios of the pixels included in the inner point set; and finally, the electronic device removing pixels in the inner point set whose light ratios are greater than the mean + N times the standard deviation, and removing pixels in the inner point set whose light ratios are less than the mean - N times the standard deviation; where N is a positive integer.

[0017] In this design, the electronic device can further improve the consistency between the image light ratio and the actual shooting scene by repeatedly performing a standard deviation filtering process; it can also further improve the quality of the fused image obtained by image fusion.

[0018] In another possible design of the first aspect, the electronic device performs image fusion on the first raw image and the second raw image according to the image light ratio to obtain a fused raw image, including: when the exposure level of a pixel in the first raw image is greater than a first level, the electronic device multiplies the pixel value of the second raw image by the image light ratio to obtain the pixel value of the fused raw image. Next, when the exposure level of a pixel in the first raw image is less than or equal to the first level, the electronic device uses the pixel value of the first raw image as the pixel value of the fused raw image.

[0019] In this design, the electronic device can obtain a fused raw image based on different images to be fused when the first raw image is exposed to different degrees, which can further improve the quality of the fused image.

[0020] In another possible design of the first aspect, the exposure level of the pixels of the first raw image being greater than the first level includes: the pixel value of the pixels of the first raw image being greater than the fusion threshold. And, the exposure level of the pixels of the first raw image being less than or equal to the first level includes: the pixel value of the pixels of the first raw image being less than or equal to the fusion threshold. Wherein, the fusion threshold is A*(2... M -1); A takes the value (0, 1], and M is the bit width of the first raw image.

[0021] In a second aspect, an electronic device is provided, the electronic device including a memory and one or more processors, the memory being coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method provided by the first aspect and any possible design of the first aspect.

[0022] Thirdly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods provided by the first aspect and any possible design of the first aspect.

[0023] Fourthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to perform the methods provided by the first aspect and any possible design of the first aspect.

[0024] Fifthly, a chip system is provided for use in an electronic device, the chip system including one or more processors for invoking computer instructions to cause the electronic device to perform the methods provided by the first aspect and any possible design of the first aspect.

[0025] Sixthly, a chip is provided, comprising a processing circuit and an interface circuit, the interface circuit being coupled to the processing circuit. The interface circuit is configured to: receive a third raw image and a fourth raw image; the third raw image and the fourth raw image are acquired by a camera through a single exposure process; the processing circuit is configured to: calculate an image light ratio using the pixel values ​​of the pixels in the third raw image and the pixel values ​​of the pixels in the fourth raw image; and perform image fusion on the third raw image and the fourth raw image based on the image light ratio to obtain a fused raw image.

[0026] The technical effects of any of the design methods in aspects two through six can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0027] Figure 1A flowchart illustrating an image fusion method provided in an embodiment of this application;

[0028] Figure 2 A flowchart illustrating another image fusion method provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram illustrating the effect of image fusion provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram illustrating the signal-to-noise ratio of a fused image, provided for an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0032] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of this application;

[0033] Figure 7 A schematic flowchart illustrating the image fusion method provided in this application embodiment;

[0034] Figure 8 A schematic diagram illustrating the actual light ratio calculation process provided for embodiments of this application;

[0035] Figure 9 A schematic diagram of a chip (system) provided in an embodiment of this application;

[0036] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0037] Figure 11 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0039] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0040] The technical solutions disclosed in this application involve the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0041] With the development of technology, users have increasingly higher requirements for image quality. Currently, electronic devices can fuse multiple images to create a merged image.

[0042] For example, an electronic device's image sensor can read out images multiple times during a single exposure in response to a shooting command. The electronic device then fuses these multiple readouts to obtain a fused image. This image sensor can also be referred to as a single exposure multiple read (SEMR) mode sensor.

[0043] The following section uses the example of an electronic device fusing two images to introduce some solutions provided in the embodiments of this application.

[0044] In some of the solutions provided in the embodiments of this application, electronic devices can use an image fusion method with theoretical light ratio to fuse images.

[0045] For example, see Figure 1 Image 1's pixel block includes two green pixels, one blue pixel, and one red pixel; each pixel corresponds to a preset weight. Similarly, Image 2's pixel block includes two green pixels, one blue pixel, and one red pixel; each pixel corresponds to a preset weight. Then, the electronic device multiplies the pixel value of each pixel by its respective weight. Next, the electronic device multiplies the weighted pixel values ​​of Image 2 by the theoretical light ratio. Finally, the pixel values ​​of Image 1 and Image 2, after the above processing, are added together to obtain the pixel values ​​of the pixel block of the fused image.

[0046] For example, the pixel blocks of image 1 include B1, R1, G1a, and G1b; the pixel blocks of image 2 include B2, R2, G2a, and G2b. The pixel blocks of the merged image include B3, R3, G3a, and G3b.

[0047] Here, Image 1 and Image 2 can be two images read from the same exposure by a sensor in SEMR mode, with different exposure times for Image 1 and Image 2. The theoretical light ratio can be calculated using relevant methods based on the exposure times of Image 1 and Image 2. For example, the theoretical light ratio can be the exposure time of Image 1 divided by the exposure time of Image 2. Alternatively, the theoretical light ratio can be preset.

[0048] from Figure 1 visible:

[0049] The pixel value of G3a = the pixel value of G1a * W1 + the pixel value of G2a * (1-W1) * theoretical light ratio;

[0050] The pixel value of B3 = the pixel value of B1 * W2 + the pixel value of B2 * (1-W2) * theoretical light ratio;

[0051] The pixel value of R3 = the pixel value of R1 * W3 + the pixel value of R2 * (1 - W3) * theoretical light ratio;

[0052] The pixel value of G3b = the pixel value of G1b * W4 + the pixel value of G2b * (1-W4) * theoretical light ratio.

[0053] Among them, at least two of W1, W2, W3 and W4 are different.

[0054] In other solutions provided in the embodiments of this application, the electronic device may use an image fusion method that guarantees color ratio to fuse images.

[0055] For example, see Figure 2 Image 1's pixel block consists of two green pixels, one blue pixel, and one red pixel; each pixel corresponds to a preset weight. Similarly, Image 2's pixel block consists of two green pixels, one blue pixel, and one red pixel; each pixel corresponds to a preset weight. The electronic device then multiplies the pixel value of each pixel by the preset weight. Next, the electronic device multiplies the weighted pixel values ​​of Image 2 by the theoretical light ratio. Finally, the pixel values ​​of Image 1 and Image 2, after the above processing, are added together to obtain the pixel values ​​of the pixel block of the fused image.

[0056] For example, the pixel blocks of image 1 include B4, R4, G4a, and G4b; the pixel blocks of image 2 include B5, R5, G5a, and G5b. The pixel blocks of the merged image include B6, R6, G6a, and G6b.

[0057] from Figure 2 visible:

[0058] The pixel value of G6a = the pixel value of G4a * W + the pixel value of G5a * (1-W) * theoretical light ratio;

[0059] The pixel value of B6 = the pixel value of B4 * W + the pixel value of B5 * (1-W) * theoretical light ratio;

[0060] The pixel value of R6 = the pixel value of R4 * W + the pixel value of R5 * (1-W) * theoretical light ratio;

[0061] The pixel value of G6b = the pixel value of G4b * W + the pixel value of G5b * (1-W) * theoretical light ratio.

[0062] It should be noted that the above Figure 1 and Figure 2 The corresponding description uses binning format images as an example. The image fusion method provided in this application does not impose any restrictions on the image format. That is to say, the image fusion method provided in this application is not only applicable to binning format images, but also applicable to hex format, quad format images, and some future evolved image formats.

[0063] It should be understood that the above Figure 1 and Figure 2 The corresponding solution is illustrated by merging two images; in practical use, the above... Figure 1 / Figure 2 The corresponding solution can also fuse more than two images, as long as the above solution is applied to each image. For example, to fuse images 1, 2, and 3, you can first fuse images 1 and 2. Figure 1 The corresponding schemes are then fused to obtain fused image 1. Next, fused image 1 and image 3 are fused to obtain fused image 2. Similarly, Figure 2 The corresponding solution for fusing two images is similar and will not be elaborated upon.

[0064] In some application scenarios, the fused image obtained by the above-mentioned image fusion method using theoretical light ratio may have color deviations due to the possible discrepancy between the theoretical light ratio used and the shooting scene.

[0065] For example, see Figure 3 Image 1 is a fused image obtained using the theoretical light ratio image fusion method, while image 2 is a fused image obtained using the image fusion method that maintains color proportions. Image 1 uses a diagonal fill to indicate that the object has an overall greenish tint.

[0066] It should be understood that, for Figure 2 The corresponding solution uses the same weight for all pixels within a pixel block. This mitigates potential color discrepancies caused by deviations between the theoretical light ratio and the shooting scene. Therefore, employing an image fusion method that preserves color proportions can alleviate color deviation to some extent.

[0067] In some use cases, the fused image obtained by the image fusion method described above, which guarantees color ratio, may have a relatively high signal-to-noise ratio.

[0068] For example, see Figure 4 , Figure 4 The image shows the signal-to-noise ratio (SNR) of the blue pixel (B) channel in a fused image obtained by fusing a short-exposure image and a long-exposure image using the aforementioned image fusion method that preserves color ratio. Both the short-exposure and long-exposure images were obtained from the same image sensor using SEMR. As can be seen from the image, the B channel exhibits a deteriorated SNR because the short-exposure image was incorporated into areas with low pixel signal (e.g., low pixel value). This results in poor exposure rate alignment between the short-exposure image and the long-exposure image based on light ratio, leading to a lower SNR in the fused image.

[0069] In view of this, embodiments of this application provide an image fusion method in which an electronic device acquires at least two images to be fused. These at least two images are obtained by an image sensor using SEMR mode. The electronic device then calculates the light ratio based on the pixel values ​​of the pixels in the at least two images. Finally, the electronic device performs image fusion on the two images based on the light ratio to obtain a fused image. In this method, the light ratio is calculated based on the images to be fused, thus more closely resembling the actual shooting scene. By using a light ratio more consistent with the actual shooting scene for image fusion, the exposure rate alignment effect of the images to be fused can be improved during the image fusion process. This method can improve the quality of the fused image. For example, it can improve the signal-to-noise ratio of the fused image; and it can alleviate color anomalies in the fused image.

[0070] The aforementioned electronic devices can also be referred to as terminals, including terminal equipment, user equipment (UE), mobile station (MS), and mobile terminal (MT). The aforementioned electronic device 100 can be a mobile phone, tablet computer, wearable device, smart screen, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other electronic devices with image processing capabilities. Optionally, the aforementioned electronic device may also have a camera function. This application does not impose any limitations on the product form of the electronic device.

[0071] Next, the hardware structure and configuration of the electronic device provided in the embodiments of this application will be described.

[0072] Figure 5 A schematic diagram of the hardware structure of electronic device 100 is shown. Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, camera 193, and display screen 194, etc. The camera 193 and the display screen 194 are optional.

[0073] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0074] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0075] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0076] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0077] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0078] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0079] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0080] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0081] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0082] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0083] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0084] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0085] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0086] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0087] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0088] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0089] It should be noted that the image fusion method provided in this application embodiment can be executed independently by a processor, which can be an AP, an ISP, or other processors that will emerge from future technological evolutions. When the ISP executes the image fusion method provided in this application embodiment, the ISP can implement the relevant steps in the image fusion method through its internal circuitry and the connection relationships of the internal circuitry. For example, the ISP may include an interface circuit and a processing circuit, the interface circuit being configured to receive at least two images to be fused. The processing circuit is configured to: calculate the light ratio based on the pixel values ​​of the pixels included in the at least two images to be fused. Next, image fusion is performed on the two images to be fused according to the light ratio to obtain a fused image. Further description of this method is provided below and will not be repeated here.

[0090] Furthermore, the image fusion method provided in this application embodiment can also be applied to electronic devices with a processor and a camera. For example, the camera of the electronic device acquires at least two images to be fused using SEMR. Next, the processor of the electronic device obtains the light ratio based on the pixel values ​​of the pixels included in the at least two images to be fused. Then, the processor of the electronic device performs image fusion on the two images to be fused based on the light ratio to obtain a fused image.

[0091] After introducing the hardware structure of the electronic device provided in the embodiments of this application, the architecture of the electronic device will be described.

[0092] The architecture of the electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture based on Android. TM Taking the system as an example, the architecture of electronic device 100 is illustrated.

[0093] For example, see Figure 6 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, Android... TM The system is divided into four layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), and kernel layer.

[0094] The application layer can include a series of application packages.

[0095] like Figure 6 As shown, an application package can include application packages for applications such as camera applications and gallery applications.

[0096] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0097] like Figure 6 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, etc.

[0098] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0099] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0100] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0101] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0102] The application framework layer may also include a camera interface. The camera interface provides communication capabilities between software modules at the application layer and those at the lower levels.

[0103] The Hardware Abstraction Layer (HAL) is an abstraction layer that sits between the hardware and higher layers. It provides a unified interface to the upper layers, allowing applications to work without needing to know the specifics of how the underlying hardware works, thus shielding them from the underlying implementation details.

[0104] The Hardware Abstraction Layer (HAL) provides a standard interface to expose device hardware functionality to higher-level application framework layers. The HAL contains multiple library modules, each implementing an interface for a specific type of hardware component. These library modules might include a camera module, and so on. When the application framework layer requests access to the device hardware, the system loads the corresponding library module for that hardware component. Manufacturers can define interfaces within the HAL.

[0105] Furthermore, the camera module may include a High-Dynamic Range (HDR) capture module, which provides the ability to generate HDR images. The functions of each of these modules will be described in detail in subsequent examples.

[0106] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, etc.

[0107] In some solutions, the application layer, application framework layer, HAL, and kernel layer described above can all be deployed on the processor of the electronic device. To facilitate the description of the image fusion method provided in the embodiments of this application within the architecture, a hardware layer may also be included below the kernel layer. The hardware layer may include hardware that is independent of the processor, such as a camera.

[0108] For example, see Figure 6 The image fusion method provided in the embodiments of this application will be introduced in conjunction with the shooting scenario of electronic devices.

[0109] After the camera app is launched, in response to the user's shooting action, the camera app sends a shooting command to the camera module via the camera interface. Then, the camera module, according to the shooting command, uses the camera driver to capture images. The camera captures at least two raw images in a single exposure. Next, the camera driver sends these two raw images to the camera module. The camera module calculates the light ratio based on the pixel values ​​of the pixels in the at least two raw images. Then, the camera module performs image fusion on the two images to be merged based on the light ratio to obtain a merged raw image. Next, the camera module sends this merged raw image to the camera app via the camera interface. The camera app displays the merged raw image in the shooting preview interface. Optionally, the gallery app can also store the merged raw image.

[0110] Below, we will take a mobile phone, which includes a camera, as an example of an electronic device. The mobile phone has the aforementioned features. Figure 5 The hardware structure shown and the above Figure 6 Taking the illustrated software architecture as an example, the image fusion method provided in this application embodiment will be further described.

[0111] For example, see Figure 7 The image fusion method provided in this application embodiment may include steps S700-S703.

[0112] S700. In response to a shooting operation, the camera acquires a first raw image and a second raw image.

[0113] The aforementioned shooting operation can be triggered by the user pressing the shooting button on the shooting preview interface. Alternatively, the shooting operation can be a user's voice command, such as "take a photo." Or, the shooting operation can be performed by other user actions; this application embodiment does not impose any limitations on the shooting operation.

[0114] As one possible implementation, the camera can acquire a first raw image and a second raw image using SEMR mode. For example, the camera can implement SEMR mode using lateral overflow integration capacitor (LOFIC) technology. Alternatively, the camera can implement SEMR mode using dual conversion gain (DCG) technology. Yet another example is the camera's use of dual analog gain (DAG) technology.

[0115] Below is an example of how to implement the SEMR mode. The SEMR mode can be divided into an exposure stage, a first readout stage, and a second readout stage.

[0116] Regarding the exposure phase: During the exposure phase, the entire sensor array of the camera is exposed, accumulating light signals. In this process, photons are converted into electrons by the photodiodes corresponding to the pixels.

[0117] Regarding the first readout phase: After exposure, the camera performs its first readout of the electrons accumulated in the photodiodes corresponding to each pixel. At this point, the number of electrons in the photodiodes is relatively large, resulting in a strong readout signal. The raw image obtained by the camera during this first readout phase can be called a long-exposure raw image. After the electrons are read out, the photodiodes corresponding to each pixel are not completely emptied; a portion is retained. This portion of electrons can be partially removed using a rapidly reversing charge.

[0118] Regarding the second readout stage: A second readout stage is performed after the first. Since the electrons in the pixels have been partially cleared, the signal read out this time is weaker. The raw image obtained by the camera in the second readout stage can be called a short-exposure raw image.

[0119] In other implementations, the camera can read out the electrons accumulated in the photodiodes corresponding to each pixel multiple times after the initial readout phase. A partial clearing operation is performed after each readout. The signal read out each time is weaker than the previous one.

[0120] In some examples, the camera can capture images in response to a shooting command sent by the mobile phone. Therefore, step S700 above includes: in response to the shooting command, the camera captures a first raw image and a second raw image through a single exposure process; the brightness of the first raw image is higher than the brightness of the second raw image.

[0121] The S701 camera module calculates the actual light ratio using the first raw image and the second raw image.

[0122] The actual light ratio can also be called the shooting light ratio, the true light ratio, the image light ratio, etc. It can be understood as the light ratio obtained based on the actual shooting scene factors, such as the light ratio calculated based on the pixel values ​​of the captured image.

[0123] It should be understood that the first and second raw images mentioned above can be acquired by the camera through SEMR mode, in which case the pixels of the first and second raw images are aligned. That is, there is a one-to-one correspondence between the pixels of the first and second raw images.

[0124] In some other implementations, before calculating the actual light ratio, such as when the first raw image and the second raw image are not pixel-aligned, the camera module may also perform a pixel alignment operation on the first raw image and the second raw image, and then calculate the actual light ratio.

[0125] As one possible implementation, during the process of calculating the actual light ratio using the first and second raw images, the camera module can filter out some low-quality pixels. It should be understood that the raw images captured by the camera will contain some dead pixels and noise; therefore, filtering out this noise during the calculation of the actual light ratio can improve the accuracy of the calculated actual light ratio.

[0126] As a possible example, the first and second raw images are captured by the camera in a single exposure process. The first raw image can be the raw image read out by the camera during the first readout phase, and the second raw image can also be the raw image read out by the camera during the first readout phase. As can be seen from the above description, the number of electrons in the photosensitive diode is relatively large during the first readout phase. Therefore, the exposure time corresponding to the first raw image is longer than that of the second raw image. In other words, the brightness of the first raw image is higher than that of the second raw image.

[0127] The exposure time for the first raw image is higher than that for the second raw image. The camera module can remove overexposed pixels from the first raw image. In other words, the camera module calculates the actual light ratio based on the non-overexposed pixels in both the first and second raw images.

[0128] It should be understood that since the brightness of the first raw image is higher than that of the second raw image, any overexposed pixels in the second raw image will also be overexposed in the first raw image. Therefore, by removing overexposed pixels from the first raw image, overexposed pixels in the second raw image can be removed simultaneously, thus improving the efficiency of image fusion.

[0129] For example, the bit width of the first and second raw images is 10 bits, meaning that the pixel values ​​of the pixels in the first and second raw images range from [0, 1023]. The camera module can classify pixels with values ​​greater than or equal to 1022 as overexposed pixels and pixels with values ​​less than 1022 as non-overexposed pixels. Alternatively, the camera module can classify pixels with values ​​greater than or equal to 1000 as overexposed pixels and pixels with values ​​less than 1000 as non-overexposed pixels.

[0130] Specifically, the camera module can calculate the ratio of the pixel value of each non-overexposed pixel in the first raw image to the pixel value of the corresponding non-overexposed pixel in the second raw image, thus obtaining the light ratio of each non-overexposed pixel. Then, the average light ratio of each non-overexposed pixel is calculated to obtain the actual light ratio.

[0131] In this approach, the camera module can remove pixels from overexposed areas of the long-exposure image (e.g., the first raw image) and calculate the light ratio based on pixels from non-overexposed areas of the long-exposure image (e.g., the first raw image). It should be understood that overexposed pixels are those whose recorded brightness exceeds the range that the camera's image sensor can accurately capture. Therefore, overexposed pixels affect the light ratio calculation. Thus, by filtering out overexposed pixels, the accuracy of the calculated actual light ratio can be further improved. Furthermore, the quality of the fused image obtained through subsequent image fusion based on the actual light ratio can be further improved.

[0132] As another possible example, after the camera module removes pixels from the overexposed areas of the long exposure image, it can further filter pixels from the non-overexposed areas. For instance, the camera module removes pixels with outliers in the non-overexposed areas to obtain target pixels. Then, the camera module calculates the light ratio of each target pixel based on the pixel corresponding to that target pixel in the second raw image. Next, the camera module calculates the average light ratio of each target pixel to obtain the actual light ratio.

[0133] The camera module can remove outliers by using standard deviation (e.g., 2x or 3x variance), clustering algorithms (e.g., density-based clustering of applications with noise, DBSCAN), box plot methods, and so on.

[0134] Understandably, outlier pixels represent pixels whose light ratio differs significantly from other pixels. These outlier pixels are considered anomalous. Therefore, in this example, removing these outlier pixels can further improve the accuracy of the calculated actual light ratio. This, in turn, can further improve the quality of the fused image obtained through subsequent image fusion based on the actual light ratio.

[0135] For example, see Figure 8 The above step S701 may include steps S800-S803.

[0136] The S800 camera module constructs an interior point set.

[0137] In this context, the interior point set can be understood as a collection of unexposed pixels from the raw image. For example, the interior point set can include a first interior point set and a second interior point set. The pixels in the first interior point set are from a first raw image, and the pixels in the second interior point set are from a second raw image.

[0138] For example, the camera module can determine non-overexposed pixels based on the pixel values ​​of the first raw image, and form a first inner point set from the non-overexposed pixels of the first raw image. Then, the camera module forms a second inner point set from the pixels of the second raw image corresponding to the non-overexposed pixels of the first raw image.

[0139] The S801 camera module calculates the light ratio of each pixel included in the internal point set.

[0140] For example, the camera module can calculate the ratio of the pixel value of each pixel in the first set of inliers to the pixel value of each pixel in the second set of inliers to obtain the light ratio of each pixel.

[0141] S802. The camera module calculates the standard deviation of the light ratio of pixels within the inner point set and the average light ratio of pixels within the inner point set, and removes pixels whose light ratio exceeds the average ± 3 * standard deviation from the inner point set. That is, pixels whose light ratio is less than the average - 3 * standard deviation are removed from the inner point set, and pixels whose light ratio is greater than the average + 3 * standard deviation are removed from the inner point set.

[0142] It is understood that in some other embodiments, the above-mentioned three times standard deviation can also be adjusted, such as to five times standard deviation, two times standard deviation, 2.5 times standard deviation, etc. The specific design can be made according to the actual use needs.

[0143] Optionally, after step S802, the camera module can immediately execute step S803. Alternatively, the camera module can also repeatedly execute step S802 until the actual light ratio calculation conditions are met.

[0144] The actual light ratio calculation conditions include one or more combinations of the following.

[0145] 1. The number of times step S802 is executed is a preset threshold. The threshold can be 3 times, 5 times, etc.

[0146] 2. Convergence of the interior point set. For example, if the interior point set obtained in the previous execution of step S802 is the same as the interior point set obtained in the current execution of step S802, that is, no pixels were removed in the current execution of step S802.

[0147] The S803 camera module uses the average light ratio of each pixel in the target's point set as the actual light ratio.

[0148] The target interior point set is the interior point set after the camera module has removed outliers in step S802; in other words, it is the interior point set that satisfies the actual light ratio calculation conditions.

[0149] Understandably, through steps S800-S803, since the camera module uses the calculated actual light ratio to perform image fusion, this actual light ratio is more closely aligned with the actual shooting scene. Therefore, the camera module can subsequently use the actual light ratio to effectively align the exposure of the first and second raw images. This results in a better signal-to-noise ratio in the resulting fused image.

[0150] Next, after step S701 or after step S803, the phone executes step S702.

[0151] S702. The camera module performs image fusion on the first raw image and the second raw image based on the actual light ratio to obtain a fused raw image.

[0152] As one possible implementation, the camera module can perform image fusion on the first raw image and the second raw image using a fusion threshold.

[0153] For example, if the pixel value of a pixel in the first raw image is greater than the fusion threshold, the pixel value of the pixel in the fused raw image can be the pixel value of the second raw image multiplied by the actual light ratio. If the pixel value of a pixel in the first raw image is less than or equal to the fusion threshold, the pixel value of the pixel in the fused raw image can be the pixel value of the first raw image.

[0154] In this implementation, since the calculated actual light ratio is more consistent with the shooting scene, using a more consistent light ratio for image fusion can mitigate color anomalies. Even when each pixel is given an independent weight during image fusion, it can still alleviate color anomalies in the resulting fused image.

[0155] The fusion threshold can be found in the following expression 1.

[0156] thres=A*(2 M -1) Expression 1.

[0157] Where thres represents the fusion threshold, and A takes the value (0, 1). M is the bit width of the pixel values ​​in the raw image. For example, the value of A can be 0.95, 0.98, 0.90, 0.85, etc.

[0158] As is understandable, bit width can also be called bit depth, which represents the number of bits for the color or grayscale level of each pixel.

[0159] As can be seen, the fusion threshold is a threshold related to the exposure level of the pixels. That is, if the exposure level of the pixels in the first raw image is not higher than the first level, the pixel value of the pixels in the fused image is the pixel value of the first raw image; if the exposure level of the pixels in the first raw image is higher than the first level, the pixel value of the pixels in the fused image is the pixel value of the second raw image multiplied by the actual light ratio.

[0160] In some possible examples, the pixel values ​​of the merged raw image can be represented by the following expression 2.

[0161]

[0162] Among them, Y m (i, j) represents the pixel value of a pixel in the raw image, Y a (i, j) represents the pixel value of a pixel in the first raw image, Y b (i, j) represents the pixel value of a pixel in the second raw image. Here, i and j are both positive integers, and their values ​​are related to the number of pixels in the camera. Thres represents the fusion threshold.

[0163] Optionally, the camera module can also perform image fusion on the first raw image and the second raw image according to pixel blocks and the actual light ratio. Specifically, as described above... Figure 1 / Figure 2 The corresponding process is similar and will not be elaborated here.

[0164] The S703 phone's display shows a merged raw image.

[0165] As one possible implementation, the mobile phone can convert the format of the merged raw image and then display the converted merged raw image on the screen. Furthermore, the mobile phone can also perform image optimization algorithms on the merged raw image. For a detailed description of this process, please refer to related technologies; this application does not impose any limitations on the embodiments described herein.

[0166] It should be noted that when using the image fusion method provided in the embodiments of this application to fuse three or more images to be fused, the images can be fused in reverse order of the exposure times of the objects to be fused until a single fused image is obtained.

[0167] In this implementation, since the mobile phone fuses the images to be fused by reversing the exposure time, the signal-to-noise ratio of the fused image can be further improved, and the color abnormality of the fused image can be further alleviated.

[0168] For example, in response to a shooting operation, the camera captures a first raw image, a second raw image, and a third raw image. The exposure time for the first raw image is longer than that for the second raw image, and the exposure time for the second raw image is longer than that for the third raw image. That is, the brightness of the first raw image is greater than the brightness of the second raw image, and greater than the brightness of the third raw image. Next, the phone can perform the aforementioned steps S701-S702 on the first and second raw images to obtain a first fused image. Then, the phone performs the aforementioned steps S701-S702 on the first fused image and the third raw image to obtain a final fused image.

[0169] This application provides a chip (system) including a processing circuit and an interface circuit; the processing circuit and the interface circuit are coupled. The processing circuit can execute the relevant method steps described in the above method embodiments through the interconnections between its internal components.

[0170] For example, see Figure 9 The aforementioned interface circuit is configured to execute step 900.

[0171] S900. Receives the first raw image and the second raw image.

[0172] The first raw image and the second raw image are acquired by the image sensor through a single exposure process, and the brightness of the first raw image is higher than that of the second raw image.

[0173] For a further description of step S900, please refer to the preceding text; the embodiments of this application will not be repeated here.

[0174] The processing circuit is configured to execute steps S901-S902.

[0175] S901. Calculate the actual light ratio based on the first raw image and the second raw image.

[0176] Step S901 is similar to step S701 above, and can be found in the relevant introduction above, so it will not be repeated here.

[0177] S902. Based on the actual light ratio, perform image fusion on the first raw image and the second raw image to obtain a fused raw image.

[0178] Step S902 is similar to step S702 above, and can be found in the relevant introduction above, so it will not be repeated here.

[0179] It should be noted that the personal information used in the technical solution of this application is limited to information for which individual consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes authorization of relevant user information before users use the function.

[0180] Based on the algorithmic steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.

[0181] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0182] This application also provides an electronic device, such as... Figure 10 As shown, the electronic device may include one or more processors 1801, memory 1802, and communication interfaces 1803.

[0183] The memory 1802, communication interface 1803, and processor 1801 are coupled together. For example, the memory 1802, communication interface 1803, and processor 1801 can be coupled together via bus 1804.

[0184] The communication interface 1803 is used for data transmission with other devices. The memory 1802 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1801, cause the electronic device to perform the relevant method steps in the above-described method embodiments of this application.

[0185] The processor 1801 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0186] The bus 1804 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The aforementioned bus 1804 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0187] This application also provides a chip system, such as... Figure 11 As shown, the chip system 2000 includes at least one processor 2001 and at least one interface circuit 2002. The processor 2001 and the interface circuit 2002 are interconnected via lines. For example, the interface circuit 2002 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 2002 can be used to send signals to other devices (e.g., the processor 2001). Exemplarily, the interface circuit 2002 can read instructions stored in the memory and send those instructions to the processor 2001. When the instructions are executed by the processor 2001, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0188] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0189] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the above method embodiments.

[0190] The electronic devices, computer-readable storage media, or computer program products provided in this application are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image fusion method, characterized in that, The method is applied to an electronic device, the electronic device including a camera; the method includes: In response to a shooting operation, the camera captures a first raw image and a second raw image; the first raw image and the second raw image are captured by the camera through a single exposure process; The pixels of the first raw image are filtered to obtain target pixels; the ratio of the pixel value of each target pixel to the pixel value of the corresponding pixel in the second raw image is calculated to obtain the light ratio of each target pixel; the mean of the light ratio of the target pixels is calculated to obtain the image light ratio. The first raw image and the second raw image are fused according to the image light ratio to obtain a fused raw image; wherein, when the exposure level of a pixel in the first raw image is greater than a first level, the pixel value of the second raw image is multiplied by the image light ratio to obtain the pixel value of the fused raw image; when the exposure level of a pixel in the first raw image is less than or equal to the first level, the pixel value of the first raw image is used as the pixel value of the fused raw image. Display the fused raw image.

2. The method according to claim 1, characterized in that, The brightness of the first raw image is higher than that of the second raw image.

3. The method according to claim 1, characterized in that, The step of filtering the pixels of the first raw image to obtain the target pixels includes: Calculate the light ratio of each pixel in the first raw image; In the first raw image, the target pixels are obtained by removing pixels with outlier light ratios.

4. The method according to claim 3, characterized in that, The method further includes: In the first raw image, pixels with values ​​greater than the overexposure value are removed to obtain the non-overexposure pixels of the first raw image; The calculation of the light ratio of the pixels in the first raw image includes: Calculate the light ratio of the non-overexposed pixels; The step of removing outliers from the pixels of the first raw image to obtain the target pixels includes: Among the non-overexposed pixels, the target pixels are obtained by removing pixels with outliers in terms of light ratio.

5. The method according to claim 4, characterized in that, The step of removing outliers from the non-overexposed pixels to obtain the target pixels includes: Among the non-overexposed pixels, the target pixel is obtained by removing pixels with outliers based on the standard deviation of the light ratio of the non-overexposed pixels.

6. The method according to claim 5, characterized in that, The process of obtaining the target pixel by removing pixels with outliers in light ratio from the standard deviation of the light ratio of non-overexposed pixels includes: Construct an inner point set, the inner point set including each of the non-overexposed pixels; The standard deviation filtering process is performed iteratively on the interior point set until the number of times the standard deviation filtering process is performed is equal to the preset number, or until the interior point set converges; After the standard deviation filtering process is performed iteratively on the inner point set, the pixels included in the inner point set are taken as the target pixels. The standard deviation screening process includes: Calculate the light ratio of each pixel in the inner point set; Calculate the average light ratio of the pixels included in the inner point set; Calculate the standard deviation of the light ratio of the pixels included in the inner point set; Pixels with a concentrated light ratio greater than the mean plus N times the standard deviation are removed, and pixels with a concentrated light ratio less than the mean minus N times the standard deviation are also removed; where N is a positive integer.

7. The method according to claim 1, characterized in that, The pixels in the first raw image are exposed to a higher degree than the first level, including: The pixel value of the first raw image is greater than the fusion threshold; The exposure level of the pixels in the first raw image is less than or equal to the first level, including: The pixel value of the first raw image is less than or equal to the fusion threshold; Wherein, the fusion threshold is A (2) M -1); the value of A is (0, 1], and M is the bit width of the first raw image.

8. A chip, characterized in that, The chip includes a processing circuit and an interface circuit, and the interface circuit is coupled to the processing circuit. The interface circuit is configured to receive a third raw image and a fourth raw image; the third raw image and the fourth raw image are acquired by the camera through a single exposure process; The processing circuit is configured to: filter the pixels of the third raw image to obtain target pixels; The ratio of the pixel value of each target pixel to the pixel value of the corresponding pixel in the fourth raw image is calculated to obtain the light ratio of each target pixel; Calculate the average light ratio of the target pixels to obtain the image light ratio; The third raw image and the fourth raw image are fused according to the image light ratio to obtain a fused raw image; wherein, if the exposure level of the pixels in the third raw image is greater than the first level, the pixel value of the pixels in the fourth raw image is multiplied by the image light ratio to obtain the pixel value of the pixels in the fused raw image. If the exposure level of a pixel in the third raw image is less than or equal to the first level, the pixel value of the pixel in the third raw image is used as the pixel value of the pixel in the fused raw image.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a camera; the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-7.

11. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.

Images