Image fusion method, chip, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of image processing, in particular to an image fusion method, a chip, electronic equipment and a storage medium. The method comprises the following steps: in response to a shooting operation, acquiring a first raw image and a second raw image by a camera of the electronic equipment; the first raw image and the second raw image are acquired by the camera through one exposure process. That is to say, the camera can obtain the first raw image and the second raw image through a single-exposure multiple-reading mode. Next, the electronic device calculates an image light ratio through the pixel values of the pixel points of the first raw image and the pixel values of the pixel points of the second raw image. And the electronic device performs image fusion on the first raw image and the second raw image according to the image light ratio to obtain a fused raw image. Then, the electronic device displays the fused raw image. Through the method, the signal-to-noise ratio of the obtained fused image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image fusion method, chip, electronic device, and storage medium. Background Art

[0002] Currently, electronic devices can fuse multiple images to obtain a fused image.

[0003] At present, how to improve the quality of fused images obtained by electronic devices through image fusion is an issue that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image fusion method, a chip, an electronic device, and a storage medium, by which the quality of a fused image obtained by image fusion can be improved.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image fusion method is provided. The method is applied to an electronic device including a camera. The method includes: in response to a capture operation, the camera captures a first raw image and a second raw image; the first raw image and the second raw image are captured by the camera through a single exposure process. In other words, the camera can obtain the first raw image and the second raw image using a single exposure multiple read mode. Next, the electronic device calculates an image light ratio based on pixel values ​​of pixels in the first raw image and pixel values ​​of pixels in the second raw image. The electronic device then fuses the first raw image and the second raw image based on the image light ratio to obtain a fused raw image. The electronic device then displays the fused raw image.

[0007] In the above method, the image light ratio is calculated by the electronic device based on the pixel values ​​of the images to be fused. This allows the image light ratio to better reflect the actual shooting scene. By performing image fusion with a light ratio that better reflects the actual shooting scene, the exposure alignment of the images to be fused can be improved during the image fusion process. This method can also improve the quality of the fused image, for example, by increasing the signal-to-noise ratio of the fused image.

[0008] In one possible design of the first aspect, the brightness of the first raw image is higher than that of the second raw image. The electronic device calculates the image light ratio using pixel values ​​of pixels in the first raw image and pixel values ​​of pixels in the second raw image, including: the electronic device filters the pixels of the first raw image to obtain target pixels. The electronic device then calculates the ratio of each target pixel to its corresponding pixel in the second raw image to obtain the light ratio of each target pixel. The electronic device then uses the average of the light ratios of each target pixel as the image light ratio.

[0009] In this design, the electronic device filters the first raw image and removes pixels that are unsuitable for calculating the image light ratio. This further improves the consistency of the image light ratio with the actual shooting scene and the quality of the fused image obtained by image fusion.

[0010] In another possible design of the first aspect, the electronic device screens pixels of the first raw image to obtain target pixels, including: calculating light ratios of the pixels of the first raw image. Next, the electronic device removes pixels with outlier light ratios from the pixels of the first raw image to obtain the target pixels.

[0011] In this design, the electronic device removes pixels whose light ratios are outliers, which can further improve the consistency between the image light ratio and the actual shooting scene; and can further improve the quality of the fused image obtained by image fusion.

[0012] In another possible design of the first aspect, the method further includes: the electronic device removing, from the pixels of the first raw image, pixels whose pixel values ​​are greater than an overexposure value, to obtain non-overexposed pixels of the first raw image. The electronic device calculating the light ratios of the pixels of the first raw image includes: the electronic device calculating the light ratios of the non-overexposed pixels. The electronic device removing, from the pixels of the first raw image, pixels whose light ratios are outliers to obtain target pixels includes: the electronic device removing, from the pixels of the non-overexposed pixels, pixels whose light ratios are outliers to obtain target pixels.

[0013] In this design, the electronic device removes overexposed pixels, which can further improve the consistency of the image light ratio with the actual shooting scene; and can further improve the quality of the fused image obtained by image fusion.

[0014] In another possible design of the first aspect, the above-mentioned electronic device removes pixels whose light ratios are outliers among non-overexposed pixels to obtain target pixels, including: the electronic device removes pixels whose light ratios are outliers among non-overexposed pixels to obtain target pixels through the standard deviation of the light ratios of the non-overexposed pixels.

[0015] In this design, the electronic device can effectively remove pixels whose light ratios are outliers through standard deviation.

[0016] In another possible design of the first aspect, the electronic device removes pixels with outlier light ratios from the light ratios of non-overexposed pixels to obtain target pixels. This process includes: the electronic device constructing an inlier set, which includes each non-overexposed pixel. Next, the electronic device iteratively performs a standard deviation screening process on the inlier set until the standard deviation screening process is repeated a preset number of times, or until the inlier set converges. After the standard deviation screening process is completed, the electronic device uses the pixels included in the inlier set as target pixels. The standard deviation screening process includes: the electronic device calculating the light ratio of each pixel included in the inlier set. Next, the electronic device calculates the mean of the light ratios of the pixels included in the inlier set. Next, the electronic device calculates the standard deviation of the light ratios of the pixels included in the inlier set. Next, the electronic device removes pixels from the inlier set whose light ratios are greater than the mean + N times the standard deviation, and removes pixels from the inlier set whose light ratios are less than the mean - N times the standard deviation, where N is a positive integer.

[0017] In this design, the electronic device can further improve the consistency between the image light ratio and the actual shooting scene by cyclically executing the standard deviation screening process; and can further improve the quality of the fused image obtained by image fusion.

[0018] In another possible design of the first aspect, the electronic device fuses the first raw image and the second raw image based on an image light ratio to obtain a fused raw image. The method includes: when the exposure level of a pixel point in the first raw image is greater than a first level, the electronic device multiplies a pixel value of a pixel point in the second raw image by the image light ratio to obtain the pixel value of the fused raw image. Subsequently, when the exposure level of a pixel point in the first raw image is less than or equal to the first level, the electronic device uses the pixel value of the pixel point in the first raw image as the pixel value of the fused raw image.

[0019] In this design, when the exposure levels of the first raw images are different, the electronic device obtains a fused raw image according to different images to be fused, which can further improve the quality of the fused image.

[0020] In another possible design of the first aspect, the exposure degree of the pixel points of the first raw image is greater than the first degree, including: the pixel value of the pixel points of the first raw image is greater than the fusion threshold. And the exposure degree of the pixel points of the first raw image is less than or equal to the first degree, including: the pixel value of the pixel points of the first raw image is less than or equal to the fusion threshold. The fusion threshold is A*(2 M -1); the value of A is (0, 1], and M is the bit width of the first raw image.

[0021] In a second aspect, an electronic device is provided, which includes a memory and one or more processors, and the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method provided by the above-mentioned first aspect and any possible design of the first aspect.

[0022] In a third aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method provided by the first aspect and any possible design of the first aspect.

[0023] In a fourth aspect, a computer program product comprising instructions is provided. When the computer program product is run on an electronic device, the electronic device can execute the method provided by the first aspect and any possible design of the first aspect.

[0024] In a fifth aspect, a chip system is provided, which is applied to an electronic device. The chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method provided by the above-mentioned first aspect and any possible design of the first aspect.

[0025] According to a sixth aspect, a chip is provided, comprising a processing circuit and an interface circuit coupled to the processing circuit. The interface circuit is configured to receive a third raw image and a fourth raw image, wherein the third raw image and the fourth raw image are captured by a camera through a single exposure process. The processing circuit is configured to calculate an image light ratio based on pixel values ​​of pixels in the third raw image and pixel values ​​of pixels in the fourth raw image, and fuse the third raw image and the fourth raw image based on the image light ratio to obtain a fused raw image.

[0026] Among them, the technical effects brought about by any design method in the second to sixth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1A flowchart of an image fusion method provided by an embodiment of the present application;

[0028] Figure 2 A flowchart of another image fusion method provided by an embodiment of the present application;

[0029] Figure 3 A diagram about the effect of a fused image provided by an embodiment of the present application;

[0030] Figure 4 A diagram about the signal-to-noise ratio of a fused image provided by an embodiment of the present application;

[0031] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present application;

[0032] Figure 6 An architectural diagram of an electronic device provided by an embodiment of the present application;

[0033] Figure 7 A flowchart of an image fusion method provided by an embodiment of the present application;

[0034] Figure 8 A diagram about the actual light ratio calculation process provided by an embodiment of the present application;

[0035] Figure 9 A diagram of a chip (system) provided by an embodiment of the present application;

[0036] Figure 10 A diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0037] Figure 11 A structural diagram of a chip system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0039] At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0040] In the technical solutions disclosed in this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0041] With the development of technology, users have higher and higher requirements for image quality. Currently, electronic devices can fuse multiple images to obtain a fused image.

[0042] For example, an electronic device's image sensor can read out images multiple times during a single exposure of the image sensor in response to a capture instruction. The electronic device then fuses the multiple readout images to produce a fused image. Such an image sensor can also be referred to as a sensor operating in single exposure multiple readout (SEMR) mode.

[0043] The following uses the example of an electronic device fusing two images to introduce some solutions provided in the embodiments of this application.

[0044] In some solutions provided in the embodiments of the present application, the electronic device may fuse images using an image fusion method based on a theoretical light ratio.

[0045] For example, see Figure 1 , the pixel block of image 1 includes two green pixels, one blue pixel and one red pixel; each pixel corresponds to a preset weight. Similarly, the pixel block of image 2 includes two green pixels, one blue pixel and one red pixel; each pixel corresponds to a preset weight. Afterwards, the electronic device multiplies the pixel value of each pixel by its respective weight. Afterwards, the electronic device multiplies the pixel value of the weighted pixel of image 2 by the theoretical light ratio. Next, the pixel values ​​of the pixel points of image 1 and the pixel values ​​of the pixel points of image 2 after the above processing are added together to obtain the pixel values ​​of the pixel block of the fused image.

[0046] For example, the pixel blocks of image 1 include B1, R1, G1a, and G1b; the pixel blocks of image 2 include B2, R2, G2a, and G2b. The pixel blocks of the fused image include B3, R3, G3a, and G3b.

[0047] Image 1 and Image 2 can be two images read out from a SEMR sensor during the same exposure. The exposure time corresponding to Image 1 and Image 2 is different. The theoretical light ratio can be calculated using a light ratio calculation method based on the exposure time corresponding to Image 1 and Image 2. For example, the theoretical light ratio can be the exposure time corresponding to Image 1 / the exposure time corresponding to Image 2. Alternatively, the theoretical light ratio can be preset.

[0048] from Figure 1 visible:

[0049] Pixel value of G3a = pixel value of G1a * W1 + pixel value of G2a * (1-W1) * theoretical light ratio;

[0050] Pixel value of B3 = pixel value of B1 * W2 + pixel value of B2 * (1-W2) * theoretical light ratio;

[0051] Pixel value of R3 = pixel value of R1 * W3 + pixel value of R2 * (1-W3) * theoretical light ratio;

[0052] Pixel value of G3b = pixel value of G1b*W4+pixel value of G2b*(1-W4)*theoretical light ratio.

[0053] Among them, at least two of W1, W2, W3 and W4 are different.

[0054] In other solutions provided in the embodiments of the present application, the electronic device may fuse images using an image fusion method that ensures color ratio.

[0055] For example, see Figure 2 , the pixel block of image 1 includes two green pixels, one blue pixel and one red pixel; each pixel corresponds to a preset weight. Similarly, the pixel block of image 2 includes two green pixels, one blue pixel and one red pixel; each pixel corresponds to a preset weight. Afterwards, the electronic device multiplies the pixel value of each pixel by the preset weight. Afterwards, the electronic device multiplies the pixel value of the weighted pixel of image 2 by the theoretical light ratio. Next, the pixel values ​​of the pixel points of image 1 after the above processing and the pixel values ​​of the pixel points of image 2 are added to obtain the pixel values ​​of the pixel block of the fused image.

[0056] For example, the pixel blocks of image 1 include B4, R4, G4a, and G4b, the pixel blocks of image 2 include B5, R5, G5a, and G5b, and the pixel blocks of the fused image include B6, R6, G6a, and G6b.

[0057] from Figure 2 visible:

[0058] Pixel value of G6a = pixel value of G4a * W + pixel value of G5a * (1-W) * theoretical light ratio;

[0059] Pixel value of B6 = pixel value of B4 * W + pixel value of B5 * (1-W) * theoretical light ratio;

[0060] Pixel value of R6 = pixel value of R4 * W + pixel value of R5 * (1-W) * theoretical light ratio;

[0061] Pixel value of G6b = pixel value of G4b*W+pixel value of G5b*(1-W)*theoretical light ratio.

[0062] It should be pointed out that the above Figure 1 and Figure 2 The corresponding introduction uses binning format images as an example. The image fusion method provided in the embodiments of the present application does not impose any restrictions on the image format. In other words, the image fusion method provided in the embodiments of the present application is not only applicable to binning format images, but also to hex format and quad format images, as well as some image formats that will be evolved in the future.

[0063] It should be understood that the above Figure 1 and Figure 2 The corresponding solution is introduced by taking the fusion of two images as an example; in actual use, the above Figure 1 / Figure 2 The corresponding scheme can also be used to fuse more than two images, as long as the above scheme is performed on each image. For example, to fuse image 1, image 2 and image 3, image 1 and image 2 can be fused first to obtain fused image 1 according to the corresponding scheme. Next, fused image 1 and image 3 are fused to obtain fused image 2. Similarly, the corresponding scheme is used to fuse two images, and is not described in detail. Figure 1 Figure 2 The corresponding scheme is used to fuse two images, and is not described in detail.

[0064] In some use scenarios, the fused image obtained by the above image fusion method using the theoretical light ratio may have color deviation due to the deviation of the theoretical light ratio used from the shooting scene.

[0065] For example, referring to Figure 3 , fused image 1 is a fused image obtained by the image fusion method using the theoretical light ratio, and fused image 2 is a fused image obtained by the image fusion method ensuring the color ratio. Fused image 1 is filled with diagonal lines, indicating that the object is overall green.

[0066] It should be understood that for Figure 2 The corresponding scheme, the same weight is used for the pixel points included in the pixel block. In this way, the color deviation due to the deviation of the theoretical light ratio from the shooting scene can be alleviated. Thus, the phenomenon of color deviation can be alleviated to a certain extent by using the image fusion method ensuring the color ratio.

[0067] In some use scenarios, the fused image obtained by the above image fusion method ensuring the color ratio may result in a relatively high signal-to-noise ratio of the fused image.

[0068] For example, referring to Figure 4 , Figure 4 The signal-to-noise ratio of the blue pixel point (B) channel of a certain fused image is shown, which is fused from a short exposure image and a long exposure image by the above image fusion method ensuring the color ratio. Among them, the short exposure image and the long exposure image are obtained by SEMR from the same image sensor. As can be seen from the figure, it can be seen that the B channel is fused into the short exposure image in the area with low pixel signal (such as low pixel value), and the short exposure image is not well aligned with the long exposure image in terms of exposure rate, which will result in poor signal-to-noise ratio of the fused image.

[0069] ​In view of this, an embodiment of the present application provides an image fusion method, in which an electronic device obtains at least two images to be fused. The at least two images to be fused are obtained by the image sensor through the SEMR mode. Afterwards, the electronic device obtains a light ratio based on the pixel values ​​of the pixel points of the at least two images to be fused. Afterwards, the electronic device performs image fusion on the two images to be fused based on the light ratio to obtain a fused image. In this method, the light ratio is calculated based on the images to be fused, so the light ratio will be more in line with the actual shooting scene. By performing image fusion with a light ratio that is more in line with the actual shooting scene, the effect of exposure alignment of the images to be fused can be improved during the image fusion process. The quality of the fused image obtained by image fusion can be improved by this method. For example, the signal-to-noise ratio of the obtained fused image can be improved; for another example, the color abnormality of the fused image can be alleviated.

[0070] Among them, the above-mentioned electronic device can also be called a terminal, terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The above-mentioned electronic device 100 can be a mobile phone, a tablet computer, a wearable device, a smart screen, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and other electronic devices with image processing functions. Optionally, the above-mentioned electronic device may also have a shooting function. The embodiments of the present application do not impose any restrictions on the product form of the electronic device.

[0071] Next, the hardware structure and structure of the electronic device provided in the embodiment of the present application are introduced.

[0072] Figure 5 The figure shows a hardware structure diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a camera 193, and a display screen 194. The camera 193 is optional. The display screen 194 is also optional.

[0073] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0075] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0076] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0077] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0078] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0079] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0080] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0081] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0082] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0083] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0084] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0087] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0088] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0089] It should be pointed out that the image fusion method provided in the embodiment of the present application can be independently executed by a processor, which can be an AP, an ISP, or other processors that will emerge from future technological evolution. When the ISP executes the image fusion method provided in the embodiment of the present application, the ISP can implement the relevant steps in the image fusion method through its internal circuits and the connection relationship between the internal circuits. For example, the ISP may include an interface circuit and a processing circuit, and the interface circuit is configured to receive at least two images to be fused. And, the processing circuit is configured to: convert the pixel values ​​of the pixel points included in the at least two images to be fused to the light ratio. Next, the two images to be fused are fused according to the light ratio to obtain a fused image. For further description of this method, please see below and will not be elaborated here.

[0090] Furthermore, the image fusion method provided in the embodiments of the present application can also be applied to electronic devices having a processor and a camera. For example, the camera of the electronic device captures at least two images to be fused using SEMR. Next, the processor of the electronic device determines a light ratio based on the pixel values ​​of the pixels in the at least two images to be fused. The processor of the electronic device then fuses the two images to be fused based on the light ratio to produce a fused image.

[0091] After introducing the hardware structure of the electronic device provided in the embodiment of the present application, the architecture of the electronic device is introduced.

[0092] The architecture of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. TM Taking the system as an example, the architecture of the electronic device 100 is exemplarily described.

[0093] For example, see Figure 6 The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL) and kernel layer.

[0094] The application layer can include a series of application packages.

[0095] like Figure 6 As shown, the application package may include application packages of camera applications, gallery applications, and other applications.

[0096] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0097] like Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, and the like.

[0098] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0099] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0100] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0101] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0102] The application framework layer may further include a camera interface, which may provide communication capabilities between software modules in the application layer and software modules in the underlying layer.

[0103] The Hardware Abstraction Layer (HAL) is an abstracted layer between the hardware and upper layers. It provides a unified interface for the upper layer, shielding upper-layer applications from the details of the underlying hardware implementation.

[0104] The Hardware Abstraction Layer (HAL) provides a standard interface that exposes device hardware capabilities to the higher-level application framework layer. The HAL contains multiple library modules, each of which implements an interface for a specific type of hardware component. These library modules might include a camera module, for example. When the application framework layer requests access to device hardware, the system loads the corresponding library module for that hardware component. Manufacturers can define interfaces in the HAL.

[0105] Furthermore, the camera module may include a high dynamic range (HDR) shooting module, which provides the ability to generate HDR images. In subsequent examples, the functions of the above modules will be described in detail.

[0106] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, etc.

[0107] In some implementations, the application layer, application framework layer, HAL, and kernel layer can all be deployed on a processor in an electronic device. To facilitate the architecture of the image fusion method provided by the embodiments of this application, a hardware layer can be included below the kernel layer. The hardware included in the hardware layer can be independent hardware like the processor, such as a camera.

[0108] For example, see Figure 6 , combined with the shooting scene of the electronic device, the image fusion method provided in the embodiment of the present application is introduced.

[0109] After the camera application is started, in response to the user's shooting operation, the camera application sends a shooting instruction to the camera module through the camera interface. Afterwards, the camera module calls the camera to capture images through the camera driver according to the shooting instruction. The camera captures at least two original (raw) images through a single exposure. Next, the camera driver sends the at least two raw images to the camera module. The camera module obtains a light ratio based on the pixel values ​​of the pixels included in the at least two original images. Afterwards, the camera module performs image fusion on the above two images to be fused based on the light ratio to obtain a fused raw image. Next, the camera module sends the fused raw image to the camera application through the camera interface. The camera application displays the fused raw image in the shooting preview interface. Optionally, the gallery application can also store the fused raw image.

[0110] Below, the electronic device is a mobile phone including a camera, and the mobile phone has the above Figure 5 The hardware structure shown above Figure 6 Taking the software architecture shown as an example, the image fusion method provided in the embodiment of the present application is further introduced.

[0111] For example, see Figure 7 The image fusion method provided in the embodiment of the present application may include steps S700-S703.

[0112] S700. In response to a shooting operation, the camera acquires a first raw image and a second raw image.

[0113] The shooting operation can be a user triggering a shooting button on the shooting preview interface. Alternatively, the shooting operation can be a user's voice command such as "take a photo." Alternatively, the shooting operation can be other user operations. The embodiments of the present application do not impose any restrictions on the shooting operation.

[0114] As a possible implementation, the camera can obtain the first raw image and the second raw image through the SEMR mode. For example, the camera can implement the SEMR mode of the camera through the lateral overflow integration capacitor (LOFIC) technology. For example, the camera can implement the SEMR mode of the camera through the dual conversion gain (DCG) technology. For another example, the camera can implement the SEMR mode of the camera through the dual analog gain (DAG) technology.

[0115] The following is an example of an implementation of the SEMR mode. The SEMR mode can be divided into an exposure phase, a first readout phase, and a second readout phase.

[0116] About the exposure phase: During the exposure phase, the entire sensor array of the camera is exposed to light and accumulates light signals. During this process, photons are converted into electrons by the photosensitive diodes corresponding to the pixels.

[0117] Regarding the first readout phase: After exposure, the camera performs its first readout of the electrons accumulated in the photodiode corresponding to the pixel. At this stage, the photodiode has a relatively high number of electrons, resulting in a stronger readout signal. The raw image obtained during this first readout phase is called a long-exposure raw image. After the electrons are read out, the photodiode corresponding to the pixel is not completely cleared of electrons; instead, some electrons remain. This partial removal can be achieved through a rapid charge reversal.

[0118] Regarding the second readout phase: After the first readout, the second readout phase occurs. Because the electrons in the pixel have been partially emptied, the signal read out this time is weaker. The raw image obtained by the camera during the second readout phase can be called a short-exposure raw image.

[0119] In other implementations, the camera can read out the electrons accumulated in the photodiode corresponding to the pixel multiple times after the first readout phase. Each readout is followed by a partial clearing operation. Each readout signal is weaker than the previous one.

[0120] In some examples, the camera can capture an image in response to a capture instruction sent by the mobile phone. Thus, step S700 includes: in response to the capture instruction, the camera captures a first raw image and a second raw image through a single exposure process; the brightness of the first raw image is higher than that of the second raw image.

[0121] S701. The camera module calculates the actual light ratio through the first raw image and the second raw image.

[0122] The actual light ratio may also be referred to as the shooting light ratio, the real light ratio, the image light ratio, etc. It can be understood as the light ratio obtained based on factors of the actual shooting scene, such as the light ratio calculated based on the pixel values ​​of the pixels in the captured image.

[0123] It should be understood that the first raw image and the second raw image may be acquired by a camera in SEMR mode, and the first raw image and the second raw image are pixel-aligned. In other words, the pixels of the first raw image and the pixels of the second raw image are in one-to-one correspondence.

[0124] In some other implementations, before calculating the actual light ratio, for example, if the first raw image and the second raw image are not pixel-aligned, the camera module may also perform a pixel alignment operation on the first raw image and the second raw image, and then calculate the actual light ratio.

[0125] As one possible implementation, when the camera module calculates the actual light ratio using the first and second raw images, the camera module can filter out some poor-quality pixels. It should be understood that the raw images captured by the camera may contain some bad pixels and noise. Therefore, filtering out this noise during the actual light ratio calculation can improve the accuracy of the calculated actual light ratio.

[0126] As one possible example, the first and second raw images are captured by a camera through a single exposure process. The first raw image can be the raw image read out by the camera during the first readout phase, and the second raw image can be the raw image read out by the camera during the second readout phase. As can be seen from the above description, during the first readout phase, the number of electrons in the photodiode is relatively large. Therefore, the exposure time corresponding to the first raw image is longer than that of the second raw image. In other words, the brightness of the first raw image is higher than that of the second raw image.

[0127] The exposure time corresponding to the first raw image is longer than the exposure time corresponding to the second raw image. The camera module may remove overexposed pixels from the first raw image. In other words, the camera module calculates the actual light ratio based on the non-overexposed pixels in the first raw image and the non-overexposed pixels in the second raw image.

[0128] It should be understood that since the brightness of the first raw image is higher than that of the second raw image, any overexposed pixels in the second raw image will also be overexposed in their corresponding pixels in the first raw image. Therefore, by removing the overexposed pixels in the first raw image, the overexposed pixels in the second raw image can also be removed, improving image fusion efficiency.

[0129] For example, the bit width of the first and second raw images is 10 bits, that is, the pixel values ​​of the first and second raw images range from [0 to 1023]. The camera module may treat pixels with values ​​greater than or equal to 1022 as overexposed pixels and pixels with values ​​less than 1022 as non-overexposed pixels. Alternatively, the camera module may treat pixels with values ​​greater than or equal to 1000 as overexposed pixels and pixels with values ​​less than 1000 as non-overexposed pixels.

[0130] The camera module can calculate a ratio of a pixel value of each non-overexposed pixel point of the first raw image to a pixel value of a pixel point corresponding to each non-overexposed pixel point of the second raw image to obtain a light ratio of each non-overexposed pixel point. Then, the camera module can calculate an average value of the light ratio of each non-overexposed pixel point to obtain the actual light ratio.

[0131] In this way, the camera module can remove the pixel points of the overexposed area of the long-exposure image (e.g., the first raw image), and calculate the light ratio according to the pixel points of the non-overexposed area of the long-exposure image (e.g., the first raw image). It should be understood that the pixel points of the overexposed area can be understood as the pixel points whose recorded brightness exceeds the range that the light sensor of the camera can accurately capture. Therefore, the pixel points of the overexposed area will affect the calculation of the light ratio. Therefore, by filtering out the pixel points of the overexposed area, the accuracy of the actual light ratio calculated can be further improved. In addition, the quality of the fused image obtained by subsequently fusing the images according to the actual light ratio can be further improved.

[0132] As another possible example, after the camera module removes the pixel points of the overexposed area of the long-exposure image, the camera module can further filter the pixel points of the non-overexposed area of the long-exposure image. For example, the camera module can remove the pixel points of the non-overexposed area whose light ratio is an outlier to obtain target pixel points. Then, the camera module can calculate the light ratio of each target pixel point according to each target pixel point and the pixel point corresponding to each target pixel point in the second raw image. Next, the camera module can calculate an average value of the light ratio of each target pixel point to obtain the actual light ratio.

[0133] The camera module can remove the outliers by using standard deviation (e.g., 2 times or 3 times of the standard deviation), by using some clustering algorithms (e.g., Density-Based Spatial Clustering of Applications with Noise, DBSCAN), by using a box plot method, or the like.

[0134] It can be understood that the pixel points of the light ratio outliers can represent that the light ratio of the pixel points is significantly different from the light ratio of other pixel points, that is, the light ratio of the pixel points is significantly different from the light ratio of other pixel points. Therefore, in this example, by removing the pixel points of the light ratio outliers, the accuracy of the actual light ratio calculated can be further improved. In addition, the quality of the fused image obtained by subsequently fusing the images according to the actual light ratio can be further improved.

[0135] For example, referring to FIG. 7, the step S701 can include steps S800-S803. Figure 8 The step S701 can include steps S800-S803.

[0136] S800. Camera module constructs internal point set.

[0137] The inlier point set can be understood as a set of non-exposed pixels in the raw image. For example, the inlier point set may include a first inlier point set and a second inlier point set. The first inlier point set includes pixels from the first raw image, and the second inlier point set includes pixels from the second raw image.

[0138] For example, the camera module may determine non-overexposed pixels based on pixel values ​​of the first raw image, and form a first inlier point set with the non-overexposed pixels of the first raw image. Subsequently, the camera module forms a second inlier point set with pixels of the second raw image corresponding to the non-overexposed pixels of the first raw image.

[0139] S801. The camera module calculates the light ratio of each pixel point included in the interior point set.

[0140] Exemplarily, the camera module may calculate the ratio of the pixel value of each pixel point included in the first interior point set to the pixel value of each pixel point included in the second first interior point set to obtain the light ratio of each pixel point.

[0141] S802. The camera module calculates the standard deviation of the light ratios of the pixels in the inlier set and the average light ratio of the pixels in the inlier set, and removes pixels whose light ratios exceed the average ± 3 * standard deviation from the inlier set. In other words, pixels whose light ratios are less than the average - 3 * standard deviation are removed from the inlier set, and pixels whose light ratios are greater than the average + 3 * standard deviation are removed from the inlier set.

[0142] It is understandable that in some other embodiments, the above-mentioned three times the standard deviation can also be adjusted, such as adjusted to 5 times the standard deviation, 2 times the standard deviation, 2.5 times the standard deviation, etc., and the specific design can be based on actual usage needs.

[0143] Optionally, after step S802, the camera module may immediately execute step S803. Alternatively, the camera module may cyclically execute step S802 until the actual light ratio calculation condition is met.

[0144] The actual light ratio calculation conditions include one or more combinations of the following.

[0145] 1. The number of executions of step S802 is a preset number threshold, which may be 3 times, 5 times, etc.

[0146] 2. The inlier point set converges. For example, the inlier point set obtained in the last execution of step S802 is the same as the inlier point set obtained in this execution of step S802, that is, no pixel is removed in this execution of step S802.

[0147] S803. The camera module takes the average light ratio of each pixel point included in the target internal point set as the actual light ratio.

[0148] The target inlier set is the inlier set from which outliers are removed after the camera module executes step S802 , that is, the inlier set that meets the actual light ratio calculation condition.

[0149] As can be appreciated, in steps S800-S803, because the camera module uses the calculated actual light ratio to perform image fusion, the actual light ratio is more consistent with the actual shooting scene. Consequently, the camera module can subsequently use the actual light ratio to effectively align the exposures of the first and second raw images. This improves the signal-to-noise ratio of the subsequently obtained fused image.

[0150] Next, after step S701 or after step S803, the mobile phone executes step S702.

[0151] S702. The camera module performs image fusion on the first raw image and the second raw image according to the actual light ratio to obtain a fused raw image.

[0152] As a possible implementation, the camera module may perform image fusion on the first raw image and the second raw image using a fusion threshold.

[0153] For example, if the pixel value of a pixel in the first raw image is greater than the fusion threshold, the pixel value of the fused raw image may be the pixel value of the pixel in the second raw image multiplied by the actual light ratio. If the pixel value of a pixel in the first raw image is less than or equal to the fusion threshold, the pixel value of the fused raw image may be the pixel value of the pixel in the first raw image.

[0154] In this embodiment, since the calculated actual light ratio is more consistent with the shooting scene, using a light ratio that is more consistent with the shooting scene for image fusion can alleviate color anomalies. Even if each pixel is weighted independently during the image fusion process, color anomalies in the subsequent fused image can be alleviated.

[0155] The fusion threshold (threshold) can be referred to in the following expression 1.

[0156] thres=A*(2 M -1)Expression 1.

[0157] Wherein, thres represents the fusion threshold, and the value of A is (0, 1). M is the bit width of the pixel value of the raw image. For example, the value of A can be 0.95, 0.98, 0.90, 0.85, etc.

[0158] It can be understood that the bit width can also be called bit depth, which represents the number of bits of the color or grayscale level of each pixel.

[0159] It can be seen that the fusion threshold is a threshold related to the exposure level of the pixel points. That is, when the exposure level of the pixel points in the first raw image is not higher than the first level, the pixel value of the pixel point in the fused image is the pixel value of the first raw image; when the exposure level of the pixel points in the first raw image is higher than the first level, the pixel value of the pixel point in the fused image is the pixel value of the second raw image multiplied by the actual light ratio.

[0160] In some possible examples, the pixel value of a pixel point of the fused raw image can be expressed by the following Expression 2.

[0161]

[0162] Among them, Y m (i, j) represents the pixel value of the raw image, Y a (i, j) represents the pixel value of the first raw image, Y b (i, j) represents the pixel value of the second raw image. Wherein, i and j are both positive integers, and their value range is related to the number of pixels of the camera. Thres represents the fusion threshold.

[0163] Optionally, the camera module can also perform image fusion on the first raw image and the second raw image according to the actual light ratio in accordance with the pixel blocks. Figure 1 / Figure 2 The corresponding process is similar and will not be described here.

[0164] S703. The display screen of the mobile phone displays the fused raw image.

[0165] As a possible implementation, the mobile phone may convert the format of the fused raw image and then display the converted fused raw image on a display screen. Furthermore, the mobile phone may also perform some image optimization algorithms on the fused raw image. For a detailed description of this process, please refer to the relevant art, and this embodiment of the present application does not impose any limitation thereto.

[0166] It should be noted that when using the image fusion method provided in the embodiment of the present application to fuse three or more images to be fused, the images to be fused can be fused in reverse order of the exposure times corresponding to the objects to be fused until a fused image is obtained.

[0167] In this embodiment, since the mobile phone fuses the fused images in reverse order of exposure time, the signal-to-noise ratio of the fused image can be further improved, and the color abnormality of the fused image can be further alleviated.

[0168] For example, in response to a capture operation, the camera captures a first raw image, a second raw image, and a third raw image. The exposure time corresponding to the first raw image is longer than the exposure time corresponding to the second raw image, and the exposure time corresponding to the second raw image is longer than the exposure time corresponding to the third raw image. In other words, the brightness of the first raw image is greater than the brightness of the second raw image, and greater than the brightness of the third raw image. Next, the mobile phone can perform steps S701-S702 on the first and second raw images to obtain a first fused image. Thereafter, the mobile phone performs steps S701-S702 on the first and third raw images to obtain a final fused image.

[0169] An embodiment of the present application provides a chip (system) comprising a processing circuit and an interface circuit; the processing circuit and the interface circuit are coupled, wherein the processing circuit can execute the relevant method steps of the above method embodiment through the connection relationship between the components within the processing circuit.

[0170] For example, see Figure 9 , the above-mentioned interface circuit is configured to perform step 900.

[0171] S900. Receive a first raw image and a second raw image.

[0172] The first raw image and the second raw image are collected by the image sensor through a single exposure process, and the brightness of the first raw image is higher than the brightness of the second raw image.

[0173] For further introduction to step S900, please refer to the above text, which will not be repeated here in the embodiment of the present application.

[0174] The processing circuit is configured to perform steps S901 - S902 .

[0175] S901. Calculate the actual light ratio according to the first raw image and the second raw image.

[0176] Among them, step S901 is similar to the above-mentioned step S701, and reference can be made to the relevant introduction above, which will not be repeated here.

[0177] S902: Perform image fusion on the first raw image and the second raw image according to the actual light ratio to obtain a fused raw image.

[0178] Among them, step S902 is similar to the above-mentioned step S702. Please refer to the relevant introduction in the previous text and will not be repeated here.

[0179] It should be noted that the personal information used in the technical solution of this application is limited to information for which the individual’s separate consent has been obtained, including but not limited to notifying and reminding the user to read the relevant user agreement (notification) and sign the agreement (authorization) including authorization of relevant user information before the user uses the function.

[0180] In conjunction with the algorithmic steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a manner driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0181] In this embodiment, the electronic device can be divided into functional modules according to the above-mentioned method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0182] The present application also provides an electronic device, such as Figure 10 As shown, the electronic device may include one or more processors 1801 , a memory 1802 and a communication interface 1803 .

[0183] The memory 1802 and the communication interface 1803 are coupled to the processor 1801. For example, the memory 1802, the communication interface 1803 and the processor 1801 may be coupled together via a bus 1804.

[0184] The communication interface 1803 is used to transmit data with other devices. The memory 1802 stores computer program code. The computer program code includes computer instructions. When the computer instructions are executed by the processor 1801, the electronic device performs the relevant method steps in the above-mentioned method embodiment of the present application.

[0185] The processor 1801 may be a processor or a controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0186] The bus 1804 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 1804 may be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, Figure 10 The fact that only one line is used does not mean that there is only one bus or one type of bus.

[0187] The present application also provides a chip system. Figure 11 As shown, the chip system 2000 includes at least one processor 2001 and at least one interface circuit 2002. The processor 2001 and the interface circuit 2002 can be interconnected via lines. For example, the interface circuit 2002 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 2002 can be used to send signals to other devices (such as the processor 2001). Exemplarily, the interface circuit 2002 can read instructions stored in the memory and send the instructions to the processor 2001. When the instructions are executed by the processor 2001, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0188] An embodiment of the present application further provides a computer-readable storage medium, in which computer program code is stored. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiment.

[0189] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the relevant method steps in the above method embodiment.

[0190] Among them, the electronic device, computer-readable storage medium or computer program product provided in this application is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0191] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0193] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0194] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the contributing part or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0196] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image fusion method, characterized in that: The method is applied to an electronic device, wherein the electronic device includes a camera; the method includes: In response to a shooting operation, the camera captures a first original raw image and a second raw image; the first raw image and the second raw image are captured by the camera through a single exposure process; Calculating an image light ratio using pixel values ​​of pixels of the first raw image and pixel values ​​of pixels of the second raw image; performing image fusion on the first raw image and the second raw image according to the image light ratio to obtain a fused raw image; Display the fused raw image.

2. The method according to claim 1, characterized in that The brightness of the first raw image is higher than the brightness of the second raw image; Calculating the image light ratio using pixel values ​​of pixels of the first raw image and pixel values ​​of pixels of the second raw image includes: Filtering the pixels of the first raw image to obtain target pixels; Calculating a ratio of each target pixel to a pixel corresponding to each target pixel in the second raw image to obtain a light ratio of each target pixel; The average value of the light ratio of each target pixel is used as the image light ratio.

3. The method according to claim 2, characterized in that The filtering of the pixels of the first raw image to obtain target pixels includes: Calculating the light ratio of pixels of the first raw image; Among the pixels of the first raw image, pixels whose light ratios are outliers are removed to obtain target pixels.

4. The method according to claim 3, characterized in that The method further comprises: removing pixels whose pixel values ​​are greater than an overexposure value from pixels of the first raw image to obtain non-overexposed pixels of the first raw image; The calculating the light ratio of the pixel points of the first raw image includes: Calculating the light ratio of the non-overexposed pixel; The step of removing pixels whose light ratios are outliers from the pixels of the first raw image to obtain target pixels includes: Among the non-overexposed pixels, pixels whose light ratios are outliers are removed to obtain target pixels.

5. The method according to claim 4, characterized in that The step of removing pixels whose light ratios are outliers from the non-overexposed pixels to obtain target pixels includes: Among the non-overexposed pixels, the target pixel is obtained by removing the pixels whose light ratios are outliers according to the standard deviation of the light ratios of the non-overexposed pixels.

6. The method according to claim 5, characterized in that The step of obtaining the target pixel by removing pixels whose light ratios are outliers based on the standard deviation of the light ratios of the non-overexposed pixels includes: Constructing an inlier point set, the inlier point set including each of the non-overexposed pixels; cyclically executing a standard deviation screening process on the interior point set until the number of times the standard deviation screening process is executed equals a preset number, or until the interior point set converges; After the standard deviation screening process is cyclically performed on the inlier point set, the pixel points included in the inlier point set are used as target pixel points; The standard deviation screening process includes: Calculating the light ratio of each pixel point included in the interior point set; Calculating the mean value of the light ratios of the pixels included in the interior point set; Calculating the standard deviation of the light ratios of the pixels included in the interior point set; Pixels whose inlier point concentrated light ratio is greater than the mean + N times the standard deviation are removed, and pixels whose inlier point concentrated light ratio is less than the mean - N times the standard deviation are removed; N is a positive integer.

7. The method according to any one of claims 1 to 6, characterized in that The performing image fusion on the first raw image and the second raw image according to the image light ratio to obtain a fused raw image includes: When the exposure degree of a pixel point of the first raw image is greater than a first degree, multiplying a pixel value of a pixel point of the second raw image by the image light ratio to obtain a pixel value of the pixel point of the fused raw image; When the exposure degree of the pixel points of the first raw image is less than or equal to the first degree, the pixel values ​​of the pixel points of the first raw image are used as the pixel values ​​of the pixel points of the fused raw image.

8. The method according to claim 7, characterized in that The exposure degree of the pixels of the first raw image is greater than a first degree, comprising: The pixel value of the pixel point of the first raw image is greater than the fusion threshold; The exposure degree of the pixels of the first raw image is less than or equal to the first degree, comprising: The pixel value of the pixel point of the first raw image is less than or equal to the fusion threshold; Wherein, the fusion threshold is A*(2 M -1); the value of A is (0, 1], and M is the bit width of the first raw image.

9. A chip, characterized in that: The chip includes a processing circuit and an interface circuit, wherein the interface circuit is coupled to the processing circuit; The interface circuit is configured to: receive a third raw image and a fourth raw image; the third raw image and the fourth raw image are acquired by the camera through a single exposure process; The processing circuit is configured to: calculate an image light ratio according to pixel values ​​of pixels of the third raw image and pixel values ​​of pixels of the fourth raw image; The third raw image and the fourth raw image are fused according to the image light ratio to obtain a fused raw image.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory and a camera; the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, and when the processor executes the above-mentioned computer instructions, the electronic device executes the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 8.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method according to any one of claims 1-8.

Images