Image processing method, electronic device and chip

Abstract

The application discloses an image processing method, an electronic device and a chip, relates to the technical field of image processing, and comprises the following steps: acquiring an image frame set obtained by photographing a target object, wherein the image frame set comprises a first reference frame, a second reference frame and a plurality of image frames, the first reference frame is a fusion frame subjected to multi-frame noise reduction processing, and the exposure time length of the first reference frame is the same as that of the second reference frame; determining a target reference frame according to the definition of the first reference frame and the second reference frame; performing registration processing on the target reference frame and the plurality of image frames to obtain a registration result; and outputting a result image according to the registration result. The method can improve the image quality of the result image and meet the business requirements.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device and chip. Background Technology

[0002] With the continuous development of image processing technology, it has been widely applied to electronic devices such as mobile phones, tablets, and cameras. Based on this, electronic devices can take pictures of objects such as people, landscapes, or buildings, obtaining images of those objects, and thus recording them through images.

[0003] In actual shooting, for high-dynamic-range (HDR) scenes, continuous shooting mode (also known as burst mode) is usually used to obtain better image quality. Continuous shooting mode refers to taking multiple exposures of the same object in succession, obtaining multiple image frames, and then processing these multiple image frames, such as fusing multiple image frames with different exposure times, to output the processed image.

[0004] However, in some cases, processing such as fusing multiple image frames does not improve image quality; on the contrary, it may reduce image quality and make it difficult to meet business requirements. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, electronic device, and chip that improves image quality and meets business needs.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides an image processing method that can be executed by an electronic device. The method includes: the electronic device acquiring a set of image frames captured of a target object, the set of image frames including a first reference frame, a second reference frame, and multiple image frames, wherein the first reference frame is a fused frame after multi-frame noise reduction processing, and the exposure time of the first reference frame is the same as that of the second reference frame; then, the electronic device compares the sharpness of the first reference frame and the second reference frame to determine a target reference frame; then, it performs registration processing on the target reference frame and the multiple image frames in the image frame set to obtain a registration result; and based on the registration result, it determines a result image.

[0008] In this method, because the target reference frame has high clarity, the electronic device uses this high-clarity target reference frame for subsequent processing, which improves the image quality of the resulting image. Furthermore, the first reference frame is a fused frame obtained after multi-frame noise reduction and fusion processing. Compared to the second reference frame, this first reference frame is clearer, allowing for subsequent processing even when the second reference frame is blurry due to user hand tremors during photography. Thus, by using this high-clarity first reference frame for subsequent processing, the electronic device can ensure the image quality of the output image and improve the user experience.

[0009] The following will provide a detailed explanation of several different scenarios:

[0010] When the target reference frame is the second reference frame, meaning the second reference frame is clearer than the first reference frame, the electronic device can use this second reference frame to perform registration processing with multiple image frames to obtain a registration result. When the registration result indicates that the second reference frame and the multiple image frames can be registered, the electronic device performs fusion processing on the second reference frame and the multiple image frames, outputting a fused frame containing the second reference frame; when the registration result indicates that the second reference frame and the multiple image frames are not registered, the second reference frame is directly output, thereby ensuring the image quality of the output image.

[0011] When the target reference frame is the first reference frame, meaning the first reference frame is clearer than the second reference frame, the electronic device can use this first reference frame to perform registration processing with multiple image frames to obtain a registration result. When the registration result indicates that the first reference frame and the multiple image frames can be registered, the electronic device performs fusion processing on the first reference frame and the multiple image frames, outputting a fused frame containing the first reference frame; when the registration result indicates that the first reference frame and the multiple image frames are not registered, the first reference frame is directly output, thereby ensuring the image quality of the output image.

[0012] When the target reference frame includes a first reference frame and a second reference frame, that is, the sharpness of the first reference frame is close to that of the second reference frame. For example, the difference between the sharpness of the first reference frame and the sharpness of the second reference frame is less than a preset difference. In this case, the electronic device can perform registration processing on the first reference frame and the second image frame with multiple image frames respectively, thereby obtaining a registration result.

[0013] When the registration result indicates that the first reference frame is registered with multiple image frames, but the second reference frame is not registered with multiple image frames, the first reference frame is fused with the multiple image frames, and a fused frame containing the first reference frame is output. Conversely, when the registration result indicates that the first reference frame is not registered with multiple image frames, but the second reference frame is registered with multiple image frames, the second reference frame is fused with the multiple image frames, and a fused frame containing the second reference frame is output. In other words, when the sharpness of the first and second reference frames is similar, the electronic device uses the fused reference frame for subsequent fusion processing with multiple image frames, thereby improving the image quality of the resulting image and meeting business requirements.

[0014] When the registration result indicates that the first reference frame is registered with multiple image frames, and the second reference frame is also registered with multiple image frames, since the first reference frame is a fused frame obtained after multi-frame noise reduction and fusion processing, the electronic device can use the first reference frame to fuse with the multiple image frames and output a fused frame containing the first reference frame. Of course, the electronic device can also use the second reference frame to fuse with the multiple image frames and output a fused frame containing the second reference frame.

[0015] When the registration result indicates that the first reference frame is not registered with the multiple image frames, and the second reference frame is also not registered with the multiple image frames, the electronic device can directly output the first reference frame, since the first reference frame is a fused frame obtained after multi-frame noise reduction and fusion processing. Of course, the electronic device can also directly output the second reference frame.

[0016] In some possible implementations, the target reference frame is obtained based on the sharpness of a first reference frame and a second reference frame. This target reference frame can be the one with higher sharpness than the first or second reference frame. Therefore, the electronic device can use this target reference frame as a reference frame for subsequent frame output, such as outputting multiple image frames. Because the target reference frame has higher sharpness, its exposure time can be reduced, resulting in a higher signal-to-noise ratio. With the exposure time of the target reference frame (reference frame) reduced, the exposure time of the sequentially output image frames by the electronic device also decreases, thereby enabling the acquisition of more image details.

[0017] Furthermore, since the exposure time of the reference frame is shortened (i.e., the exposure time of the short exposure frame is even shorter), and the method also introduces a first image frame whose exposure time is longer than that of the first reference frame (i.e., the exposure time of the long exposure frame is even longer), the resulting image obtained by the electronic device after fusing the target reference frame with multiple image frames has a wider dynamic range, further improving the image quality of the resulting image.

[0018] In some possible implementations, the multi-frame image includes a first image frame and a second image frame, wherein the exposure time of the first image frame is greater than the exposure time of the first reference frame, and the exposure time of the second image frame is less than the exposure time of the first reference frame.

[0019] Secondly, this application provides an electronic device, including: a memory and a processor;

[0020] The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the electronic device performs the method as described in any one of the first aspects.

[0021] Thirdly, this application provides a computer storage medium including computer instructions that, when executed on a mobile terminal, cause the electronic device to perform the method as described in any one of the first aspects.

[0022] Fourthly, this application provides a computer program product including instructions; when the instructions are executed by an electronic device, the electronic device performs the method as described in any one of the first aspects.

[0023] Fifthly, this application provides a chip for processing image signals acquired by a camera to perform the steps of the method as described in any of the first aspects.

[0024] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0025] Figure 1 A schematic diagram of an electronic device provided in an embodiment of this application;

[0026] Figure 2A A schematic diagram of a desktop interface provided in an embodiment of this application;

[0027] Figure 2BA schematic diagram of a camera interface provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram of an image signal processor provided in an embodiment of this application;

[0029] Figure 4 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0030] Figure 5A A schematic diagram of a result image provided in an embodiment of this application;

[0031] Figure 5B A schematic diagram of yet another resulting image provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0034] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0036] Multi-frame fusion technology refers to the process of fusing multiple dynamic range images with different exposure times to obtain a high dynamic range image. High dynamic range images have a wider brightness range, thus exhibiting better visual effects.

[0037] However, in some cases, the image quality of the merged image deteriorates after processing such as fusing multiple image frames. For example, when there are low-quality images among multiple dynamic range images with different exposure times, fusing these multiple dynamic range images with different exposure times will actually reduce the image quality of the high dynamic range image, making it difficult to meet business requirements.

[0038] In view of this, embodiments of this application provide an image processing method that can be executed by an electronic device. Specifically, the method includes: the electronic device acquiring a set of image frames obtained by photographing a target object, the set of image frames including a first reference frame, a second reference frame, and multiple image frames, wherein the first reference frame is an image frame that has undergone noise reduction processing, and the exposure time of the first reference frame is the same as that of the second reference frame; then, the electronic device can compare the sharpness of the first reference frame and the second reference frame to determine a target reference frame, and then use the target reference frame for subsequent processing. The electronic device performs registration processing with the target reference frame and the multiple image frames to obtain a registration result; then, the electronic device outputs a result image based on the registration result.

[0039] In this method, the first reference frame is an image frame that has undergone multi-frame noise reduction processing, and a second reference frame is also introduced. Before fusing the multi-frame image, the electronic device compares the sharpness of the first and second reference frames. Thus, the electronic device can select the image frame with higher sharpness between the first and second reference frames as the target reference frame. Then, the electronic device performs registration processing on the target reference frame and the multi-frame image to obtain a registration result. Based on this registration result, the result image is determined. If the registration result indicates that the target reference frame and the multi-frame image can be registered, the electronic device performs fusion processing on the target reference frame and the multi-frame image; otherwise, the electronic device directly outputs the target reference frame. Because the target reference frame has higher sharpness, the resulting image also has higher sharpness, thereby improving the quality of the resulting image and meeting business requirements.

[0040] In some examples, the electronic device can be a camera, mobile phone, tablet computer, desktop, laptop, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the aforementioned electronic device. In this embodiment, the structure of the electronic device can be as follows: Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0041] like Figure 1As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0042] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0043] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, in this application, processor 110 may be an image signal processor.

[0044] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0045] Generally, an image signal processor includes modules such as an image front end (IFE), a Bayer processing segment (BPS), an image processing engine (IPE), and an encoder. Optionally, the encoder can be based on the JPEG (Joint Photographic Experts Group) algorithm.

[0046] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0047] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. After the photosensitive element converts the light signal into an electrical signal, it transmits the electrical signal to an ISP for conversion into a digital image signal. The ISP processes the digital image signal to obtain an image signal in formats such as YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0048] like Figure 2A As shown, the electronic device can present a desktop interface to the user, which includes multiple applications. The user can launch an application by clicking its icon. Based on this click, the electronic device can launch the camera application. For example, the user can click the camera application icon 201, and the electronic device can launch the camera application based on this click.

[0049] like Figure 2BAs shown, this figure is a schematic diagram of a camera interface provided in an embodiment of this application. The electronic device can provide multiple shooting modes, such as front-facing HDR shooting mode and rear-facing HDR shooting mode. The camera interface includes a shooting control 210, a mode selection control 220, and an image preview area 230. The shooting control 210 is used to trigger a shooting event, the mode selection control 220 is used to switch shooting modes, and the image preview area 230 is used to preview the shooting scene. Figure 2B A schematic diagram of a camera interface for a front-facing HDR shooting mode is shown. The image processing method provided in this application embodiment can be applied to scenarios in front-facing HDR shooting mode. Of course, in other embodiments, this image processing method can also be applied to scenarios in rear-facing HDR shooting mode. It can also be used for scenarios such as shooting videos or video calls.

[0050] Taking the front-facing HDR shooting mode as an example, after the electronic device launches the camera application, camera 193 begins to capture a preview frame sequence of the target object and stores it in a buffer for user preview or use in other steps. The target object can be a person, landscape, building, etc. In some examples, the preview frame sequence may be as shown in Table 1 below:

[0051] Table 1

[0052] Image frame Preview Frame 1 Preview Frame 2 … Preview Frame 7 Preview Frame 8 type N N N N N EV value 0 0 0 0 0

[0053] Here, type N (normal) indicates that the image frame is a normal exposure frame, meaning the exposure time is the baseline time; EV (exposure values) is used to characterize the exposure time of the image frame. Specifically, ev = 0 indicates that the exposure time of the image frame is the baseline time, such as 100ms or 30ms.

[0054] It should be noted that the pre-buffer stores multiple preview frames. Table 1 above only shows the 8 preview frames retrieved by the electronic device when the user triggers the photo-taking event; it is also possible to retrieve 5 preview frames.

[0055] During the process of the electronic device presenting a preview image of the target image to the user, the user can click the shooting control 210 to trigger a photo-taking event. After the electronic device detects the user's click operation on the shooting control 210, it acquires a set of image frames for subsequent output of the result image.

[0056] In some examples, the set of image frames can be as shown in Table 2 below:

[0057] Table 2

[0058] Image frame Reference Frame 1 Reference Frame 2 Image Frame 1 Image frame 2 Image frame 3 Image frame 4 type N N L S ES ES EV value 0 0 0.5 -2 -4 -5

[0059] Where L (long) indicates that the image frame is a long exposure frame, meaning the exposure time is longer than the reference time; S (short) indicates that the image frame is a short exposure frame, meaning the exposure time is shorter than the reference time; and ES (extra short) indicates that the image frame is an ultra-short exposure frame, meaning the exposure time is shorter than the short exposure frame. ev = 0.5 indicates that the exposure time of the image frame is longer than the reference time. When the reference time is 100ms, the exposure time of image frame 1 is 141.42ms, similarly, the exposure time of image frame 2 is 25ms, the exposure time of image frame 3 is 6.25ms, and the exposure time of image frame 4 is 3.125ms. Reference frame 1 is the fused frame obtained after multi-frame noise reduction processing of the preview frames in Table 1. Reference frame 2 and image frame 1 are newly added image frames, while image frames 2, 3, and 4 are existing image frames, which will be described below. See also... Figure 3 As shown, when a photo-taking event occurs, the data fed back by the camera (sensor) first reaches the IFE (In-Frame Filter). For example, the data fed back by the camera can be 5 or 8 frames of raw image data, such as the 8 preview frames shown in Table 1 above. The IFE can first perform color correction and other processing. Then, noise reduction processing can be performed on the 8 preview frames output by the IFE based on BPS (Browser Precision Processing) and IPE (In-Frame Processing). In BPS, noise reduction processing such as removing bad pixels, phase focusing, and de-mosaic can be performed on the image frames output by the IFE. In IPE, multi-frame noise reduction (MFNR) technology can be used to perform multi-frame noise reduction and fusion processing on the 8 preview frames output by the IFE to obtain the first reference frame (i.e., reference frame 1 shown in Table 2).

[0060] The exposure table of reference frame 2 is the same as that of reference frame 1. In this application, reference frame 2 can be used as a reference frame to adjust subsequent output frames, such as outputting image frame 1, image frame 2, image frame 3, and image frame 4. When the user triggers a photo-taking event, the electronic device outputs reference frame 2, image frame 1, image frame 2, image frame 3, and image frame 4 in sequence.

[0061] See also Figure 3The ISP can compare the sharpness between the first and second reference frames to determine the target reference frame. This target reference frame is the one with a sharpness no lower than the other reference frame; that is, the reference frame with higher sharpness is selected from the first and second reference frames for subsequent processing. Then, this target reference frame is used to perform registration processing with subsequent multiple image frames (e.g., image frames 1 to 4). When the target reference frame and the multiple image frames meet the preset registration conditions, the target reference frame and the multiple image frames are fused, and a fused frame containing the target reference frame is output. When the target reference frame and the multiple image frames do not meet the preset registration conditions, the target reference frame is directly output. The preset registration conditions can be that a portion (e.g., 80%) of the multiple images is logically identical, meaning that a portion of the multiple images reflects the same target area. This results in a higher quality output image and improves the user experience.

[0062] The electronic device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The display screen 194 is used to display images, such as the resulting image described above.

[0063] To make the technical solution of this application clearer and easier to understand, the image processing method provided in the embodiments of this application will be described below from the perspective of an electronic device, in conjunction with the accompanying drawings. Figure 4 As shown, this figure is a flowchart of an image processing method provided in an embodiment of this application. The method includes:

[0064] S401: The electronic device acquires a set of image frames obtained by capturing images of the target object.

[0065] The image frame set includes a first reference frame (reference frame 1 as shown in Table 2 above), a second reference frame (reference frame 2 as shown in Table 2 above), and multiple image frames (image frames 1 to 4 as shown in Table 2 above). Among them, the first reference frame is a fused frame after noise reduction and fusion processing, and the first reference frame and the second reference frame have the same exposure time.

[0066] As mentioned above Figure 3As shown, when a user triggers a photo capture event, the camera outputs 8 raw images (or 5 or 6 raw images) to the IFE (Integrated Front-End). After IFE processing, through pre-filtering, blending, and post-filtering, noise reduction can be performed on the IFE output image based on BPS (Browser Per Second) and IPE (Integrated Per Second), resulting in a noise-reduced and blended YUV image, which is used as the first reference frame.

[0067] In some examples, when a user triggers a photo-taking event, the electronic device can sequentially acquire a first reference frame, a second reference frame, and multiple image frames. To enable the electronic device to achieve WYSIWYG (What You See Is What You Get) functionality, it can utilize zero-shutter-lag (ZSL) technology to acquire multiple preview frames. When a photo-taking event occurs (e.g., the user triggers the shutter), the electronic device retrieves these preview frames from the preview buffer, performs multi-frame noise reduction and fusion processing, and thus obtains the first reference frame. The second reference frame and the multiple image frames are obtained sequentially. Therefore, the delay between the first and second reference frames acquired by the electronic device is relatively large, for example, 300ms, while the delay between the second reference frame and the multiple image frames is relatively small, for example, 20ms.

[0068] It should be noted that the electronic device's method of performing multi-frame noise reduction and fusion processing on multiple preview frames retrieved from the preview buffer to obtain the first reference frame is merely an example. In other embodiments, the electronic device may also perform multi-frame noise reduction and fusion processing on the 8 image frames before the user triggers the shutter event (e.g., image frames before reference frame 2) to obtain the first reference frame.

[0069] The exposure time of the first reference frame differs from that of each image frame in the multi-frame image set. Taking an exposure time of 100ms for the first reference frame as an example, the exposure time of image frame 2 in the multi-frame image set could be 25ms, image frame 3 could be 6.25ms, and image frame 4 could be 3.125ms. Thus, the dynamic range of the fused frame obtained after fusing the first reference frame with image frames 2, 3, and 4 is wider. Furthermore, in this embodiment, an additional image frame 1 is added, whose exposure time is longer than that of the first reference frame; the exposure time of image frame 1 could be 141.42ms. Due to the introduction of image frame 1 with a longer exposure time, the dynamic range of the fused frame obtained after fusing the first reference frame with image frames 1 to 4 is further widened, improving the image quality of the resulting image. Similarly, fusing the second reference frame with image frames 1 to 4 can also further improve the dynamic range of the fused frame obtained after the fusion process.

[0070] S402: The electronic device determines the target reference frame based on the sharpness of the first reference frame and the second reference frame.

[0071] Image sharpness can refer to the level of detail in the picture. In some examples, an electronic device can calculate the sharpness of a first reference frame and a second reference frame separately, then compare the sharpness of the first reference frame and the second reference frame to determine the reference frame with higher sharpness between the first reference frame and the second reference frame as the target reference frame.

[0072] Taking the calculation of the sharpness of a first reference frame by an electronic device as an example, the device can calculate the square of the gray-level difference between each pair of adjacent pixels in the first reference frame based on a gradient function (such as the Brenner function), then sum the squares of the gray-level differences between each pair of adjacent pixels and calculate the mean. This mean is used as the sharpness of the first reference frame; the larger the mean, the sharper the image. Of course, in other examples, the electronic device can also calculate the sharpness of the first reference frame based on other gradient functions (such as the tenengrad function or the laplacian function). Similarly, the electronic device can calculate the sharpness of a second reference frame in the same way.

[0073] This application does not specifically limit the method by which the electronic device calculates the sharpness of the first reference frame or the second reference frame. Those skilled in the art can choose other methods to calculate the sharpness of the first reference frame or the second reference frame according to actual needs.

[0074] The electronic device can determine the sharpness of the first reference frame and the second reference frame. When the sharpness of the first reference frame is higher than that of the second reference frame, the target reference frame is the first reference frame. When the sharpness of the second reference frame is higher than that of the first reference frame, the target reference frame is the second reference frame. When the sharpness of the first reference frame is the same as that of the second reference frame, the target reference frame can be either the first reference frame or the second reference frame.

[0075] It should be noted that the same sharpness between the first reference frame and the second reference frame can mean that the difference between the sharpness of the first reference frame and the second reference frame is less than a preset difference, which indicates that the sharpness of the first reference frame is close to that of the second reference frame.

[0076] S403: The electronic device performs registration processing on the target reference frame and multiple image frames to obtain the registration result.

[0077] In some examples, the electronic device can also register the target reference frame with higher resolution between the first and second reference frames with multiple image frames to obtain the registration result of the target reference frame and the multiple image frames. In this embodiment, the electronic device performs subsequent image processing based on the target reference frame with higher resolution, which can improve the resolution of the output image, enhance the detail of the image, and improve the image quality of the result image.

[0078] Taking the registration process of a first reference frame and multiple image frames by an electronic device as an example, the electronic device can output a pixel displacement matrix for the red, green, and blue (RGB) channels based on the pixel displacements between the first reference frame and the multiple image frames to be fused. This pixel displacement matrix is ​​then subjected to Gaussian blurring to obtain a Gaussian blur matrix, which is input into the meanStdDev function to obtain the standard deviation. When the standard deviation is greater than a preset threshold, it indicates that the first reference frame and the multiple image frames are not registered; when the standard deviation is less than or equal to the preset threshold, it indicates that the first reference frame and the multiple image frames can be registered. Similarly, the electronic device can also perform registration processing on a second reference frame and multiple image frames in a similar manner to obtain the registration result of the second reference frame and the multiple image frames.

[0079] The embodiments of this application do not specifically limit the method by which the electronic device performs registration processing on the target reference frame and multiple image frames. Those skilled in the art can choose other methods to perform registration processing on the target reference frame and multiple image frames according to actual needs, such as considering the efficiency of the algorithm, in order to obtain the registration result.

[0080] S404: The electronic device outputs the resulting image based on the registration result.

[0081] Different registration results result in different images. Specifically, the resulting image can be a first reference frame, a second reference frame, a first fused frame, or a second fused frame, where the first fused frame is a fused frame of the first reference frame and multiple image frames, and the second fused frame is a fused frame of the second reference frame and multiple image frames.

[0082] In some examples, the electronic device can determine a reference frame whose sharpness is no less than that of the other reference frame between the first and second reference frames; that is, it determines the target reference frame. The electronic device then combines the registration results to output the target reference frame or a fused frame containing the target reference frame. Specifically, when the registration results indicate that the target reference frame is registered with multiple image frames, the target reference frame is fused with the multiple image frames to output a fused frame containing the target reference frame. For example, the electronic device performs weight calculations based on the target reference frame and the multiple image frames to obtain the fused frame. When the registration results indicate that the target reference frame is not registered with the multiple image frames, the target reference frame is output.

[0083] For example, the fact that the sharpness of the first reference frame is not lower than that of the second reference frame can include two cases: the sharpness of the first reference frame is higher than that of the second reference frame, and the sharpness of the first reference frame is the same as that of the second reference frame.

[0084] When the sharpness of the first reference frame is higher than that of the second reference frame, the electronic device determines the first reference frame as the target reference frame. The electronic device then combines the registration results to output either the first reference frame or the first fused frame. Specifically, when the registration results indicate that the first reference frame is registered with multiple image frames, the electronic device fuses the first reference frame with the multiple image frames and outputs the first fused frame; when the registration results indicate that the first reference frame and the multiple image frames are not registered, the electronic device directly outputs the first reference frame.

[0085] In some scenarios, after a user triggers the shutter, the movement of the handheld electronic device causes the camera to shift. This can result in poor clarity in the second reference frame and subsequent multi-frame image, which are generated sequentially by the electronic device; for example, the second reference frame and the multi-frame image may appear blurry. The first reference frame is obtained by the electronic device from multiple preview frames retrieved from the preview buffer based on the moment of the shooting event, and then performing multi-frame noise reduction and fusion processing. Therefore, even if the user experiences hand shakiness after triggering the shutter, the electronic device can use this first reference frame as the final image, ensuring image quality and improving the user's photography experience.

[0086] When the sharpness of the first reference frame is consistent with that of the second reference frame, the electronic device determines either the first or second reference frame as the target reference frame. The electronic device then combines the registration results to output either a first fused frame or a second fused frame. Specifically, when the registration result indicates that the first reference frame and multiple image frames are registered, the electronic device fuses the first reference frame with the multiple image frames and outputs a first fused frame. When the registration result indicates that the second reference frame and multiple image frames are registered, the electronic device fuses the second reference frame with the multiple image frames and outputs a second fused frame. When the registration result indicates that both the first and second reference frames are registered, the electronic device can choose either the first or second reference frame to fuse with the multiple image frames, using the resulting fused frame as the final image. Since the first reference frame is a fused frame obtained after multi-frame noise reduction and fusion processing, the electronic device can also fuse the first reference frame with the multiple image frames, using the resulting first fused frame as the final image, further improving the image quality of the final image.

[0087] Similarly, the statement that the sharpness of the second reference frame is not lower than that of the first reference frame can also include two cases: the sharpness of the second reference frame is higher than that of the first reference frame, and the sharpness of the second reference frame is the same as that of the first reference frame.

[0088] When the clarity of the second reference frame is higher than that of the first reference frame, the electronic device determines the second reference frame as the target reference frame. The electronic device combines the registration result and outputs the second reference frame or the second fused frame. Specifically, when the registration result indicates that the second reference frame is registered with multiple image frames, the electronic device performs fusion processing on the second reference frame and multiple image frames and outputs the second fused frame. When the registration result indicates that the second reference frame is not registered with multiple image frames, the electronic device directly outputs the second reference frame.

[0089] When the sharpness of the first reference frame is the same as that of the second reference frame, the process of the electronic device outputting the resulting image can be referred to the example above, and will not be repeated here.

[0090] In some embodiments, the multi-frame image includes multiple image frames with different exposure times. For example, the multi-frame image includes image frame 2, image frame 3, and image frame 4. The exposure time of image frame 2 is longer than that of image frame 3, the exposure time of image frame 3 is longer than that of image frame 4, and the exposure time of the target reference frame is longer than that of image frame 2. The electronic device can perform fusion processing on the aforementioned image frames 2, 3, and 4 with the target reference frame, thereby improving the dynamic range of the fused frame obtained after fusion processing.

[0091] In other examples, the multi-frame image also includes image frame 1, whose exposure time is longer than that of the target reference frame, which in turn is longer than that of image frames 2, 3, and 4. Image frame 1 has a longer exposure time, meaning the exposure time of a long-exposure frame becomes longer. The target image frame is the image frame with higher clarity between the first and second reference frames. Taking the first reference frame as an example, this first reference frame is a fused frame after multi-frame noise reduction and fusion processing. The electronic device does not need a longer exposure time to achieve a higher signal-to-noise ratio for the first reference frame, thus reducing its exposure time. Because the reference time is reduced, the exposure time of the multi-frame image obtained by sequentially extracting the frames also decreases; that is, the exposure time of short-exposure frames (such as image frames 2 to 4 as shown in Table 2) becomes shorter, allowing for the acquisition of more image details. After the electronic device fuses the first reference frame with the multi-frame image, the resulting fused frame has a wider dynamic range, further improving the visual effect of the fused frame.

[0092] Based on this, the image processing method provided in this application can be applied to scenes with poor lighting conditions, such as shooting street scenes at dusk. In some examples, the electronic device can set the camera's ISO sensitivity to 1000 or other values ​​greater than 1000, such as 1250, 1600, etc. This application also introduces long-exposure frames (such as image frame 1 shown in Table 2). Thus, after fusing multiple image frames and the target reference frame, the resulting fused frame has a wider dynamic range. Especially in scenes with poor lighting conditions, the electronic device can also capture images with better visual effects, meeting business requirements.

[0093] like Figure 5A and Figure 5B As shown, Figure 5A This diagram illustrates an image obtained using a conventional method. Figure 5B This is a schematic diagram of an image obtained after processing by the image processing method provided in the embodiments of this application. From Figure 5A and Figure 5B It can be seen from this that Figure 5B and Figure 5A Compared to the former, it is not only clearer, but also has a wider dynamic range.

[0094] In other examples, the electronic device can select a predetermined number of image frames whose sharpness meets a preset condition from image frames 2 to 4, and then perform a fusion process on the target reference frame, the predetermined number of image frames, and image frame 1 to obtain a fused frame. The predetermined number can be an even number, and the predetermined number of image frames whose sharpness meets the preset condition can be the top even-numbered image frames in terms of sharpness.

[0095] For example, in the above multi-frame image, the clarity of image frame 2 is greater than that of image frame 3, and the clarity of image frame 4 is greater than that of image frame 2. The electronic device can perform fusion processing on image frame 2, image frame 4, image frame 1, and the target reference frame.

[0096] This application does not specifically limit the method by which electronic devices fuse target reference frames and multiple image frames. Those skilled in the art can choose an appropriate method to perform fusion processing according to actual needs.

[0097] Based on the above description, this application provides an image processing method. The method introduces a second reference frame and, based on the sharpness of the second reference frame and the first reference frame, determines a target reference frame for subsequent processing. This target reference frame is the one between the first and second reference frames whose sharpness is not lower than that of the other reference frame. Using this target reference frame for subsequent processing can improve the sharpness of the resulting image. The first reference frame is a fused frame obtained after multi-frame noise reduction and fusion processing. Compared to the second reference frame, this first reference frame is sharper, thus enabling subsequent processing even when the second reference frame is blurry due to user hand tremors during photography. In this way, the electronic device can determine the reference frame with higher sharpness between the first and second reference frames, i.e., determine the target reference frame. Then, the electronic device performs registration processing on the target reference frame with multiple image frames. If the target reference frame can be registered with the multiple image frames, then the target reference frame is fused with the multiple image frames. Otherwise, the electronic device directly outputs the target reference frame. Because the target reference frame has high resolution, the resulting image also has high resolution, thus improving the quality of the resulting image frame and meeting business requirements.

[0098] Furthermore, before performing multi-frame fusion, the electronic device compares the sharpness of the first and second reference frames. It then fuses the target reference frame, which has higher sharpness, with the multi-frame image. Because the target reference frame has higher sharpness, a lower exposure time is achieved, resulting in a higher signal-to-noise ratio. For the multi-frame image, the exposure time is also shortened, allowing for the capture of more image details. This further improves the dynamic range of the resulting image after fusion. The lower exposure time also reduces the probability of ghosting, further enhancing the quality of the final image.

[0099] Furthermore, during the fusion process, the electronic device introduces a long-exposure frame (as shown in image frame 1 in Table 2) into the multi-frame image set. The exposure time of this long-exposure frame is longer than that of the target reference frame. Thus, the long-exposure frame has a longer exposure time, and the short-exposure frame has a shorter exposure time. After fusing the multi-frame image set with the target reference frame, the electronic device further improves the dynamic range of the resulting image and enhances its image quality.

[0100] Some embodiments of this application also provide an electronic device, such as... Figure 6As shown, the electronic device may include one or more cameras 601, one or more processors 602, a memory 603, and one or more computer programs 604. These devices can be connected via one or more communication buses 605. The one or more computer programs 604 are stored in the memory 603 and configured to be executed by the one or more processors 602. The one or more computer programs 604 include instructions that can be used to perform actions such as... Figure 4 The various steps performed by the mobile phone in the corresponding embodiment. Of course, Figure 6 The electronic device shown may also include other components such as a sensor module, an audio module, and a SIM card interface; however, this application does not impose any limitations on these components.

[0101] Optionally, the camera 601 is used to acquire a set of image frames obtained by shooting a target object. The set of image frames includes a first reference frame, a second reference frame, and multiple image frames. The first reference frame is a fused frame after multi-frame noise reduction processing, and the exposure time of the first reference frame is the same as the exposure time of the second reference frame. The processor 602 is used to determine a target reference frame based on the sharpness of the first reference frame and the second reference frame; perform registration processing on the target reference frame and the multiple image frames to obtain a registration result; and output a result image based on the registration result.

[0102] Optionally, when the target reference frame is the second reference frame, the processor 602 is specifically used to fuse the second reference frame with the multiple image frames when the registration result indicates that the second reference frame is registered with the multiple image frames, and output a fused frame containing the second reference frame.

[0103] Optionally, the processor 602 is specifically configured to output the second reference frame when the registration result indicates that the second reference frame and the multi-frame image frames are not registered.

[0104] Optionally, when the target reference frame is the first reference frame, the processor 602 is specifically used to fuse the first reference frame and the multiple image frames when the registration result indicates that the first reference frame is registered with the multiple image frames, and output a fused frame containing the first reference frame.

[0105] Optionally, the processor 602 is specifically configured to output the first reference frame when the registration result indicates that the first reference frame and the multi-frame image frames are not registered.

[0106] Optionally, when the target reference frame includes the first reference frame and the second reference frame, the processor 602 is specifically used for:

[0107] When the registration result indicates that the first reference frame is registered with the multiple image frames, and the second reference frame is not registered with the multiple image frames, the first reference frame and the multiple image frames are fused together, and a fused frame containing the first reference frame is output; or,

[0108] When the registration result indicates that the first reference frame and the multiple image frames are not registered, and the second reference frame is registered with the multiple image frames, the second reference frame and the multiple image frames are fused together, and a fused frame containing the second reference frame is output; or,

[0109] When the registration result indicates that the first reference frame is registered with the multiple image frames, and the second reference frame is registered with the multiple image frames, the first reference frame and the multiple image frames are fused together, and a fused frame containing the first reference frame is output; or,

[0110] When the registration result indicates that the first reference frame is not registered with the multi-frame image, and the second reference frame is not registered with the multi-frame image, the first reference frame is output.

[0111] This application also provides a chip for processing image signals captured by a camera to perform the methods described in the above embodiments.

[0112] This embodiment also provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform... Figure 4 The relevant methods and steps are described in the above embodiments to implement the methods described.

[0113] This embodiment also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform actions such as... Figure 4 The relevant methods and steps are described in the above embodiments to implement the methods described.

[0114] This embodiment also provides a control device, which includes a processor and a memory. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the control device performs actions such as... Figure 4 The relevant method steps implement the methods in the above embodiments. The control device can be an integrated circuit (IC) or a system-on-a-chip (SoC). The integrated circuit can be a general-purpose integrated circuit, a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0115] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0116] In the several embodiments provided in this example, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, in each embodiment of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: A set of image frames obtained by capturing images of a target object is acquired. The set of image frames includes a first reference frame, a second reference frame, and multiple image frames. The first reference frame is a fused frame obtained by performing multi-frame noise reduction and fusion processing on multiple preview frames in the preview buffer. The exposure time of the first reference frame is the same as that of the second reference frame. The second reference frame and the multiple image frames are obtained by sequentially capturing frames in response to a shooting event, and the exposure time of the first reference frame and each of the image frames is different. The target reference frame is determined based on the sharpness of the first reference frame and the second reference frame; Based on the pixel displacement between the target reference frame and the multiple image frames, the target reference frame and the multiple image frames are registered to obtain a registration result; When the registration result indicates that the target reference frame and the multi-frame image frames are registered, the fused frame of the target reference frame and the multi-frame image frames is output. If the registration result indicates that the target reference frame is not registered with the multi-frame image frame, the target reference frame is output.

2. The method according to claim 1, characterized in that, When the target reference frame is the second reference frame, the step of outputting the fused frame of the target reference frame and the multi-frame image frames includes: When the registration result indicates that the second reference frame is registered with the multi-frame image, the second reference frame and the multi-frame image are fused together to output a fused frame containing the second reference frame.

3. The method according to claim 1, characterized in that, When the target reference frame is the second reference frame, outputting the target reference frame includes: When the registration result indicates that the second reference frame is not registered with the multi-frame image frame, the second reference frame is output.

4. The method according to claim 1, characterized in that, When the target reference frame is the first reference frame, the step of outputting the fused frame of the target reference frame and the multi-frame image frames includes: When the registration result indicates that the first reference frame is registered with the multiple image frames, the first reference frame and the multiple image frames are fused together to output a fused frame containing the first reference frame.

5. The method according to claim 1, characterized in that, When the target reference frame is the first reference frame, outputting the target reference frame includes: When the registration result indicates that the first reference frame is not registered with the multi-frame image, the first reference frame is output.

6. The method according to claim 1, characterized in that, When the target reference frame includes the first reference frame and the second reference frame, the step of outputting the result image based on the registration result includes: When the registration result indicates that the first reference frame is registered with the multiple image frames, and the second reference frame is not registered with the multiple image frames, the first reference frame and the multiple image frames are fused together, and a fused frame containing the first reference frame is output; or, When the registration result indicates that the first reference frame and the multiple image frames are not registered, and the second reference frame is registered with the multiple image frames, the second reference frame and the multiple image frames are fused together, and a fused frame containing the second reference frame is output; or, When the registration result indicates that the first reference frame is registered with the multiple image frames, and the second reference frame is registered with the multiple image frames, the first reference frame and the multiple image frames are fused together, and a fused frame containing the first reference frame is output; or, When the registration result indicates that the first reference frame is not registered with the multi-frame image, and the second reference frame is not registered with the multi-frame image, the first reference frame is output.

7. The method according to any one of claims 1-6, characterized in that, The multi-frame image includes a first image frame and a second image frame, wherein the exposure time of the first image frame is greater than the exposure time of the first reference frame, and the exposure time of the second image frame is less than the exposure time of the first reference frame.

8. An electronic device, characterized in that, include: Memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1 to 7.

9. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes instructions; when the instructions are executed by the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images