A night scene photographing method, an electronic device, and a storage medium

Abstract

The application provides a night scene shooting method, an electronic device and a storage medium, and relates to the technical field of terminals. The method comprises the following steps: after receiving a night scene shooting instruction issued by a user to a terminal device, the terminal device first determines a reasonable framing mode and an exposure time of a camera in response to the instruction, and then acquires a first target image with low sensitivity and a second target image with high sensitivity through the camera under the framing mode and the exposure time. The first target image and the second target image are two images about the same night scene shot by the camera. Then, image fusion processing is performed on the two images to obtain a fused target image, and a pre-constructed noise model is fused and processed by using the two images to obtain a fused noise model. Furthermore, the fused noise model is used to perform noise reduction and tone mapping processing and post-processing of the night scene on the fused target image, so that a night scene shooting image with better visual effect is obtained.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a night scene photography method, electronic device and storage medium. Background Technology

[0002] With the continuous development of technology, terminal devices, represented by mobile phones and tablets, are increasingly used in people's lives and work, bringing great convenience. For example, people can use these devices to take photos or record videos in various scenarios, both day and night.

[0003] However, when users take photos in high dynamic range (HDR) night scene scenarios using terminal devices, they are easily affected by the complex lighting in the surrounding environment, resulting in problems such as banding. This means that there will be regular stripes of light and dark on the captured images, resulting in poor visual effects of the photos. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a night scene photography method, electronic device, and storage medium. The aim is to eliminate banding and other problems in night scene photos by performing HDR dual-frame image fusion followed by noise reduction processing, combined with a noise model calibration-fusing construction method. This eliminates regular alternating bright and dark stripes in the captured night scene photos, thereby improving the visual effect of night scene photos and the user's photography experience.

[0005] Firstly, this application provides a night scene photography method, which includes: upon receiving a night scene photography command from a user to a terminal device, the terminal device first responds to the command, flexibly determines a reasonable frame output mode and exposure time for the camera based on the dynamic range requirements and banding anti-flicker requirements of the current high dynamic range night scene, and then, under the frame output mode and exposure time, acquires a first target image with a first sensitivity (i.e., low sensitivity) and a second target image with a second sensitivity (i.e., high sensitivity) through the camera. The first target image and the second target image are two frames of the same night scene captured by the camera. Next, image fusion processing is performed on these two frames to obtain a fused target image. Simultaneously, these two frames can be used to fuse a pre-constructed noise model (which can be obtained by calibrating the noise of N training image datasets at different sensitivities and fitting the corresponding noise models for each of the N training image datasets) to obtain a fused noise model. Furthermore, the fused noise model can be used to perform noise reduction, tone mapping, and night scene post-processing on the fused target image to obtain a night scene photography image with better visual effects.

[0006] As can be seen, in the above-mentioned night scene photography method, this embodiment, when shooting HDR night scene, first performs HDR dual-frame image fusion and then performs noise reduction processing, combined with the noise model first calibrated and then fused construction method, to eliminate problems such as banding in night scene photos, that is, to eliminate the regular alternating bright and dark stripes on the captured night scene photos, thereby improving the visual effect of night scene photos and the user's photography experience.

[0007] In one possible implementation, the outgoing frame mode can be bracketing exposure mode, dual conversion gain mode, or lateral overflow collection capacitance mode.

[0008] In one possible implementation, the method may further include: acquiring N training image datasets under N preset working modes of the camera; wherein the N training image datasets include training images at a first sensitivity and training images at a second sensitivity; and N is a positive integer greater than 1; performing noise calibration on each of the N training image datasets, and fitting a noise model corresponding to each of the N training image datasets during the calibration process. This improves the accuracy of constructing each noise model, thereby improving the accuracy of subsequent noise model fusion modeling.

[0009] In one possible implementation, noise calibration is performed on N training image datasets, and during the calibration process, a noise model corresponding to each of the N training image datasets is fitted. This can include: obtaining image information of each training image in the h-th training image dataset from the N training image datasets; where h is a positive integer greater than 0 and not greater than N; using the image information of each training image in the h-th training image dataset, calculating the mean and variance information corresponding to the h-th training image dataset; fitting these mean and variance information to the information of different analog gains in the h-th training image dataset to obtain the relationship curve between the mean and variance information; fitting the relationship between the coefficients of the relationship curve between the mean and variance information and the analog gain to obtain the noise model corresponding to the h-th training image dataset, and so on, fitting the noise model corresponding to each of the N training image datasets. This improves the fitting effect of the noise model.

[0010] In one possible implementation, image fusion processing is performed on the first target image and the second target image to obtain a fused target image. This may include: fusing corresponding pixels in the first target image and the second target image according to a preset fusion coding threshold, a fusion weight corresponding to the first target image, and a fusion weight corresponding to the second target image, to obtain the fused target image. This improves the accuracy of the fusion result.

[0011] In one possible implementation, a pre-built noise model is fused using a first target image and a second target image to obtain a fused noise model. This includes: determining the noise level corresponding to the s-th pixel in the first target image with a first sensitivity based on the pre-built noise model, where s is a positive integer greater than 0; determining the noise level corresponding to the s-th pixel in the second target image with a second sensitivity based on the pre-built noise model; calculating a first product of the noise level corresponding to the s-th pixel in the first target image and the fusion weight corresponding to the first target image; and calculating a second product of the noise level corresponding to the s-th pixel in the second target image and the fusion weight corresponding to the second target image; calculating the sum of the first and second products as the pixel noise level corresponding to the s-th pixel output by the fused noise model, and so on, to obtain the fused noise level of each pixel output by the fused noise model. This improves the accuracy of the fusion result.

[0012] In one possible implementation, the fused target image is denoised using a fused noise model, and then tone mapping and night scene post-processing are performed on the denoised target image to obtain a night scene photograph. This can include: denoising the fused target image using the fused noise levels of each pixel output by the fused noise model to obtain a denoised target image; performing global and local contrast processing on the denoised target image using a preset tone mapping method to obtain a contrast-processed target image; and performing night scene post-processing on the contrast-processed target image to generate a night scene photograph. This improves the visual effect of the night scene photograph.

[0013] In one possible implementation, the method may further include: inputting the fused target image into the preview path of the camera for noise reduction to obtain a denoised first preview image; performing tone mapping processing and post-preview processing on the denoised first preview image to obtain a first night scene preview image. This improves the consistency between the first night scene preview image and the night scene photographed image.

[0014] In one possible implementation, the method may further include: inputting a first target image into the preview path of a camera for noise reduction to obtain a noise-reduced second preview image; inputting the second target image into the preview path of the camera for noise reduction to obtain a noise-reduced third preview image; performing image fusion processing on the noise-reduced second and third preview images to obtain a fused preview image; and performing tone mapping processing and post-preview processing on the fused preview image to obtain a second night scene preview image. This improves the consistency between the second night scene preview image and the night scene photographed image.

[0015] In one possible implementation, the frame output mode can be a dual conversion gain mode; the first target image can be a low conversion gain image; and the second target image can be a high conversion gain image.

[0016] In one possible implementation, the training image for the first sensitivity can be a low conversion gain training image; the training image for the second sensitivity can be a high conversion gain training image and / or a lateral overflow set capacitance training image.

[0017] Secondly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is used to call and execute the computer program to implement the night scene photography method described in any of the first aspects above.

[0018] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when run by a processor of an electronic device, is used to implement the night scene photography method described in any one of the first aspects above.

[0019] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the night scene photography method as described in any one of the first aspects. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0021] Figure 2 A schematic diagram of an electronic device provided in an embodiment of this application;

[0022] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0023] Figure 4 A flowchart illustrating the night scene photography method provided in this application embodiment;

[0024] Figure 5 An example diagram of a 5x5 window in a high-sensitivity image provided in an embodiment of this application;

[0025] Figure 6 An example diagram of a 5x5 window in a low-sensitivity image provided in an embodiment of this application;

[0026] Figure 7 This is one of the schematic diagrams illustrating the overall implementation process of night scene photography provided in the embodiments of this application;

[0027] Figure 8 This is the second schematic diagram illustrating the overall implementation process of night scene photography provided in this application embodiment. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0031] To enable those skilled in the art to better understand the solution of this application, the application scenario of the technical solution of this application will be described first.

[0032] See Figure 1 The illustration shows a scenario diagram provided by an embodiment of this application.

[0033] This example scenario showcases a user's photo of a mobile advertisement displayed on a large screen, taken in a high dynamic range (HDR) night scene using a mobile phone or other terminal device. It can be seen that due to the influence of the large screen and other nighttime lighting, the image suffers from banding, resulting in regular alternating bright and dark stripes and a poor visual effect. Furthermore, in such complex lighting conditions, the limitations of exposure time and the need for de-banding lead to a lower signal-to-noise ratio, thus reducing the user's shooting experience.

[0034] To overcome the above technical problems, this application provides a night scene photography method. By utilizing a pre-installed camera on the terminal device and appropriately selecting a reasonable frame output method to avoid banding issues, and by performing HDR dual-frame image fusion followed by noise reduction, combined with a noise model calibration-fusing construction method, the method compensates for problems such as insufficient exposure time and increased noise caused by banding. This not only avoids regular alternating bright and dark stripes in the captured night scene photos, improving the visual effect of night scene photos, but also enhances the user's shooting experience.

[0035] It should be noted that the night scene photography method provided in this application embodiment can be applied to electronic devices (i.e., terminal devices) such as mobile phones, tablets, personal digital assistants (PDAs), desktop, laptop, and notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0036] To enable those skilled in the art to better understand the night scene photography method provided in this application, the hardware architecture and software system architecture of the electronic device for real-time night scene photography will be described in detail below.

[0037] See Figure 2 The diagram illustrates an electronic device provided in an embodiment of this application.

[0038] like Figure 2 As shown, the electronic device 200 may include a processor 210, a mobile communication module 220, a wireless communication module 230, a sensor module 240, a display screen 250, an internal memory 260, a camera 270, an audio module 280, a speaker 280A, a receiver 280B, a microphone 280C, a headphone jack 280D, an antenna group 1, and an antenna group 2.

[0039] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0040] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. For example, when capturing HDR night scene images using a camera 270 pre-installed on electronic device (terminal device) 200, noise reduction can be performed after HDR dual-frame image fusion, combined with a noise model calibration-fusing construction method. This can compensate for insufficient exposure time and increased noise caused by banding, thereby eliminating regular alternating bright and dark stripes in the night scene photo and improving its visual effect.

[0041] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0042] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0043] The sensor module 240 can be used to acquire data signals related to various aspects of the electronic device 200, serving as a basis for implementing corresponding functions. In some embodiments, the sensor module 240 may include, but is not limited to, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, temperature sensors, image sensors, etc.

[0044] The display screen 250 is used to display images, videos, etc., such as displaying captured night scene photos, for example. Figure 1 The terminal device (mobile phone or tablet computer, etc.) used by the user for night scene photography can display promotional photos taken on a large screen. The display screen 250 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or P displays 250, where P is a positive integer greater than 1.

[0045] Internal memory 260 can be used to store computer executable program code, including instructions. Internal memory 260 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound capture, image capture, etc.). The data storage area may store data created during the use of electronic device 200 (such as audio data, image data, etc.). Furthermore, internal memory 260 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in internal memory 260 and / or instructions stored in memory located within the processor.

[0046] In some embodiments, the internal memory 260 stores instructions for executing a night scene photography method. The processor 210 can execute the instructions stored in the internal memory 260 to specifically achieve the following functions: upon receiving a night scene photography instruction from a user to the electronic device 200, it can first respond to the instruction by flexibly determining the appropriate frame output mode and exposure time of the camera 270, taking into account the dynamic range requirements and banding anti-flicker requirements of the current high dynamic range night scene, i.e., determining the appropriate operating mode of the camera 270. Then, under this frame output mode and exposure time, it acquires a low-sensitivity exposure frame image and a high-sensitivity exposure frame image through the camera 270; wherein these two frames are two images of the same night scene captured by the camera 270. Next, image fusion processing is performed on the two frames to obtain the fused target image. The two frames are then used to fuse a pre-built noise model to obtain a fused noise model. The fused noise model can then be used to reduce noise in the fused target image. Tone mapping and night scene post-processing are then performed on the denoised target image to obtain a night scene photograph with better visual effects.

[0047] The camera 270 is used to capture still images or videos. For example, after a user holds the electronic device 200 in their hand, they can use the camera 270 installed on the electronic device 200 to capture images of various scenes at night and during the day, for example... Figure 1 When a user holds a mobile phone, they can use the phone's camera to take a photo of a mobile phone advertisement displayed on a large screen at night. In some embodiments, the electronic device 200 may include one or K cameras 270, where K is a positive integer greater than 1.

[0048] Electronic device 200 implements display functions through a GPU, display screen 250, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 250 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0049] Electronic device 200 can implement audio functions through audio module 280, speaker 280A, receiver 280B, microphone 280C, headphone jack 280D, and application processor, such as music playback and recording for voice input and output.

[0050] The audio module 280 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 280 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 280 may be located in the processor 210, or some functional modules of the audio module 280 may be located in the processor 210.

[0051] The speaker 280A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or make hands-free calls through the speaker 280A.

[0052] The receiver 280B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 200 answers a telephone call or voice message, the receiver 280B can be brought close to the listener's ear to hear the voice.

[0053] Microphone 280C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 280C, inputting the sound signal into microphone 280C. Electronic device 200 may have at least one microphone 280C. In some embodiments, electronic device 200 may have two microphones 280C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 200 may have three, four, or more microphones 280C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0054] The headphone jack 280D is used to connect wired headphones and does not restrict the standard attributes of the jack.

[0055] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200.

[0056] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 220, wireless communication module 230, modem processor, and baseband processor.

[0057] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0058] The mobile communication module 220 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 200. The mobile communication module 220 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 220 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 220 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 220 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 220 and at least some modules of the processor 210 may be housed in the same device.

[0059] The wireless communication module 230 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 230 can be one or more devices integrating at least one communication processing module. The wireless communication module 230 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 230 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0060] In addition, the electronic device 200 runs an operating system on top of the aforementioned components. Examples include iOS, Android, and Windows operating systems. Applications can be installed and run on this operating system.

[0061] See Figure 3 It shows a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0062] The software system of electronic device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 200.

[0063] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0064] The application layer can include a series of application packages (APPs). For example... Figure 3 As shown, the application package may include applications such as camera, call, navigation, WLAN, Bluetooth, and gallery. This is used when the user is holding the electronic device 200 and taking photos (e.g., taking pictures). Figure 1 When viewing night scene images, the camera application can communicate with camera-related devices in the camera access interface of the frame layer to request camera functions and obtain image data, etc.

[0065] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 3 As shown, the application framework layer can include a window manager, notification manager, resource manager, and camera access interface (including but not limited to camera management, camera devices, etc.). The application framework layer can be used to implement interaction between camera services and camera APIs. That is, it provides a unified interface, enabling different camera hardware to interact with different camera applications. The framework layer also handles many common aspects of camera functionality, such as autofocus.

[0066] The window manager is used to manage window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture screenshots, among other things.

[0067] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0068] In this embodiment, the Hardware Abstraction Layer (HAL) provides a set of standard interfaces, enabling the framework layer to communicate with camera hardware from various manufacturers without needing to understand the underlying hardware details. The HAL stores the hardware abstraction layer itself and camera algorithm libraries, etc. It should be noted that the night scene photography algorithm provided in this application can be stored in the camera algorithm library of the HAL layer, such as... Figure 3 As shown.

[0069] The night scene photography algorithm utilizes a method of first fusing HDR dual-frame images and then performing noise reduction, combined with a noise model that is calibrated before fusing, to eliminate banding and other issues in night scene photos, thereby improving their visual quality. Specifically, when a user issues a night scene shooting command to the terminal device, the appropriate frame output mode and exposure time for the camera 270 are first determined in response to the command, i.e., the appropriate working mode of the camera 270 is determined. Then, the camera 270 acquires low-sensitivity and high-sensitivity exposure frame images of the same night scene in an HDR night scene. Next, image fusion processing is performed on these dual-frame images to obtain a fused target image. These dual-frame images are then used to fuse pre-constructed noise models corresponding to images with different sensitivities, resulting in a fused noise model. This fused noise model is then used to reduce noise in the fused target image, followed by HDRTone mapping and post-processing (such as color retouching) to obtain a night scene photo with better visual effects.

[0070] The hardware layer may include the hardware components of the aforementioned electronic device. For example, Figure 3 The exhibits included cameras, image signal processors, digital signal processors, and graphics processors.

[0071] The camera is used for focusing and exposure processing when capturing images.

[0072] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 200 selects a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0073] In this way, through the application layer, framework layer, HAL layer, driver layer and hardware layer, interoperability between different camera applications and hardware devices on the Android platform can be achieved, which can eliminate problems such as banding in night scene photos, thereby improving the visual effect of night scene photos and the user's photography experience.

[0074] The technical solutions involved in the following embodiments can all be implemented in electronic devices with the above-described hardware and software architectures.

[0075] Next, the specific implementation process of the night scene photography method provided in this application will be described in detail:

[0076] like Figure 4 As shown, the specific implementation process of this night scene photography method may include the following steps S401-S405:

[0077] S401: In response to the user's shooting command to the terminal device, determine the camera's frame output mode and exposure time.

[0078] In this embodiment, when a user uses a mobile phone or other terminal device to view an HDR night scene (such as...) Figure 1 When shooting a mobile phone promotional video or image on a large screen at night (as shown), in response to the user's shooting operation triggered by the terminal device, such as the user's click on the "camera" icon on the mobile phone display screen, the terminal device will immediately combine the dynamic range requirements and banding anti-flicker requirements of the current HDR night scene to flexibly determine the reasonable frame output mode and exposure time of the camera, that is, determine the reasonable working mode of the camera, and launch the camera APP under the frame output mode and exposure time to execute the subsequent step S402.

[0079] It should be noted that this embodiment does not limit the specific content of the camera's frame output mode. A more reasonable frame output mode can be selected based on actual conditions and empirical values. One optional implementation is that the frame output mode can be, but is not limited to, bracketing exposure, dual conversion gain (DCG), or lateral overflow integration capacitor (Lofic) mode, etc.

[0080] Among them, the bracketing exposure method refers to using multiple exposures, each with a different exposure time and gain, to fully capture the dynamic range of the scene.

[0081] DCG mode refers to a method of acquiring low dual conversion gain (LCG) image frames and high dual conversion gain (HCG) image frames through a single exposure and multiple readouts, using dual conversion gain to fully capture the dynamic range of the scene.

[0082] Lofic mode refers to using additional capacitors to increase the dynamic range of photosensitive pixels. This mode can output frames using Lofic frames, LCG+Lofic mode, or HCG+Lofic mode to fully capture the dynamic range of the scene.

[0083] Specifically, the selection of frame output mode can be based on dynamic range statistics and banding sensor data to determine the scene's dynamic range and banding status. For example, in low to medium dynamic range scenes where banding is not detected, constant exposure, bracketing, or DCG methods can be used for frame output; in low to medium dynamic range scenes where banding is detected, stagger methods can be used; in medium to high dynamic range scenes where banding is detected, stagger methods can be used; and in high and ultra-high dynamic range scenes (such as sunlight, sunrise, and sunset), Lofic methods can be used. In practical applications, these methods can be flexibly adjusted according to the requirements of the terminal device system.

[0084] Regarding the choice of exposure time, for night scene photography, it is generally desirable to obtain a longer exposure time and use the smallest possible analog gain and digital gain to enhance brightness. In addition, urban night scene scenes are often affected by large screens and other various night scene lighting sources. For example, when there are a large proportion of large screens or other light sources in the field of view, it is necessary to use de-banding logic to synchronize the exposure time with the frequency of the light source, which is usually 50Hz or 60Hz. In other words, the choice of night scene exposure time can be synchronized with the frequency of other light sources or the refresh rate of large screens. For example, assuming the illumination frequency of the light source is 50Hz, the exposure time can be chosen as an integer multiple of 1 / 100s (i.e., half a cycle). That is, the minimum exposure time should not be less than 1 / 100s, such as 1 / 50s or 1 / 25s.

[0085] S402: In frame output mode and exposure time, acquire a first target image and a second target image through the camera; wherein, the first target image and the second target image are two frames of the same night scene captured by the camera; the first sensitivity of the first target image is lower than the second sensitivity of the second target image.

[0086] After determining the reasonable frame output mode and exposure time of the camera in step S401, the terminal device can further capture images of the same night scene through the camera under the reasonable frame output mode and exposure time, and take any two consecutive (or non-consecutive) high-sensitivity exposure frames captured by the terminal device as target images to be processed, and define them as the first target image and the second target image respectively, for use in executing subsequent step S402.

[0087] In this design, the first sensitivity of the first target image is lower than the second sensitivity of the second target image; that is, the first target image is a low-sensitivity image, and the second target image is a high-sensitivity image. The specific sensitivity value is not limited. One possible implementation is that, when the frame output mode is Dual Conversion Gain (DCG), the first target image can be a low conversion gain (LCG) image, and the second target image can be a high conversion gain (HCG) image.

[0088] It should be noted that this embodiment does not limit the types of the first target image and the second target image. For example, the first target image and the second target image can be color images composed of the three primary colors of red (R), green (G), and blue (B), or they can be grayscale images, etc.

[0089] Furthermore, it should be noted that when acquiring the first target image and the second target image, these two HDR dual-frame images can follow the same exposure time constraint to avoid stripes; or, only one frame image (i.e., the first target image or the second target image) can be constrained to meet the above-mentioned exposure time constraint. For the other frame image that does not meet the above-mentioned exposure time constraint, an additional supplementary algorithm can be used to compensate for the frame image with stripes, and then the subsequent step S403 is executed to achieve image fusion.

[0090] S403: Perform image fusion processing on the first target image and the second target image to obtain the fused target image.

[0091] After the terminal device acquires the first target image and the second target image with different light sensitivity through the camera in step S402, it can further use existing or future image fusion algorithms, such as HDR linear fusion algorithm, to perform image detail information fusion processing on the first target image and the second target image to obtain the fused target image, which is then used to execute the subsequent step S405 to achieve noise reduction processing on the fused target image.

[0092] Specifically, an optional implementation manner is that the corresponding pixels in the first target image and the second target image may be fused according to a preset fusion coding threshold (the specific value is not limited), the fusion weight corresponding to the first target image (i.e., the fusion weight corresponding to the first sensitivity image), and the fusion weight corresponding to the second target image (i.e., the fusion weight corresponding to the second sensitivity image), so as to obtain a fused target image, thereby improving the fusion accuracy of the target image. The manner for determining the fusion weight corresponding to the first target image (i.e., the fusion weight corresponding to the first sensitivity image) and the fusion weight corresponding to the second target image (i.e., the fusion weight corresponding to the second sensitivity image) is as follows:

[0093] First, a linear fusion manner (such as an HDR linear fusion algorithm) may be adopted to fuse a low-sensitivity raw image (which can be understood as the first sensitivity image) and a high-sensitivity raw image (which can be understood as the second sensitivity image), and the consistency of the Bayer four-channel color ratio is considered (the purpose is not to destroy the color ratio of the original image). The fusion is mainly performed using the brightness ratio (Exposure Ratio) of the low-sensitivity image and the high-sensitivity image. In order to perform fusion within the selected brightness region and ensure the consistency of the Color Ratio of each channel, a calculation manner of a small region (such as 5*5) needs to be adopted, and the value of the brightest channel in the region (which can be expressed as Max_Value_In_Area) is selected.

[0094] For example, taking fusing a low conversion gain LCG image and a high conversion gain HCG image in the gray scale range of 512DN to 1023DN as an example, the camera of the terminal device will determine whether to enter the fusion (combine) processing step according to whether the brightest channel value among the four channels R / Gr / Gb / B is within 512DN to 1023DN. If 512DN < R < 1023DN in a certain region, and B, Gr, and Gb are all not greater than 512DN, then all four channels will be combined, and the combine weights of the four channels are the same, where the specific value of the weight is not limited, and can be set according to actual conditions and empirical values, for example, a value of 0.5 can be taken.

[0095] To ensure the color ratio of the merged image is not disrupted, the combine weight can be the same even though the densities (DN) of B / Gb / Gr / R channels are different. That is, when the DN is the same, different combine weights are used for the B / Gb / Gr / R channels. For example, if the G channel is 600 DN, HCG will be used more than LCG*Ratio; but if the R channel is 600 DN, it means the channel might be 900 DN. In this case, LCG*Ratio will be used more than HCG. Therefore, when ISP is OFF and the x-axis is DN to view noise, noise separation will be seen between 512 DN and 1023 DN.

[0096] For example, taking the current center pixel of a Raw image as the R channel, within a small M*M region (5*5 in this example), such as... Figure 5 The 5x5 window in the high-sensitivity image shown, and as... Figure 6 The image shown is a 5x5 window in a low-sensitivity image. To calculate whether the current pixel is blended and the blending ratio, it is necessary to consider the information of each channel within the window; in such a case... Figure 5 (or Figure 6 Within the current 5*5 window range shown, the maximum value among the four channels is obtained and denoted as Max_Value_In_Area. Similarly, the maximum channel value image of the neighborhood of each pixel position in the entire image can be obtained and denoted as Max_Value_In_Area_Img.

[0097] Based on this, the formula for calculating the fusion weight of the pixel at coordinates (i,j) in the image can be determined as follows:

[0098] Low_Sensitive_Frame_weight = 1 if Max_Value_In_Area_Img(i,j) >Fusion_Code_Max (1)

[0099] Among them, Low_Sensitive_Frame_weight represents the fusion weight corresponding to the low-sensitivity image (such as the first target image), and the typical value can be 1; Fusion_Code_Max represents the preset maximum fusion threshold, and the specific value is not limited. It can be set according to the actual situation and experience, such as setting Fusion_Code_Max to 1023DN, etc.

[0100] High_Sensitive_Frame_Weight= 1 if Max_Value_In_Area_Img(i,j) <Fusion_Code_Min (2)

[0101] Among them, High_Sensitive_Frame_Weight represents the fusion weight corresponding to the high-sensitivity image (such as the second target image), and the typical value can be 1; Fusion_Code_Min represents the preset minimum fusion threshold, and the specific value is not limited. It can be set according to the actual situation and experience, such as setting Fusion_Code_Min to 512DN, etc.

[0102] Alternatively, when both of the following conditions are met:

[0103] Condition 1: Max_Value_In_Area_Img(i,j)>Fusion_Code_Min

[0104] Condition 2: Max_Value_In_Area_Img(i,j) <Fusion_Code_Max

[0105] The values ​​of Low_Sensitive_Frame_weight and High_Sensitive_Frame_weight are shown in the following formulas:

[0106] Low_Sensitive_Frame_weight=Weight_LUT(Max_Value_In_Area_Img(i,j)) (3)

[0107] High_Sensitive_Frame_weight=1-Low_Sensitive_Frame_weight (4)

[0108] Among them, Weight_LUT represents a predefined lookup table for fusion weights (the specific content is not limited and can be set according to actual conditions and experience).

[0109] Thus, after completing the above image fusion process, we can obtain the fusion weight information map (which can be represented as Low_Sensitive_Frame_weight_Img) corresponding to the low-sensitivity image and the fusion weight information map (which can be represented as High_Sensitive_Frame_Weight_Img) corresponding to the high-sensitivity image.

[0110] Based on this, when performing the fusion processing of the first target image (i.e., the low-sensitivity image) and the second target image (i.e., the high-sensitivity image), the maximum value of the four channels R / Gr / Gb / B in the M*M (e.g., 5*5) neighborhood of the pixel to be fused in the image can be obtained first. Then, it is determined whether the value falls within the preset fusion range. The value of this range is not limited. For example, it can be a range [Fusion_Code_Min, Fusion_Code_Max] composed of the preset minimum fusion threshold Fusion_Code_Min and the preset maximum fusion threshold Fusion_Code_Max, such as 512DN~1023DN.

[0111] Next, if the value is determined to fall within a preset fusion range, data below Fusion_Code_Min can use information from the high-sensitivity image, and data above Fusion_Code_Max can use information from the low-sensitivity image. For data falling within the preset fusion range, the following formula can be used for fusion:

[0112] Imag_Fusion_Output=Low_Sensitive_Frame*Low_Sensitive_Frame_weight+

[0113] High_Sensitive_Frame*(1-Low_Sensitive_Frame_weight)(5)

[0114] Wherein, Imag_Fusion_Output represents the fused target image; Low_Sensitive_Frame represents the first target image (i.e., the low-sensitivity image); Low_Sensitive_Frame_weight represents the fusion weight corresponding to the first target image (i.e., the low-sensitivity image), and the specific value is determined by the above formulas (1) and (3); High_Sensitive_Frame represents the second target image (i.e., the high-sensitivity image); High_Sensitive_Frame_Weight represents the fusion weight corresponding to the second target image (i.e., the high-sensitivity image), and the specific value is determined by the above formulas (2) and (4).

[0115] S404: The pre-constructed noise model is fused using the first target image and the second target image to obtain the fused noise model.

[0116] It should be noted that, due to the inconsistency of noise models corresponding to images with different sensitivities, it is not possible to perform noise reduction after fusing high-sensitivity and low-sensitivity images. In other words, it is not possible to perform image fusion processing on the first and second target images before performing noise reduction in the subsequent step S405. Furthermore, selecting two frames with different sensitivities for separate noise reduction and then fusion is limited by the inconsistent processing methods of the camera's preview and capture paths, resulting in a lack of consistency between preview and capture, thus reducing the user's shooting experience in night scenes.

[0117] Therefore, in order to achieve the fusion and subsequent noise reduction of high-sensitivity and low-sensitivity images, this application utilizes a method of pre-calibrating the noise of images with different sensitivities to construct noise models corresponding to the high-sensitivity and low-sensitivity images respectively. Then, the first target image and the second target image obtained in step S402 are used to fuse and model the pre-constructed noise models to obtain a fused noise model, which is used to execute the subsequent step S405 to achieve noise reduction of the fused target image. The specific calibration process is shown in steps AB below:

[0118] Step A: Under the N preset working modes of the camera, acquire N training image datasets respectively; wherein, the N training image datasets include training images of the first sensitivity and training images of the second sensitivity; and N is a positive integer greater than 1.

[0119] Step B: Perform noise calibration on each of the N training image datasets, and during the calibration process, fit the noise model corresponding to each of the N training image datasets.

[0120] In this implementation, to pre-calibrate noise for images with different sensitivities, data is first collected under N preset working modes of the camera (such as different frame output methods like LCG, HCG, and Lofic) to obtain N training image datasets. These N training image datasets include training images with the first sensitivity and training images with the second sensitivity. Then, noise calibration is performed on the training image datasets corresponding to a single working mode to obtain the noise models corresponding to the images with different sensitivities contained in the training image datasets under each working mode. These can be represented as Noise_Model1, Noise_Model2, ..., Noise_ModelN, etc.

[0121] Specifically, firstly, N preset working modes of the camera can be selected (such as different frame output methods like LCG, HCG, and Lofic). Then, in each working mode, such as LCG, corresponding image cards are acquired by traversing all supported gains to obtain at least one training image dataset (represented as Dataset_LCG). The training images containing the first sensitivity are low conversion gain (LCG) training images. Similarly, this can be extended to other modes such as HCG and Lofic to obtain corresponding training image datasets, which can be represented as Dataset_HCG, Dataset_Lofic, etc. The training image dataset Dataset_HCG contains high conversion gain (HCG) training images for the second sensitivity, and the training image dataset Dataset_Lofic contains lateral overflow capacitance (Lofic) training images for the second sensitivity.

[0122] Next, we can obtain the image information of each training image in the h-th training image dataset (the specific value is not limited, as long as h is a positive integer greater than 0 and not greater than N) of the N training image datasets. That is, we need to collect the image information of each color block on the test card corresponding to the h-th training image dataset. Then, using this color block image information, we can calculate the mean and variance information corresponding to the h-th training image dataset. In this way, we can use this mean and variance information to fit the information of different simulated gains in the h-th training image dataset to obtain a series of relationship curves between the mean and variance information.

[0123] Furthermore, by fitting the relationship between the coefficients of the curve representing these mean and variance information and the simulated gain, the noise model corresponding to the i-th training image dataset can be obtained. Similarly, the noise models corresponding to each of these N training image datasets can be fitted. Examples include Noise_Model_LCG, Noise_Model_HCG, and Noise_Model_Lofic.

[0124] It should be noted that this embodiment does not limit the specific expression of the noise model (such as Noise_Model_LCG, Noise_Model_HCG, Noise_Model_Lofic, etc.) corresponding to images with different sensitivities in various training image datasets under various working modes. It can be set according to the actual situation and empirical values. For example, the pre-built noise model can be a curve in a Cartesian coordinate system where the horizontal axis represents pixel values ​​and the vertical axis represents noise levels.

[0125] This can effectively improve the accuracy of constructing each noise model, thereby improving the accuracy of subsequent noise model fusion modeling.

[0126] It should be noted that when the pre-built noise model is a function that constructs the relationship between the ISO value of the current frame image, the current pixel value, and the noise level of the current pixel, the noise level of the current pixel can be obtained by substituting the ISO value of the current frame and the current pixel value into the noise model. That is, the noise level of the current pixel = noise model(current pixel value, ISO value of the current frame image). For example, when the ISO value of the input current frame image is 100 and the current pixel value is 128, substituting them into the noise model, the noise level of the current pixel can be calculated to be 5.

[0127] Based on this, an optional implementation method is as follows: the implementation process of step S404, "using the first target image and the second target image to fuse the pre-constructed noise model to obtain the fused noise model," can be as follows: First, determine the noise level corresponding to the s-th pixel in the first target image based on the noise model corresponding to the pre-constructed first photosensitivity image, and determine the noise level corresponding to the s-th pixel in the second target image based on the noise model corresponding to the pre-constructed second photosensitivity image, where s is a positive integer greater than 0. Then, calculate the first product of the noise level corresponding to the s-th pixel in the first target image and the fusion weight corresponding to the first target image; and calculate the second product of the noise level corresponding to the s-th pixel in the second target image and the fusion weight corresponding to the second target image. Next, calculate the sum of the first product and the second product as the pixel noise level corresponding to the s-th pixel output by the fused noise model. Taking the first target image as a low conversion gain LCG image and the second target image as a high conversion gain HCG image as an example, the specific calculation formula for the fused noise level of the s-th pixel output by the fused noise model is as follows:

[0128] Noise_Level_Fusion_Output

[0129] =Noise_Model_LCG(Low_Sensitive_Frame(i,j))*Low_Sensitive_Frame_weight+

[0130] Noise_Model_HCG(High_Sensitive_Frame(i,j))*High_Sensitive_Frame_weight(6)

[0131] Wherein, Noise_Level_Fusion_Output represents the fusion noise level of the s-th pixel at coordinate (i,j) output by the fused noise model; Low_Sensitive_Frame(i,j) represents the pixel value of the s-th pixel at coordinate (i,j) of the first target image (i.e., the low conversion gain LCG image); High_Sensitive_Frame(i,j) represents the pixel value of the s-th pixel at coordinate (i,j) of the second target image (i.e., the high conversion gain HCG image); Noise_Model_LCG represents the noise model corresponding to the pre-constructed LCG image; Noise_Model_HCG represents the noise model corresponding to the pre-constructed HCG image; Low_Sensitive_Frame_weight represents the fusion weight corresponding to the first target image, and its specific value is determined by the above formulas (1) and (3); High_Sensitive_Frame_Weight represents the fusion weight corresponding to the second target image, and its specific value is determined by the above formulas (2) and (4).

[0132] By following this logic, the fused noise level of each pixel output by the fused noise model can be obtained, thus achieving the fusion of the pre-built noise model.

[0133] It should be noted that during the model fusion process, the Max_Value_In_Area_Img or its 2x downsampled information can be saved according to the requirements of the fused noise model, so as to obtain the corresponding noise information during the subsequent noise reduction process of step S405.

[0134] In addition, an alternative implementation is to save the fusion weight information map (which can be represented as Low_Sensitive_Frame_weight_Img) corresponding to the low-sensitivity image and the fusion weight information map (which can be represented as High_Sensitive_Frame_Weight_Img) corresponding to the high-sensitivity image, or information downsampled by twice, so as to obtain the corresponding noise information during the noise reduction process in the subsequent step S405.

[0135] S405: The fused target image is denoised using the fused noise model, and then tone mapping and night scene post-processing are performed on the denoised target image to obtain a night scene photograph.

[0136] After obtaining the fused target image in step S403 and the fused noise model in step S404, the terminal device can further reduce the noise of the fused target image by using the fused noise level of each pixel output by the fused noise model, thus obtaining a denoised target image. Then, a preset tone mapping processing method (such as HDR Tone Mapping) can be used to perform global contrast and local contrast processing on the denoised target image, resulting in a contrast-processed target image. Finally, after performing night scene post-processing (such as color retouching) on ​​the contrast-processed target image, a night scene image with better visual effects can be obtained.

[0137] It should be noted that this application does not limit the specific method of denoising the fused target image; either traditional methods or artificial intelligence (AI) methods can be used for denoising.

[0138] Traditional noise reduction methods refer to the use of spatial domain noise reduction algorithms such as filtering and block matching, or frequency domain filtering algorithms such as wavelet or Fourier transform. In the noise reduction process, special attention needs to be paid to the use of noise level information, and the fused noise information is used to guide the noise reduction process. Taking the block matching nonlocal-means noise reduction method as an example, if the actual calculation position is a pixel at coordinates (i,j), first calculate the difference (represented as Diff) between the neighborhood of the current pixel position and other positions at coordinates (p,q). Then compare this difference Diff with the fusion noise level (i.e., Noise_Level_Fusion_Output(i,j)). If Diff is greater than the fusion noise level (i.e., Noise_Level_Fusion_Output(i,j)), it is considered that the matching degree between the current position at coordinates (i,j) and the position at coordinates (p,q) is poor, and the fusion intensity of these two positions needs to be reduced. Conversely, if Diff is less than the fusion noise level (i.e., Noise_Level_Fusion_Output(i,j)), it is considered that the matching degree between the current position at coordinates (i,j) and the position at coordinates (p,q) is relatively close, and the difference in matching is the noise signal, so the fusion intensity of these two positions needs to be increased to eliminate the noise.

[0139] AI-based noise reduction refers to using a network based on UNet or other types of networks. The fused target image and the corresponding fused noise level image are input into the AI ​​network for processing. The network is then trained to perform noise reduction by referring to the fused noise level information.

[0140] Furthermore, after denoising is completed using traditional or AI methods and the denoised target image (which can be represented as Imag_Fusion_NR_Output) is obtained, global contrast and local contrast can be processed by using tone mapping based on traditional or AI methods.

[0141] The global contrast ratio is primarily handled using the Global_Tone_LUT method, as shown in the following formula:

[0142] Imag_Fusion_NR_Global_Tone_Output(i,j)=

[0143] Global_Tone_LUT(Imag_Fusion_NR_Tone_Output(i,j))(7)

[0144] Local contrast processing can employ traditional methods or AI-based approaches like HDRNet. This involves dividing the denoised target image into M*N grids and statistically analyzing the brightness information of each grid to obtain M*N dimensional local brightness statistics. Based on this, a local mapping curve (represented as Local_Tone_Curves) is obtained for each grid using either traditional or AI-based methods. Then, Local_Tone_Curves is applied to each grid image for mapping, and Alpha Blending is used for smoothing between grids to prevent brightness abrupt changes at grid edges. After these processing steps, the contrast-processed target image (represented as Imag_Fusion_NR_Global_Local_Tone_Output) is obtained.

[0145] Then, the contrast-processed target image (Imag_Fusion_NR_Global_Local_Tone_Output) is further processed by existing or future night scene post-processing methods, such as color retouching, to generate a night scene photograph with better visual effects, free from banding issues and with a better signal-to-noise ratio, which can be represented as Imag_Fusion_NR_Global_Local_Tone_Final_Output.

[0146] Thus, when shooting HDR night scene photos, this application eliminates banding and other issues in night scene photos by performing HDR dual-frame image fusion followed by noise reduction, combined with a noise model calibration-fusing construction method. This improves the visual effect of night scene photos and enhances the user's photography experience.

[0147] Furthermore, it should be noted that due to the inconsistency of noise models corresponding to images with different sensitivities, it is not possible to perform noise reduction after fusing high-sensitivity and low-sensitivity images. On the other hand, selecting two images with different sensitivities for noise reduction and then fusing them separately is limited by the inconsistency between the camera's preview and shooting paths, which makes it impossible to guarantee the consistency between preview and shooting. These issues will reduce the user's shooting experience for night scene photography and preview.

[0148] To overcome the above technical problems, this application, by performing the night scene photography method of steps S401-S405 above, first performs HDR dual-frame image fusion and then noise reduction, and combines a method of constructing a noise model corresponding to high and low sensitivity images by first calibrating and then fusing, eliminates problems such as banding in night scene photos. In order to better solve the problems of night scene noise reduction and high dynamic range fusion, this application also sends the processed data to the preview processing stream for processing. In this way, the user can also see information consistent with the noise and dynamic range of the photo on the preview side, thereby improving the photo effect while ensuring the consistency between the preview and the photo, and further enhancing the user's shooting experience.

[0149] Specifically, on the preview path side, in order to ensure a consistent experience between previewing and taking photos, that is, to keep the brightness and noise levels as consistent as possible with the photos, one possible implementation is that after the terminal device obtains the fused target image through step S403, it can input the fused target image into the preview path of the camera for noise reduction to obtain a noise-reduced preview image (here defined as the first preview image); then, the noise-reduced first preview image is subjected to tone mapping processing and preview post-processing to obtain the first night scene preview image.

[0150] In this implementation, a processing method similar to that of the image capture path can be adopted. The fused target image is used as the input of the camera preview path, and the lightweight noise reduction and tone mapping processing modules on the preview path are used to achieve brightness and noise processing effects similar to those of the image capture path. In this way, after the preview frame data processed by the preview path is processed and then sent for display after some preview post-processing, the consistency between the obtained first night scene preview image and the night scene captured image can be higher.

[0151] Another optional implementation is that after the terminal device acquires the low-sensitivity first target image and the high-sensitivity second target image in step S402, it can first input the first target image into the camera's preview path for noise reduction to obtain a noise-reduced preview image (here defined as the second preview image), and then input the second target image into the camera's preview path for noise reduction to obtain a noise-reduced preview image (here defined as the third preview image). Then, it performs image fusion processing on the noise-reduced second and third preview images to obtain a fused preview image. The fusion method can be implemented as described in step S403 above, and the specific process will not be elaborated further. Next, it can perform tone mapping processing and post-preview processing on the fused preview image to obtain the second night scene preview image.

[0152] In this implementation, noise reduction is performed separately for the first target image with low sensitivity and the second target image with high sensitivity. After fusion in the preview path, tone mapping is performed to achieve brightness and noise reduction effects that are closer to those in the shooting path. In this way, after some post-preview processing, the preview frame data processed in the preview path is sent for display, which can improve the consistency between the obtained second night scene preview image and the night scene photographed image.

[0153] In this way, when users use their mobile phones or other terminal devices to view HDR night scene scenes (such as...) Figure 1 When taking a photo of a mobile phone promotional video or image on a large screen at night (as shown), the terminal device can eliminate problems such as banding in night scene photos by executing the above steps S401-S405 and subsequent preview processing steps. Figure 1 The night scene photo shown has regular alternating light and dark stripes, which can improve the photo effect while ensuring the consistency between the preview and the photo, thereby improving the user's shooting experience.

[0154] Specifically, one implementation method is that the overall process of night scene photography can be as follows: Figure 7As shown: Upon receiving a night scene shooting command from a user, the terminal device first responds to the command, flexibly determining the appropriate frame output mode and exposure time for the camera based on the dynamic range requirements and banding anti-flicker requirements of the current high dynamic range night scene. Then, under this frame output mode and exposure time, the camera acquires a first target image with low sensitivity and a second target image with high sensitivity. Next, image fusion processing is performed on these two frames to obtain the fused target image. Simultaneously, these two frames can be used to fuse a pre-built noise model (which can be obtained by fitting the noise model corresponding to each of the N training image datasets with different sensitivities after noise calibration) to obtain the fused noise model. Furthermore, on the one hand, the fused noise model can be used to perform noise reduction and tone mapping processing, as well as post-processing of night scenes, on the fused target image using traditional or AI methods to obtain night scene photos with better visual effects; on the other hand, the fused target image can be input into the camera's preview path for noise reduction and tone mapping processing, as well as preview post-processing, to achieve brightness and noise processing effects similar to those of the photo capture path, resulting in a first night scene preview image with higher consistency with the night scene photo image.

[0155] Another approach is that the overall process of night scene photography described above can be as follows: Figure 8 As shown: Upon receiving a night scene shooting command from a user, the terminal device first responds to the command, flexibly determining the appropriate frame output mode and exposure time for the camera based on the dynamic range requirements and banding anti-flicker requirements of the current high dynamic range night scene. Then, under this frame output mode and exposure time, the camera acquires a first target image with low sensitivity and a second target image with high sensitivity. Next, image fusion processing is performed on these two frames to obtain a fused target image. Simultaneously, these two frames are used to fuse a pre-built noise model (which can be obtained by calibrating the noise of N training image datasets at different sensitivities and fitting the corresponding noise models for each of the N training image datasets) to obtain a fused noise model. The fused noise model is then used to perform traditional or AI-based noise reduction, tone mapping, and night scene post-processing on the fused target image to obtain a night scene photo with better visual effects. On the other hand, the first target image and the second target image can be input into the preview path of the camera for noise reduction to obtain the second and third preview images after noise reduction. Then, the two are fused together, and the fused target image is subjected to tone mapping and post-preview processing to achieve brightness and noise reduction effects similar to those of the shooting path, resulting in a second night scene preview image with high consistency with the night scene shooting image.

[0156] This can eliminate problems such as banding in night scene photos (e.g., eliminating...). Figure 1 The regular alternating light and dark stripes in the night scene photo improve the photo quality and ensure consistency between the preview and the photo, thereby enhancing the user's shooting experience.

[0157] Furthermore, this application also provides an electronic device (i.e., a terminal device). For details regarding the hardware structure and software framework of the electronic device, please refer to [link to relevant documentation]. Figure 2 and Figure 3 The corresponding explanation is as follows: The electronic device includes a memory and a processor. The memory stores a computer program, and the processor calls and executes the computer program to implement the night scene photography method provided in the above description.

[0158] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by the processor of a terminal device, is used to implement the night scene photography method described above.

[0159] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for taking night scene photos, characterized in that, Applied to a terminal device equipped with a camera, the method includes: In response to the user's shooting command to the terminal device, and considering the dynamic range requirements and banding anti-flicker requirements of the current HDR night scene, the camera's frame output mode and exposure time are determined. The selection of the frame output mode specifically includes: in low to medium dynamic range scenes, when no banding is detected, using a constant exposure mode / bracketing mode / DCG mode for frame output; in low to medium dynamic range scenes, when banding is detected, using a stagger mode for frame output; in medium to high dynamic range scenes, when banding is detected, using a stagger mode for frame output; and in high dynamic range and ultra-high dynamic range scenes, using a Lofic mode for frame output. The exposure time is selected in sync with the frequency of the light source. Under the specified frame output mode and exposure time, a first target image and a second target image are acquired by the camera; the first target image and the second target image are two frames of the same night scene captured by the camera; the first sensitivity of the first target image is lower than the second sensitivity of the second target image. Based on the preset fusion coding threshold, the fusion weight determined by the maximum brightness values ​​of the four channels R / Gr / Gb / B in the pixel to be fused and its surrounding neighborhood in the first target image and the second target image, the corresponding pixels in the first target image and the second target image are fused to obtain the fused target image. The pre-constructed noise model is fused using the first target image and the second target image to obtain the fused noise model; wherein, the same fusion weights are used for the fusion processing of the noise model and the fusion processing of corresponding pixels in the first target image and the second target image. The fused target image is denoised using the fused noise model, and then tone mapping and night scene post-processing are performed on the denoised target image to obtain a night scene photograph. The step of fusing the pre-constructed noise model using the first target image and the second target image to obtain the fused noise model includes: The noise level corresponding to the s-th pixel in the first target image with the first sensitivity is determined according to a pre-constructed noise model; where s is a positive integer greater than 0. The noise level corresponding to the s-th pixel in the second target image with the second sensitivity is determined based on a pre-built noise model. Calculate the first product of the noise level corresponding to the s-th pixel in the first target image and the fusion weight corresponding to the first target image; and calculate the second product of the noise level corresponding to the s-th pixel in the second target image and the fusion weight corresponding to the second target image; Calculate the sum of the first and second products, which is used as the pixel noise level corresponding to the s-th pixel in the output of the fused noise model. Repeat this process to obtain the fused noise level of each pixel in the output of the fused noise model.

2. The method according to claim 1, characterized in that, The frame output mode is either bracketing exposure mode, dual conversion gain mode, or lateral overflow collection capacitance mode.

3. The method according to claim 1, characterized in that, The method further includes: Under the N preset working modes of the camera, N training image datasets are acquired respectively; the N training image datasets include training images at the first sensitivity and training images at the second sensitivity; N is a positive integer greater than 1; Noise calibration is performed on each of the N training image datasets, and during the calibration process, a noise model corresponding to each of the N training image datasets is fitted.

4. The method according to claim 3, characterized in that, The step of performing noise calibration on the N training image datasets respectively, and fitting a noise model corresponding to each of the N training image datasets during the calibration process, includes: Obtain the image information of each training image in the h-th training image dataset from the N training image datasets; where h is a positive integer greater than 0 and not greater than N; Using the image information of each training image in the h-th training image dataset, calculate the mean and variance information corresponding to the h-th training image dataset; The mean and variance information are fitted onto the information of different simulated gains in the h-th training image dataset to obtain the relationship curve between the mean and variance information; By fitting the relationship between the coefficients of the mean and variance information curve and the simulated gain, the noise model corresponding to the h-th training image dataset is obtained. Similarly, the noise models corresponding to the N training image datasets are fitted.

5. The method according to claim 1, characterized in that, The process involves using a fused noise model to denoise the fused target image, and then performing tone mapping and night scene post-processing on the denoised target image to obtain a night scene photograph, including: The fused target image is denoised using the fused noise level of each pixel output by the fused noise model to obtain the denoised target image. The target image after noise reduction is processed by global and local contrast using a preset tone mapping method to obtain a contrast-processed target image. The contrast-processed target image is then subjected to night scene post-processing to generate a night scene photograph.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The fused target image is input into the preview path of the camera for noise reduction to obtain the first preview image after noise reduction; The first preview image after noise reduction is subjected to tone mapping and post-preview processing to obtain the first night scene preview image.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: The first target image is input into the preview path of the camera for noise reduction to obtain a noise-reduced second preview image; The second target image is input into the preview path of the camera for noise reduction to obtain a noise-reduced third preview image; The denoised second preview image and the denoised third preview image are subjected to image fusion processing to obtain the fused preview image; The fused preview image is subjected to tone mapping and post-preview processing to obtain a second night scene preview image.

8. The method according to claim 1, characterized in that, The frame output mode is a dual conversion gain mode; the first target image is a low conversion gain image; the second target image is a high conversion gain image.

9. The method according to claim 3, characterized in that, The training image for the first sensitivity is a low conversion gain training image; the training image for the second sensitivity is a high conversion gain training image and / or a lateral overflow collection capacitance training image.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor being used to invoke and execute the computer program to implement the method of any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by an electronic device, implements the method of any one of claims 1-9.

Images