Terminal image processing method, device and terminal equipment

Abstract

An embodiment of the present application provides an image processing method, apparatus, and terminal device for a terminal, wherein the terminal includes a black-and-white camera and a color camera. The method includes: after the camera function of the terminal is operated, during a preview process, collecting a color RAW image through the color camera and collecting a black-and-white RAW image through the black-and-white camera; obtaining a shooting instruction, controlling the color camera to shoot the current shooting scene, and obtaining a color RAW image of the shooting scene; performing RAW domain processing on the color RAW image to obtain a processed color RAW image; processing the processed color RAW image to obtain a color YUV image; and processing the black-and-white RAW image collected during the preview process through a second Bayer domain processing algorithm link to obtain a black-and-white YUV image; and fusing the color YUV image and the black-and-white YUV image to obtain a target color image, thereby retaining more image details and improving the color reproduction of the target color image.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of intelligent terminals, and in particular to a method, apparatus, and terminal device for processing terminal images. Background Art

[0002] Taking photos has become a common feature in mobile phones. With the increasing popularity of smartphones, cameras are being used more and more widely, and their capabilities are becoming increasingly advanced. Excellent camera performance has become a major selling point for smartphones. Technological innovations surrounding this function are constantly emerging, and apertures and sensor sizes are increasing. These technological and hardware upgrades have significantly improved both image quality and the overall photography experience. While pursuing higher image quality, the trend towards thinner and lighter smartphone bodies is also a major development trend. Therefore, most current smartphones use at least two camera modules, which not only improves image quality but also meets the demand for thinner and lighter phones.

[0003] However, taking a smartphone with two camera modules as an example, directly using the dual cameras will result in two different images. Therefore, an image fusion algorithm is needed to fuse the two dual-camera images to obtain the final dual-camera enhanced image.

[0004] Existing image fusion methods lose a lot of image details, resulting in poor color reproduction of the final image; especially when shooting scenery and / or still life in front-lit scenes, due to the limited capabilities of a single camera device and the detail enhancement capabilities of the algorithm, some details, especially high-frequency details, cannot be fully restored. Summary of the Invention

[0005] The embodiments of the present application provide a terminal image processing method, apparatus, server, and terminal device. The embodiments of the present application also provide a computer-readable storage medium to enable the terminal device to first perform RAW domain processing on the color RAW image captured by the color camera to convert it into a color YUV image, and then fuse it with the black and white YUV image, thereby retaining more image details and improving the color reproduction of the final image.

[0006] In a first aspect, the present application provides an image processing method for a terminal, the terminal including a black-and-white camera and a color camera. The method comprises: after the terminal's camera function is activated, during a preview process, capturing a color RAW image via the color camera and a black-and-white RAW image via the black-and-white camera; after the terminal device receives a shooting instruction, responding to the shooting instruction to obtain the color RAW image captured during the preview process, performing RAW domain processing on the color RAW image captured during the preview process to obtain a color YUV image; and processing the black-and-white RAW image captured during the preview process via a second Bayer domain processing algorithm link to obtain a black-and-white YUV image. Finally, the terminal device fuses the color YUV image with the black-and-white YUV image to obtain a target color image.

[0007] In one possible implementation, obtaining the color RAW image captured during the preview process in response to the shooting instruction may be: the terminal device obtains the color RAW image captured during the preview process from a cache in response to the shooting instruction.

[0008] In one possible implementation, the terminal device performs RAW domain processing on the color RAW image collected during the preview process. Before obtaining the color YUV image, the terminal device can also perform linearization and bad pixel correction on the color RAW image collected during the preview process through the first Bayer domain processing algorithm link.

[0009] In one possible implementation, before the terminal device obtains the shooting instruction, it can also determine that the current shooting scene is a non-high dynamic scene based on the color RAW image collected during the preview process.

[0010] In one possible implementation, the color RAW image captured by the terminal device during the preview process includes at least two frames of color RAW images. Thus, performing RAW domain processing on the color RAW image captured during the preview process to obtain a color YUV image includes: preprocessing the at least two frames of color RAW images; performing noise reduction processing on the at least two preprocessed color RAW images to fuse the at least two frames of color RAW images into one frame of color RAW image; converting the fused color RAW image into a color RGB image; adjusting the color and brightness of the above-mentioned color RGB image; and converting the color RGB image after the color and brightness adjustment into a color YUV image.

[0011] In one possible implementation, the terminal device preprocesses the at least two frames of color RAW images, including: aligning the sizes of the at least two frames of color RAW images; mapping the pixels in each two frames of the at least two frames of color RAW images to obtain a correspondence between the pixels; performing error detection on the correspondence and correcting the erroneous correspondence.

[0012] In one possible implementation, the terminal device fuses the color YUV image and the black-and-white YUV image to obtain the target color image by: aligning the sizes of the color YUV image and the black-and-white YUV image; obtaining the area to be fused in the color YUV image and the area to be fused in the black-and-white YUV image from the aligned color YUV image and the aligned black-and-white YUV image; then, mapping the pixels in the area to be fused in the color YUV image with the pixels in the area to be fused in the black-and-white YUV image to obtain the correspondence between the pixels; next, performing error detection on the above correspondence and correcting the erroneous correspondence; finally, fusing the clear pixels in the area to be fused in the color YUV image and the area to be fused in the black-and-white YUV image to the corresponding blurred pixels.

[0013] In one possible implementation, during the preview process, after determining that the current shooting scene is a non-high-dynamic scene, the color RAW image captured by the terminal device through the color camera includes a short-exposure color RAW image and a normal-exposure color RAW image; the above method may also include: using overlapping exposure high-dynamic technology to fuse the short-exposure color RAW image and the normal-exposure color RAW image; during the preview process, displaying the fused color image.

[0014] In one possible implementation, during the preview process, after the terminal device captures the color RAW image through the color camera, it can also cache the above-mentioned normally exposed color RAW image.

[0015] In the image processing method of the terminal provided in the embodiment of the present application, the terminal device first performs RAW domain processing on the color RAW image, converts the color RAW image into a color YUV image, and then merges it with the black and white YUV image, so as to retain more image details and improve the color reproduction of the target color image.

[0016] In a second aspect, an embodiment of the present application provides an image processing device for a terminal, which is included in a terminal device and has the function of implementing the terminal device behavior described in the first aspect and possible implementations of the first aspect. The functions can be implemented through hardware or through hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, a sending module or unit, etc.

[0017] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a black and white camera and a color camera; one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the above instructions are executed by the terminal device, the terminal device performs the following steps: after the camera function of the terminal device is run, during the preview process, a color RAW image is collected through the color camera, and a black and white RAW image is collected through the black and white camera; after obtaining a shooting instruction, the color RAW image collected during the preview process is obtained in response to the above shooting instruction, and the color RAW image collected during the preview process is processed in the RAW domain to obtain a color YUV image; the black and white RAW image collected during the preview process is processed through a second Bayer domain processing algorithm link to obtain a black and white YUV image; the color YUV image and the black and white YUV image are merged to obtain a target color image.

[0018] In one possible implementation, when the above-mentioned instruction is executed by the terminal device, the terminal device executes the step of obtaining the color RAW image captured during the preview process in response to the above-mentioned shooting instruction, including: obtaining the color RAW image captured during the preview process from the cache in response to the above-mentioned shooting instruction.

[0019] In one possible implementation, when the above-mentioned instruction is executed by the terminal device, the terminal device performs RAW domain processing on the color RAW image captured during the preview process to obtain a color YUV image, and further performs the following steps: linearizing and performing bad pixel correction processing on the color RAW image captured during the preview process through a first Bayer domain processing algorithm link.

[0020] In one possible implementation, when the above instruction is executed by the terminal device, before the terminal device executes the step of obtaining the shooting instruction, it also executes the following steps: based on the color RAW image captured during the preview process, determine that the current shooting scene is a non-high dynamic scene.

[0021] In one possible implementation, the color RAW image captured during the preview process includes at least two frames of color RAW images; when the above instruction is executed by the terminal device, the terminal device performs RAW domain processing on the color RAW image captured during the preview process, and the step of obtaining a color YUV image may include: preprocessing the at least two frames of color RAW images; performing noise reduction processing on the at least two preprocessed color RAW images to fuse the at least two frames of color RAW images into one frame of color RAW image; converting the fused color RAW image into a color RGB image; adjusting the color and brightness of the color RGB image; and converting the color RGB image after the color and brightness adjustment into a color YUV image.

[0022] In one possible implementation, when the above-mentioned instruction is executed by the terminal device, the terminal device performs the steps of pre-processing the above-mentioned at least two frames of color RAW images, including: aligning the sizes of the above-mentioned at least two frames of color RAW images; mapping the pixels in each two frames of the above-mentioned at least two frames of color RAW images to obtain a correspondence between the pixels; performing error detection on the above-mentioned correspondence and correcting the erroneous correspondence.

[0023] In one possible implementation method, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the color YUV image and the black and white YUV image, and the steps for obtaining the target color image may be: aligning the sizes of the color YUV image and the black and white YUV image; obtaining the area that needs to be fused in the color YUV image and the area that needs to be fused in the black and white YUV image from the aligned color YUV image and the aligned black and white YUV image; mapping the pixels in the area that needs to be fused in the color YUV image with the pixels in the area that needs to be fused in the black and white YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; and fusing the clear pixels in the area that needs to be fused in the color YUV image and the area that needs to be fused in the black and white YUV image to the corresponding blurred pixels.

[0024] In one possible implementation, during the preview process, after determining that the current shooting scene is a non-high-dynamic scene, the color RAW image captured by the terminal device through the color camera includes a short-exposure color RAW image and a normal-exposure color RAW image; when the above instruction is executed by the terminal device, the terminal device also performs the following steps: using overlapping exposure high-dynamic technology to fuse the short-exposure color RAW image and the normal-exposure color RAW image; during the preview process, the fused color image is displayed.

[0025] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the following steps after the step of capturing the color RAW image through the color camera during the preview process: caching the above-mentioned normally exposed color RAW image.

[0026] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0029] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0031] FIG2( a ) is a schematic diagram of a display interface of a terminal device provided in one embodiment of the present application;

[0032] FIG2( b ) is a schematic diagram of a display interface of a terminal device provided in another embodiment of the present application;

[0033] Figure 3 A schematic diagram of an image processing method for a terminal provided in one embodiment of the present application;

[0034] Figure 4 A schematic diagram of an image processing method for a terminal provided in another embodiment of the present application;

[0035] Figure 5 A flowchart of color and black and white image fusion provided for one embodiment of the present application;

[0036] Figure 6 A schematic structural diagram of a terminal device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0037] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0038] In an image fusion solution provided by existing related technologies, the original (RAW) images captured by a color camera and a black and white camera are converted into a color YUV image and a black and white YUV image respectively, and then the color YUV image and the black and white YUV image are fused. This image fusion solution loses a lot of image details, resulting in poor color reproduction of the final image.

[0039] Especially when shooting scenery and / or still life in front-lit scenes, due to the limited capabilities of a single camera device and the detail enhancement capabilities of the algorithm, some details, especially high-frequency details, cannot be fully restored.

[0040] Based on the above problems, an embodiment of the present application provides an image processing method for a terminal, which can enable the terminal device to first perform RAW domain processing on the color RAW image captured by the color camera, and then convert it into a color YUV image, and merge it with the black and white YUV image, thereby retaining more image details and improving the color reproduction of the final image.

[0041] The image processing method of the terminal provided in the embodiment of the present application can be applied to terminal devices, wherein the above-mentioned terminal devices can be smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks or personal digital assistants (PDAs) and other devices; the embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0042] For example, Figure 1 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application is shown as follows: Figure 1 As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0045] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0046] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0047] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0048] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (DCL). In some embodiments, the processor 110 may include multiple I2C busses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the terminal device 100.

[0049] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0050] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0051] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0052] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the terminal device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0053] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0054] The USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100, or to transfer data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0055] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0056] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device 100 via the power management module 141.

[0057] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0058] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0059] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0060] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0061] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0062] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0063] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0064] The terminal device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0065] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0066] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0067] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0068] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include N cameras 193, where N is a positive integer greater than 1. In the embodiment of the present application, the N cameras 193 include a black and white camera and a color camera.

[0069] Specifically, during the shooting process, the user turns on the camera, and light is transmitted to the photosensitive element through the lens (which can correspond to the camera 193 described previously). In other words, the lens can project the ambient light signal onto the photosensitive area of ​​the photosensitive element, and the photosensitive element converts the light signal into an image visible to the naked eye through photoelectric conversion. The photosensitive element then transmits the internal original image (Bayer format, also known as Bayer format or Bayer format) to the ISP module. The ISP module processes the image in the RGB spatial domain through the algorithm and outputs it to the back-end acquisition unit, which is displayed in the image preview area of ​​the terminal device 100 or on the display screen of the terminal device 100. In this process, the processor controls the lens, photosensitive element and ISP module accordingly through the firmware program running on it, thereby completing the image preview or shooting function.

[0070] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0071] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows terminal device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0072] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in the terminal device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0073] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0074] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0075] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0076] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0077] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0078] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0079] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0080] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0081] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the location of the touch based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0082] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0083] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0084] The magnetic sensor 180D includes a Hall effect sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Furthermore, based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0085] Accelerometer 180E can detect the magnitude of acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0086] The distance sensor 180F is used to measure distance. The terminal device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0087] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect when the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0088] Ambient light sensor 180L is used to sense ambient light brightness. Terminal device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether terminal device 100 is in a pocket to prevent accidental touches.

[0089] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0090] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to prevent the terminal device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0091] The touch sensor 180K is also called a "touch control device." The touch sensor 180K can be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be provided on the surface of the terminal device 100, at a location different from that of the display screen 194.

[0092] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0093] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0094] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0095] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0096] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the terminal device 100 by inserting or removing it from the SIM card interface 195. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0097] For ease of understanding, the Bayer domain and the RAW domain mentioned in the following embodiments are first described.

[0098] 1. Bayer Domain: Each lens on a digital camera has a light sensor that measures light brightness. However, to obtain a full-color image, three light sensors are generally required to obtain information about the three primary colors of red, green, and blue. To reduce the cost and size of digital cameras, manufacturers typically use CCD or CMOS image sensors. Typically, the raw image output by a CMOS image sensor is in Bayer Domain RGB format, where a single pixel contains only one color value. To obtain the image's grayscale value, the color information of each pixel must be interpolated and then the grayscale value of each pixel must be calculated. In other words, Bayer Domain refers to a raw image format within a digital camera.

[0099] 2. RAW Domain: RAW domain images, or raw images, contain data processed from the image sensor of a digital camera, scanner, or film scanner. They are so named because they have not yet been processed, printed, or edited. RAW domain images contain the original image information and have not undergone the nonlinear processing used in the ISP process.

[0100] The following examples of this application will be Figure 1 Taking the terminal device with the structure shown as an example, the image processing method of the terminal provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0101] Specifically, after the user clicks the "camera" icon in the display interface of the terminal device, the terminal device runs the photo-taking function in response to the user's operation. During the preview process, the terminal device collects color RAW images through the color camera and collects black and white RAW images through the black and white camera. Then, based on the color RAW images collected during the preview process, the terminal device determines that the current shooting scene is a non-high-dynamic scene, wherein a non-high-dynamic scene refers to a scene with sufficient lighting and a small brightness span in the current shooting scene, such as: buildings or scenery. After determining that the current shooting scene is a non-high-dynamic scene, the terminal device can display the current shooting scene as a non-high-dynamic scene in the current display interface, as shown in Figure 2(a). Figure 2(a) is a schematic diagram of the display interface of the terminal device provided in one embodiment of the present application, and Figure 2(a) takes the non-high-dynamic scene of a building as an example.

[0102] Specifically, the terminal device can determine that the current shooting scene is a non-high-dynamic-state scene based on the color RAW image captured during the preview process by obtaining the maximum brightness and minimum brightness of the color RAW image captured during the preview process and calculating the ratio of the maximum brightness to the minimum brightness; if the ratio is less than a predetermined ratio threshold, the current shooting scene can be determined to be a non-high-dynamic-state scene. The size of the predetermined ratio threshold can be set according to system performance and / or implementation requirements during specific implementation, and this embodiment does not limit the size of the predetermined ratio threshold.

[0103] In addition, it should be noted that the current shooting scene can be determined by the terminal device in the above manner, or can be selected by the user in the camera function interface.

[0104] In addition, if during the preview process, the terminal device detects that the black-and-white camera among the N cameras 193 is blocked, a prompt message "Black-and-white camera is blocked" can be displayed on the current display interface, as shown in Figure 2(b), to remind the user that the black-and-white camera is blocked. This allows the user to remove the obstruction of the black-and-white camera as soon as possible to ensure the quality of the captured image. Figure 2(b) is a schematic diagram of the display interface of a terminal device provided in another embodiment of the present application.

[0105] Further, refer to Figure 3 , Figure 3 A schematic diagram of an image processing method for a terminal provided in one embodiment of the present application, from Figure 3 It can be seen that after determining that the current shooting scene is a non-high-dynamic scene (35), the terminal device obtains a shooting instruction, and in response to the shooting instruction, the terminal device obtains the color RAW image captured during the preview process. Specifically, in response to the shooting instruction, the terminal device can obtain the color RAW image captured during the preview process from the cache; that is, during the preview process, after the terminal device captures the color RAW image through the color camera, it can store the captured color RAW image in the cache, and after obtaining the shooting instruction, it can obtain the color RAW image captured during the preview process from the cache.

[0106] Then, the terminal device performs linearization and bad pixel correction processing on the color RAW image collected during the preview process through the Bayer domain processing algorithm link 31. The Bayer domain processing algorithm link 31 is a first Bayer domain processing algorithm link for processing the color RAW image.

[0107] Furthermore, the terminal device performs RAW domain processing 32 on the color RAW image collected during the preview process to obtain a color YUV image.

[0108] During the preview process, the terminal device captures a black-and-white RAW image via a black-and-white camera. In response to the capture instruction, the terminal device obtains the black-and-white RAW image captured during the preview process and processes the black-and-white RAW image through Bayer domain processing algorithm link 33 to obtain a black-and-white YUV image. Bayer domain processing algorithm link 33 is a second Bayer domain processing algorithm link, and is used to process the black-and-white RAW image.

[0109] Finally, the terminal device fuses the color YUV image and the black and white YUV image (34) to obtain the target color image and displays the target color image (i.e. Figure 3 (Photograph and print in the).

[0110] Also, see Figure 3 During the preview process, after determining that the current shooting scene is a non-high dynamic range scene, the color RAW image captured by the terminal device through the color camera may include a short-exposure color RAW image and a normal-exposure color RAW image; in this way, the terminal device may also use the stagger high dynamic range (Stagger HDR) technology to fuse the above-mentioned short-exposure color RAW image and the normal-exposure color RAW image (36); then, during the above-mentioned preview process, the fused color image is displayed (37).

[0111] In a specific implementation, the terminal device can process the above-mentioned normally exposed color RAW image through image processing front-end 0, and process the above-mentioned short-time exposed color RAW image through image processing front-end 1, and then use Stagger HDR to fuse the color RAW images processed by image processing front-end 0 and image processing front-end 1 (36), and then preview and display the fused color image (37). When using Stagger HDR to fuse the color RAW images processed by image processing front-end 0 and image processing front-end 1, the fusion ratio of the color RAW images processed by image processing front-end 0 and image processing front-end 1 can be determined according to the light intensity in the current shooting scene; for example, if the current shooting scene has sufficient light and is a front-lit scene, then the fusion ratio of the color RAW images processed by image processing front-end 0 and image processing front-end 1 can be determined to be 95% and 5% respectively, so that the effect of the fused color image is similar to that of the image without fusion.

[0112] In addition, for the black and white RAW images captured by the black and white camera during the preview process, one path is sent to the Bayer domain processing algorithm link 33 for processing, and another path is sent to the image processing front end 2 for processing. The black and white RAW images processed by the image processing front end 2 are sent to the 3A statistics / semantic understanding module 38. The 3A statistics / semantic understanding module 38 mainly realizes the functions of automatic color, brightness and / or focus based on the black and white RAW images, and recognizes the semantic content of the image and uses it for decision-making.

[0113] The following combination Figure 4 , the image processing method of the terminal provided in the embodiment of the present application is described in detail, Figure 4 A schematic diagram of an image processing method for a terminal provided in another embodiment of the present application.

[0114] In this embodiment, during the preview process, the terminal device collects color RAW images through the color camera. Then, the terminal device determines that the current shooting scene is a non-high dynamic scene based on the color RAW images collected during the preview process. After that, the images collected by the terminal device through the color camera include normally exposed color RAW images and short-exposed color RAW images, and the normally exposed color RAW images are cached in the color image cache.

[0115] For a black-and-white RAW image captured by a terminal device through a black-and-white camera, the terminal device caches the black-and-white RAW image in a black-and-white image cache. The black-and-white RAW image is a normally exposed black-and-white RAW image.

[0116] Next, the terminal device obtains a shooting instruction, and in response to the shooting instruction, obtains 8 frames of normally exposed color RAW images from the color image buffer.

[0117] Then, the terminal device performs linearization and bad pixel correction processing on the 8 frames of color RAW images collected during the preview process through the Bayer domain processing algorithm link 41. In this embodiment, Figure 4 The Bayer domain processing algorithm in link 41 corresponds to Figure 3 The Bayer domain processing algorithm link 31 in is used to process color RAW images.

[0118] Next, the terminal device caches the 8 frames of color RAW images after linearization and bad pixel correction into the image cache pool 42, and then performs RAW domain processing (43) on the 8 frames of color RAW images in the image cache pool 42 to obtain color YUV images, and caches the color YUV images into the image cache 44. Figure 4 As can be seen from FIG. 4 , the number of bits of the color YUV image cached in the image cache 44 is 10.

[0119] Specifically, performing RAW domain processing (43) on the 8 frames of color RAW images in the image buffer pool 42 to obtain a color YUV image may include:

[0120] Step 1: preprocess the eight color RAW images. Preprocessing the eight color RAW images may include aligning the sizes of the eight color RAW images, mapping pixels between every two frames of the eight color RAW images to obtain a correspondence between the pixels, performing error detection on the correspondence, and correcting any erroneous correspondence.

[0121] Step 2: performing noise reduction processing on the pre-processed 8 frames of color RAW images to fuse the 8 frames of color RAW images into one frame of color RAW image.

[0122] Step 3: Convert the fused color RAW image into a color RGB image.

[0123] Step 4: Adjust the color and brightness of the color RGB image. Specifically, adjusting the color and brightness of the color RGB image may include performing brightness and color processing procedures such as automatic white balance (AWB), shading, and gamma correction on the color RGB image.

[0124] Step 5: Convert the color RGB image after adjusting the color and brightness into a color YUV image.

[0125] Also, see Figure 4 During the preview process, for a black-and-white RAW image captured by a black-and-white camera on a terminal device, after the terminal device receives a shooting instruction, in response to the shooting instruction, the terminal device obtains a frame of black-and-white RAW image from the black-and-white image buffer, and sequentially performs linearization, bad pixel correction, Bayer processing, graph transformation matching (GTM), and color correction (Gamma) processing on the black-and-white RAW image through the Bayer domain processing algorithm link 45 to obtain a black-and-white YUV image, which is then cached in the image buffer 46. The number of bits of the black-and-white YUV image cached in the image buffer 46 is 10. In this embodiment, the Bayer domain processing algorithm link 45 corresponds to Figure 3 The Bayer domain processing algorithm link 33 in is used to process black and white RAW images.

[0126] Finally, the terminal device fuses (47) the color YUV image in the image buffer 44 and the black-and-white YUV image in the image buffer 46 to obtain a target color image, and displays the target color image.

[0127] Specifically, Figure 5 A flowchart of color and black and white image fusion is provided for one embodiment of the present application, such as Figure 5 As shown, fusing the color YUV image and the black and white YUV image to obtain the target color image may include:

[0128] Step 501: align the sizes of the color YUV image and the black-and-white YUV image.

[0129] Step 502 : Obtain, from the resized color YUV image and the resized black-and-white YUV image, an area to be fused in the color YUV image and an area to be fused in the black-and-white YUV image.

[0130] Step 503 : Mapping pixels in the area where the color YUV image needs to be fused with pixels in the area where the black and white YUV image needs to be fused to obtain a corresponding relationship between the pixels.

[0131] Step 504: perform error detection on the above correspondences and correct any erroneous correspondences.

[0132] Step 505: Blend the clear pixels in the area where the color YUV image needs to be fused and the area where the black-and-white YUV image needs to be fused onto the corresponding blurred pixels. For example, pixel A in the area where the color YUV image needs to be fused corresponds to pixel A' in the area where the black-and-white YUV image needs to be fused. In one case, assuming that pixel A is the clear pixel between pixel A and pixel A', and pixel A' is the blurred pixel between the two, the color of pixel A can be fused onto pixel A'. In another case, assuming that pixel A is the blurred pixel between pixel A and pixel A', and pixel A' is the clear pixel between the two, the details of pixel A' can be fused onto pixel A.

[0133] In the image processing method of the terminal provided in the embodiment of the present application, the terminal device first performs RAW domain processing on the color RAW image, converts the above-mentioned color RAW image into a color YUV image, and then merges the color YUV image with the black and white YUV image, so that more image details can be retained. In particular, when shooting scenery and / or still lifes in a front-lit scene, the image details, especially high-frequency details, can be better restored, thereby improving the color reproduction of the target color image.

[0134] In addition, during the preview process, after determining that the current shooting scene is a non-high-dynamic scene, the color RAW image collected by the terminal device through the color camera includes a short-exposure color RAW image and a normal-exposure color RAW image. After the above-mentioned short-exposure color RAW image and the normal-exposure color RAW image are fused, the fused image is previewed and displayed, which can reduce the difference between the image obtained by shooting and the image displayed for preview, thereby improving the user experience.

[0135] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in a different order than those presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0136] It is understandable that, in order to implement the above functions, the terminal device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0137] This embodiment can divide the terminal device into functional modules based on the above method embodiment. For example, each functional module can be divided according to its function, or two or more functions can be integrated into a single module. The above integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division. In actual implementation, other division methods may be used.

[0138] Figure 6 This is a schematic diagram of the structure of a terminal device provided in another embodiment of the present application, in which each functional module is divided into corresponding functional modules. Figure 6 A possible schematic diagram of the terminal device 600 involved in the above embodiment is shown. Figure 6 As shown, the terminal device 600 may include: a receiving unit 601, a processing unit 602 and a sending unit 603;

[0139] The processing unit 602 can be used to support the terminal device 600 to implement the present application. Figure 2(a) to Figure 5 The technical solution described in the illustrated embodiment.

[0140] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0141] The terminal device 600 provided in this embodiment is used to execute the above-mentioned terminal image processing method, and thus can achieve the same effect as the above-mentioned method.

[0142] It should be understood that the terminal device 600 may correspond to Figure 1 The terminal device 100 shown in FIG. The functions of the receiving unit 601 and the sending unit 603 can be Figure 1 The processor 110, antenna 1 and mobile communication module 60 in the terminal device 100 shown, and / or, are implemented by the processor 110, antenna 2 and wireless communication module 160; the function of the processing unit 602 can be implemented by Figure 1 The processor 110 in the terminal device 100 shown is implemented.

[0143] In the case of adopting an integrated unit, the terminal device 600 may include a processing module, a storage module, and a communication module.

[0144] The processing module can be used to control and manage the actions of the terminal device 600. For example, it can be used to support the terminal device 600 in executing the steps performed by the receiving unit 601, processing unit 602, and sending unit 603. The storage module can be used to support the terminal device 600 in storing program code and data. The communication module can be used to support communication between the terminal device 600 and other devices.

[0145] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0146] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 600 involved in this embodiment can be a Figure 1 Device with the structure shown.

[0147] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 2(a) to Figure 5 The method provided by the illustrated embodiment.

[0148] The present invention also provides a computer program product, which includes a computer program that, when executed on a computer, enables the computer to execute the present invention. Figure 2(a) to Figure 5 The method provided by the illustrated embodiment.

[0149] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0150] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0153] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A terminal image processing method, characterized in that: The terminal includes a black and white camera and a color camera, and the method includes: After the camera function of the terminal is running, during the preview process, a color RAW image is captured by the color camera, a normally exposed black and white RAW image is captured by the black and white camera, and the normally exposed black and white RAW image is cached in a black and white image cache; After determining that the current shooting scene is a non-high dynamic range scene based on the color RAW image, capturing a normally exposed color RAW image and a short-exposure color RAW image through the color camera, and caching the normally exposed color RAW image in a color image cache; wherein the non-high dynamic range scene refers to the current shooting scene being a front-lit scene with sufficient lighting and a small brightness span; Acquire and respond to a shooting instruction, obtain multiple frames of the normally exposed color RAW images from the color image buffer, perform linearization and bad pixel correction processing, and RAW domain processing on the multiple frames of the normally exposed color RAW images, and obtain a frame of color YUV image; performing linearization, bad pixel correction, Bayer processing, image transformation matching, and color correction on a frame of the normally exposed black-and-white RAW image in the black-and-white image buffer through a second Bayer domain processing algorithm link to obtain a black-and-white YUV image; The color YUV image and the black-and-white YUV image are fused to obtain a target color image.

2. The method according to claim 1, characterized in that The step of performing linearization and bad pixel correction processing and RAW domain processing on the multiple frames of normally exposed color RAW images to obtain a frame of color YUV image includes: The first Bayer domain processing algorithm chain is used to perform linearization and bad pixel correction processing on each frame of the normally exposed color RAW image.

3. The method according to claim 2, characterized in that The step of performing linearization and bad pixel correction processing and RAW domain processing on the multiple frames of normally exposed color RAW images to obtain a frame of color YUV image includes: Preprocessing each frame of the normally exposed color RAW image after linearization and bad pixel correction processing; performing noise reduction processing on the pre-processed frames of the normally exposed color RAW images to fuse the frames of the normally exposed color RAW images into one frame of color RAW image; Convert the fused color RAW image into a color RGB image; Adjusting the color and brightness of the color RGB image; Converts a color RGB image to a color YUV image after adjusting color and brightness.

4. The method according to claim 3, characterized in that The preprocessing of each frame of the normally exposed color RAW image after the linearization and bad pixel correction processing includes: aligning the sizes of the normally exposed color RAW images of each frame after linearization and bad pixel correction processing; Mapping pixels in every two frames of the aligned normally exposed color RAW images to obtain a corresponding relationship between the pixels; Error detection is performed on the corresponding relationship, and the corresponding relationship in which an error occurs is corrected.

5. The method according to claim 1, wherein The fusing the color YUV image and the black-and-white YUV image to obtain a target color image includes: Aligning the sizes of the color YUV image and the black and white YUV image; Obtaining the area to be fused in the color YUV image and the area to be fused in the black and white YUV image from the resized color YUV image and the resized black and white YUV image; Mapping pixels in the area where the color YUV image needs to be fused with pixels in the area where the black and white YUV image needs to be fused to obtain a correspondence between the pixels; Performing error detection on the corresponding relationship and correcting the corresponding relationship where an error occurs; The clear pixels in the area where the color YUV image needs to be fused and the area where the black-and-white YUV image needs to be fused are fused onto the corresponding blurred pixels.

6. The method according to claim 1, characterized in that The method further comprises: Using overlapping exposure high dynamic technology to fuse the short-exposure color RAW image and the normal-exposure color RAW image; During the preview process, the fused color image is displayed.

7. An image processing device for a terminal, characterized in that: The image processing device includes a receiving unit, a processing unit, and a sending unit, wherein the receiving unit, the processing unit, and the sending unit are configured to execute the method according to any one of claims 1 to 6.

8. A terminal device, characterized in that: The terminal device comprises a black and white camera and a color camera; one or more processors; a memory; a plurality of application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory and the one or more computer programs include instructions. When the instructions are executed by the terminal device, the terminal device performs the following steps: After the camera function of the terminal is running, during the preview process, a color RAW image is captured by the color camera, a normally exposed black and white RAW image is captured by the black and white camera, and the normally exposed black and white RAW image is cached in a black and white image cache; After determining that the current shooting scene is a non-high dynamic range scene based on the color RAW image, capturing a normally exposed color RAW image and a short-exposure color RAW image through the color camera, and caching the normally exposed color RAW image in a color image cache; wherein the non-high dynamic range scene refers to the current shooting scene being a front-lit scene with sufficient lighting and a small brightness span; Acquire and respond to a shooting instruction, obtain multiple frames of the normally exposed color RAW images from the color image buffer, perform linearization and bad pixel correction processing, and RAW domain processing on the multiple frames of the normally exposed color RAW images, and obtain a frame of color YUV image; performing linearization, bad pixel correction, Bayer processing, image transformation matching, and color correction on a frame of the normally exposed black-and-white RAW image in the black-and-white image buffer through a second Bayer domain processing algorithm link to obtain a black-and-white YUV image; The color YUV image and the black-and-white YUV image are fused to obtain a target color image.

9. The terminal device according to claim 8, characterized in that When the instruction is executed by the terminal device, the terminal device performs linearization and bad pixel correction processing, as well as RAW domain processing on a plurality of frames of normally exposed color RAW images to obtain a frame of color YUV image, including the following steps: The first Bayer domain processing algorithm chain is used to perform linearization and bad pixel correction processing on each frame of the normally exposed color RAW image.

10. The terminal device according to claim 9, characterized in that When the instruction is executed by the terminal device, the terminal device performs linearization and bad pixel correction processing, as well as RAW domain processing on a plurality of frames of normally exposed color RAW images to obtain a frame of color YUV image, including the following steps: Preprocessing each frame of the normally exposed color RAW image after linearization and bad pixel correction processing; performing noise reduction processing on the pre-processed frames of the normally exposed color RAW images to fuse the frames of the normally exposed RAW images into one frame of color RAW image; Convert the fused color RAW image into a color RGB image; Adjusting the color and brightness of the color RGB image; Converts a color RGB image to a color YUV image after adjusting color and brightness.

11. The terminal device according to claim 10, characterized in that When the instruction is executed by the terminal device, the step of causing the terminal device to perform the pre-processing of each frame of the normally exposed color RAW image after the linearization and bad pixel correction processing includes: aligning the sizes of the normally exposed color RAW images of each frame after linearization and bad pixel correction processing; Mapping pixels in every two frames of the aligned normally exposed color RAW images to obtain a corresponding relationship between the pixels; Error detection is performed on the corresponding relationship, and the corresponding relationship in which an error occurs is corrected.

12. The terminal device according to claim 8, characterized in that When the instruction is executed by the terminal device, the terminal device performs the step of fusing the color YUV image and the black-and-white YUV image to obtain a target color image, which includes: Aligning the sizes of the color YUV image and the black and white YUV image; Obtaining the area to be fused in the color YUV image and the area to be fused in the black and white YUV image from the resized color YUV image and the resized black and white YUV image; Mapping pixels in the area where the color YUV image needs to be fused with pixels in the area where the black and white YUV image needs to be fused to obtain a correspondence between the pixels; Performing error detection on the corresponding relationship and correcting the corresponding relationship where an error occurs; The clear pixels in the area where the color YUV image needs to be fused and the area where the black-and-white YUV image needs to be fused are fused onto the corresponding blurred pixels.

13. The terminal device according to claim 8, characterized in that During the preview process, when the instruction is executed by the terminal device, the terminal device further performs the following steps: Using overlapping exposure high dynamic technology to fuse the short-exposure color RAW image and the normal-exposure color RAW image; During the preview process, the fused color image is displayed.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 6.

Images