Image processing method and terminal device

Abstract

An embodiment of the present application provides an image processing method and a terminal device, wherein the terminal includes a black and white camera, a color camera, an ultra-wide-angle camera and a telephoto camera. In the image processing method of the terminal, after the camera function of the terminal device is run, the current zoom factor is obtained, and after obtaining a shooting instruction, in response to the shooting instruction, a color RAW image captured by the color camera is obtained, and a RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera is obtained according to the current zoom factor, and then the color RAW image is fused with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain a target color image, so that in the scenario where the user adjusts the zoom factor, the detail effect of the image at different focal lengths can be improved, the overall clarity of the image can be improved, and the user experience can be improved.

Description

[0001] This application is a divisional application of the invention patent with application date of August 12, 2021, application number 202110923166.0, and invention name “Image processing method, device and terminal device of terminal”. Technical Field

[0002] The embodiments of the present application relate to the field of intelligent terminal technology, and in particular to an image processing method and a terminal device. Background Art

[0003] With the development of smart phones, the function of mobile phone cameras in users' mobile phones is becoming more and more important, gradually developing from single-camera to dual-camera, triple-camera and even more cameras, achieving the effect comparable to that of SLR cameras. Small camera size and large zoom range are the two important functions of current mobile phone cameras. Although optical zoom can obtain high-quality zoom images by using optical lenses, it will inevitably increase the size of the camera and greatly increase the cost of implementation; while ordinary single-camera digital zoom can control the size of the camera and reduce the cost of implementation, the quality of the image obtained by using the single-camera digital zoom solution is poor.

[0004] Therefore, the technology of simulating optical zoom using multiple cameras with different focal lengths came into being. For example, the terminal device can be equipped with a color camera, a black and white camera, an ultra-wide-angle camera and a telephoto camera. Since the hardware parameters of different cameras themselves (for example, optical center, focal length, field of view (FOV) and / or distortion and other hardware parameters) are different, and the installation arrangements of the modules are different (for example, the baseline, relative angle and / or position of the arrangement, etc.), the ultra-wide-angle camera and telephoto camera on the same module will inevitably obtain images with different FOVs, different relative positions and different occlusions when shooting the same object. Summary of the invention

[0005] The embodiments of the present application provide an image processing method and a terminal device. The embodiments of the present application also provide a computer-readable storage medium, so that the terminal device can fuse the color RAW image captured by the color camera with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor of the camera to obtain a target color image, thereby improving the image quality obtained by shooting in a scenario where the user adjusts the zoom factor and improving the user experience.

[0006] In a first aspect, the present application provides an image processing method for a terminal, wherein the terminal includes a black and white camera, a color camera, an ultra-wide-angle camera and a telephoto camera, and the method includes: after the camera function of the terminal is run, obtaining the current zoom factor; after obtaining a shooting instruction, in response to the shooting instruction, obtaining a color RAW image captured by the color camera, and obtaining a RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor; fusing the color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain a target color image.

[0007] In one possible implementation, obtaining a RAW image captured by one of the black-and-white camera, the ultra-wide-angle camera, and the telephoto camera according to the current zoom factor can be: if the current zoom factor is less than 1 times the focal length, obtaining the ultra-wide-angle RAW image captured by the ultra-wide-angle camera; if the current zoom factor is greater than or equal to 1 times the focal length, and less than or equal to N times the focal length, obtaining the black-and-white RAW image captured by the black-and-white camera; if the current zoom factor is greater than N times the focal length, obtaining the telephoto RAW image captured by the telephoto camera; wherein N is a positive number, and N>1.

[0008] In one possible implementation, the color RAW image is fused with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain the target color image: if the current zoom factor is less than 1 focal length, after acquiring the ultra-wide-angle RAW image, the ultra-wide-angle RAW image is processed through the Bayer domain processing algorithm link to obtain the ultra-wide-angle YUV image; and the color RAW image is processed to obtain the color YUV image; and the ultra-wide-angle YUV image is fused with the color YUV image to obtain the target color image.

[0009] In one possible implementation, after obtaining the current zoom factor, if the current zoom factor is less than 1x focal length, an ultra-wide-angle RAW image captured by the ultra-wide-angle camera is obtained during the preview process; the ultra-wide-angle RAW image captured during the preview process is processed by the image processing front end; and the processed ultra-wide-angle RAW image is displayed during the preview process.

[0010] In one possible implementation, the ultra-wide-angle YUV image and the color YUV image are merged to obtain the target color image by: aligning the sizes of the ultra-wide-angle YUV image and the color YUV image; obtaining the area to be merged in the ultra-wide-angle YUV image and the area to be merged in the color YUV image from the aligned ultra-wide-angle YUV image and the aligned color YUV image; mapping the pixels in the area to be merged in the ultra-wide-angle YUV image with the pixels in the area to be merged in the color YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; and merging the clear pixels in the area to be merged in the ultra-wide-angle YUV image and the area to be merged in the color YUV image onto the corresponding blurred pixels.

[0011] In one possible implementation, the color RAW image is fused with the RAW image captured by one of the black-and-white camera, the ultra-wide-angle camera, and the telephoto camera to obtain the target color image: if the current zoom factor is greater than or equal to 1 times the focal length and less than or equal to N times the focal length, after acquiring the black-and-white RAW image, the black-and-white RAW image is processed through the Bayer domain processing algorithm link to obtain a black-and-white YUV image; and the color RAW image is processed in the RAW domain to obtain a color YUV image; and the black-and-white YUV image is fused with the color YUV image to obtain the target color image.

[0012] In one possible implementation, the black-and-white YUV image and the color YUV image are merged to obtain the target color image by: aligning the sizes of the color YUV image and the black-and-white YUV image; obtaining the area to be merged in the color YUV image and the area to be merged in the black-and-white YUV image from the aligned color YUV image and the aligned black-and-white YUV image; mapping the pixels in the area to be merged in the color YUV image with the pixels in the area to be merged in the black-and-white YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; and merging the clear pixels in the area to be merged in the color YUV image and the area to be merged in the black-and-white YUV image onto the corresponding blurred pixels.

[0013] In one possible implementation, the color RAW image is fused with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain the target color image: if the current zoom factor is greater than N times the focal length, after acquiring the telephoto RAW image, the telephoto RAW image is processed by a Bayer domain processing algorithm link to obtain a telephoto YUV image; and the color RAW image is processed to obtain a color YUV image; and the telephoto YUV image is fused with the color YUV image to obtain the target color image.

[0014] In one possible implementation, the telephoto YUV image and the color YUV image are fused to obtain the target color image by: aligning the sizes of the telephoto YUV image and the color YUV image; obtaining the area to be fused in the telephoto YUV image and the area to be fused in the color YUV image from the aligned telephoto YUV image and the aligned color YUV image; mapping the pixels in the area to be fused in the telephoto YUV image with the pixels in the area to be fused in the color YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; and fusing the clear pixels in the area to be fused in the telephoto YUV image and the area to be fused in the color YUV image to the corresponding blurred pixels.

[0015] In one possible implementation, after obtaining the current zoom factor, if the current zoom factor is greater than N times the focal length, a color RAW image captured by the color camera is obtained during the preview process; the color RAW image captured during the preview process is processed by the image processing front end; and the processed color RAW image is displayed during the preview process.

[0016] In one possible implementation, the color RAW image is processed to obtain a color YUV image by linearizing and correcting bad pixels on the color RAW image through a Bayer domain processing algorithm link, and RAW domain processing is performed on the linearized and bad pixel corrected color RAW image to obtain a color YUV image.

[0017] In a second aspect, an embodiment of the present application provides an image processing device for a terminal, which is included in a terminal device, and has the function of implementing the behavior of the terminal device in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, a sending module or unit, etc.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, comprising a black and white camera, a color camera, an ultra-wide-angle camera and a telephoto camera; one or more processors; a memory; multiple applications; and one or more computer programs, wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the terminal device, cause the terminal device to perform the following steps: after the camera function of the terminal device is run, obtain the current zoom factor; after obtaining the shooting instruction, in response to the shooting instruction, obtain the color RAW image captured by the color camera, and obtain the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor; fuse the color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain a target color image.

[0019] In one possible implementation, when the above instruction is executed by the terminal device, the step of enabling the terminal device to obtain a RAW image captured by one of the black-and-white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor may be: if the current zoom factor is less than 1 times the focal length, then the ultra-wide-angle RAW image captured by the ultra-wide-angle camera is obtained; if the current zoom factor is greater than or equal to 1 times the focal length and less than or equal to N times the focal length, then the black-and-white RAW image captured by the black-and-white camera is obtained; if the current zoom factor is greater than N times the focal length, then the telephoto RAW image captured by the telephoto camera is obtained; wherein N is a positive number, and N>1.

[0020] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera, and the step of obtaining the target color image can be: if the current zoom factor is less than 1 focal length, after acquiring the ultra-wide-angle RAW image, the ultra-wide-angle RAW image is processed through the Bayer domain processing algorithm link to obtain an ultra-wide-angle YUV image; and the color RAW image is processed to obtain a color YUV image; and the above ultra-wide-angle YUV image is fused with the above color YUV image to obtain the target color image.

[0021] In one possible implementation, when the above instruction is executed by the terminal device, after the terminal device executes the step of obtaining the current zoom factor, it also executes the following steps: if the current zoom factor is less than 1x focal length, then during the preview process, an ultra-wide-angle RAW image captured by the ultra-wide-angle camera is obtained; the ultra-wide-angle RAW image captured during the preview process is processed by the image processing front end; and the processed ultra-wide-angle RAW image is displayed during the preview process.

[0022] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the ultra-wide-angle YUV image and the color YUV image, and the steps of obtaining the target color image may be: aligning the sizes of the ultra-wide-angle YUV image and the color YUV image; obtaining the area that needs to be fused in the ultra-wide-angle YUV image and the area that needs to be fused in the color YUV image from the ultra-wide-angle YUV image after the size is aligned and the color YUV image after the size is aligned; mapping the pixels in the area that needs to be fused in the ultra-wide-angle YUV image with the pixels in the area that needs to be fused in the color YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; fusing the clear pixels in the area that needs to be fused in the ultra-wide-angle YUV image and the area that needs to be fused in the color YUV image to the corresponding blurred pixels.

[0023] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera, and the step of obtaining the target color image can be: if the current zoom factor is greater than or equal to 1 times the focal length and less than or equal to N times the focal length, after obtaining the black and white RAW image, the black and white RAW image is processed through the Bayer domain processing algorithm link to obtain a black and white YUV image; and the color RAW image is processed in the RAW domain to obtain a color YUV image; and the black and white YUV image is fused with the color YUV image to obtain the target color image.

[0024] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the above black and white YUV image with the above color YUV image, and the steps of obtaining the target color image may be: aligning the sizes of the color YUV image and the black and white YUV image; obtaining the area that needs to be fused in the color YUV image and the area that needs to be fused in the black and white YUV image from the color YUV image after the size is aligned and the black and white YUV image after the size is aligned; mapping the pixels in the area that needs to be fused in the color YUV image with the pixels in the area that needs to be fused in the black and white YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; fusion of the clear pixels in the area that needs to be fused in the color YUV image and the area that needs to be fused in the black and white YUV image to the corresponding blurred pixels.

[0025] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera, and the step of obtaining the target color image can be: if the current zoom factor is greater than N times the focal length, after obtaining the above telephoto RAW image, the above telephoto RAW image is processed through the Bayer domain processing algorithm link to obtain a telephoto YUV image; and the above color RAW image is processed to obtain a color YUV image; and the above telephoto YUV image is fused with the above color YUV image to obtain the target color image.

[0026] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device executes the fusion of the above telephoto YUV image and the above color YUV image, and the step of obtaining the target color image may be: aligning the sizes of the above telephoto YUV image and the above color YUV image; obtaining the area that needs to be fused in the telephoto YUV image and the area that needs to be fused in the color YUV image from the aligned telephoto YUV image and the aligned color YUV image; mapping the pixels in the area that needs to be fused in the telephoto YUV image with the pixels in the area that needs to be fused in the color YUV image to obtain the correspondence between the pixels; performing error detection on the above correspondence and correcting the erroneous correspondence; fusing the clear pixels in the area that needs to be fused in the telephoto YUV image and the area that needs to be fused in the color YUV image to the corresponding blurred pixels.

[0027] In one possible implementation, when the above instruction is executed by the terminal device, after the terminal device executes the step of obtaining the current zoom factor, it also executes the following steps: if the current zoom factor is greater than N times the focal length, then during the preview process, a color RAW image captured by the color camera is obtained; the color RAW image captured during the preview process is processed by the image processing front end; and the processed color RAW image is displayed during the above preview process.

[0028] In one possible implementation, when the above instruction is executed by the terminal device, the terminal device processes the color RAW image, and the step of obtaining the color YUV image can be: linearizing and bad pixel correction processing is performed on the color RAW image through the Bayer domain processing algorithm link, and RAW domain processing is performed on the color RAW image after linearization and bad pixel correction processing to obtain a color YUV image.

[0029] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0032] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0034] Figure 2 A flowchart of an image processing method for a terminal provided by an embodiment of the present application;

[0035] Figure 3 A schematic diagram of image fusion at different zoom factors provided by an embodiment of the present application;

[0036] Figure 4 A flowchart of an image processing method for a terminal provided by another embodiment of the present application;

[0037] Figure 5 A flowchart of an image processing method for a terminal provided in yet another embodiment of the present application;

[0038] Figure 6 A flowchart of an image processing method for a terminal provided in yet another embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a terminal device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0040] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0041] Currently, in the process of changing from ultra-wide angle to wide angle and to telephoto, the collaborative work between multiple cameras is not fully utilized, so that image details cannot be fully restored in some focal lengths. Based on the above problems, an embodiment of the present application provides an image processing method for a terminal. The terminal device fuses the color RAW image captured by the color camera with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor of the camera to obtain a target color image, which can enhance the detail effect of the image at different focal lengths, improve the overall clarity of the image, and improve the user experience.

[0042] The image processing method of the terminal provided in the embodiment of the present application can be applied to terminal devices, wherein the above-mentioned terminal devices can be smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks or personal digital assistants (PDA) and other devices; the embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0043] For example, Figure 1 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application is shown in FIG. Figure 1 As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0045] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0046] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0047] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0048] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0049] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (DCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0050] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0051] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0052] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0053] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0054] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0055] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0056] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0057] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device 100 through the power management module 141.

[0058] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0059] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0060] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0061] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0062] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0063] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0064] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0065] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0066] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0067] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0068] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0069] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include N cameras 193, where N is a positive integer greater than 1. In the embodiment of the present application, the N cameras 193 include a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera.

[0070] Specifically, during the shooting process, the user turns on the camera, and the light is transmitted to the photosensitive element through the lens (which may correspond to the camera 193 described previously). In other words, the lens can project the ambient light signal to the photosensitive area of ​​the photosensitive element, and the photosensitive element converts the light signal into an image visible to the naked eye through photoelectric conversion. The photosensitive element then transmits the internal original image (Bayer format, also called Bayer format or Bayer format) to the ISP module. The ISP module, after algorithm processing, outputs the image in the RGB spatial domain to the back-end acquisition unit, which is displayed in the image preview area of ​​the terminal device 100 or on the display screen of the terminal device 100. In this process, the processor controls the lens, photosensitive element and ISP module accordingly through the firmware program running thereon, thereby completing the image preview or shooting function.

[0071] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0072] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0073] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0074] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0075] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0076] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0077] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0078] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0079] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0080] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0081] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0082] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0083] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0084] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0085] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0086] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0087] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0088] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0089] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0090] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0091] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0092] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0093] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0094] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0095] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0096] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0097] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0098] For ease of understanding, the Bayer domain and the RAW domain mentioned in the following embodiments are first described.

[0099] 1. Bayer domain: Each lens on a digital camera has a light sensor to measure the brightness of light. However, to obtain a full-color image, three light sensors are generally required to obtain the information of the three primary colors of red, green, and blue. In order to reduce the cost and size of digital cameras, manufacturers usually use CCD or CMOS image sensors. Usually, the original image output by the CMOS image sensor is in the Bayer domain RGB format. A single pixel contains only one color value. To obtain the gray value of the image, it is necessary to first interpolate the color information of each pixel and then calculate the gray value of each pixel. In other words, Bayer domain refers to an original image format inside a digital camera.

[0100] 2. RAW domain: RAW domain images, that is, original images contain data processed from the image sensor of a digital camera, scanner or film scanner. It is named so because RAW domain images have not been processed, printed or used for editing. RAW domain images contain the most original information of the image and have not been processed nonlinearly in the ISP process.

[0101] The following embodiments of this application will be based on Figure 1 Taking the terminal device with the shown structure as an example, the image processing method of the terminal provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0102] Figure 2 A flowchart of an image processing method for a terminal provided by an embodiment of the present application is shown in FIG. Figure 2 As shown, the image processing method of the terminal may include:

[0103] Step 201: After the camera function of the terminal device 100 is running, the terminal device 100 obtains the current zoom factor.

[0104] Step 202, after the terminal device 100 obtains the shooting instruction, it responds to the above shooting instruction to obtain the color RAW image captured by the color camera, and obtains the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor.

[0105] Specifically, the RAW image captured by one of the black-and-white camera, the ultra-wide-angle camera and the telephoto camera is obtained according to the current zoom ratio: if the current zoom ratio is less than 1 times the focal length, the ultra-wide-angle RAW image captured by the ultra-wide-angle camera is obtained; if the current zoom ratio is greater than or equal to 1 times the focal length, and less than or equal to N times the focal length, the black-and-white RAW image captured by the black-and-white camera is obtained; if the current zoom ratio is greater than N times the focal length, the telephoto RAW image captured by the telephoto camera is obtained; wherein N is a positive number, and N>1.

[0106] In specific implementation, the size of N can be set according to system performance and / or implementation requirements. This embodiment does not limit the size of N. For example, N can be 1.8.

[0107] In step 203, the terminal device 100 fuses the color RAW image with a RAW image captured by one of the black and white camera, the ultra-wide-angle camera, and the telephoto camera to obtain a target color image.

[0108] Specifically, see Figure 3 , Figure 3 This is a schematic diagram of image fusion at different zoom factors provided by an embodiment of the present application. Figure 3 As shown, if the current zoom factor is less than 1 times the focal length, the terminal device 100 obtains a color RAW image captured by the color camera and an ultra-wide-angle RAW image captured by the ultra-wide-angle camera, and then fuses the color RAW image with the ultra-wide-angle RAW image, and uses the color image to fill in the details of the middle part of the ultra-wide-angle image to enhance the image quality of the ultra-wide-angle photo;

[0109] If the current zoom factor is greater than or equal to 1 times the focal length and less than or equal to N times the focal length, the terminal device 100 obtains a color RAW image captured by the color camera and a black-and-white RAW image captured by the black-and-white camera, and then fuses the color RAW image with the black-and-white RAW image, using the black-and-white image to supplement the picture details of the color image and enhance the image quality of the color image;

[0110] If the current zoom factor is greater than N times the focal length, the terminal device 100 obtains the color RAW image captured by the color camera and the telephoto RAW image captured by the telephoto camera, and then merges the color RAW image with the telephoto RAW image, using the color image to fill in the details of the middle part of the telephoto image to enhance the image quality of the telephoto image.

[0111] In specific implementation, at different zoom factors, the terminal device 100 may perform image fusion according to the strategy shown in Table 1.

[0112] Table 1

[0113]

[0114]

[0115]

[0116] In Table 1, W represents a color camera, i.e., the main camera; Mono represents a black-and-white camera; UW represents an ultra-wide-angle camera; and Tele represents a telephoto camera. It should be noted that the zoom multiples, equivalent focal lengths, module fusion schemes, preview stream module usage strategies, camera module combination strategies, scene categories, and / or image fusion strategies listed in Table 1 are all specific examples and are not intended to limit the embodiments of the present application.

[0117] In the image processing method of the above-mentioned terminal, after the camera function of the terminal device 100 is run, the terminal device 100 obtains the current zoom factor, and after obtaining the shooting instruction, in response to the above-mentioned shooting instruction, obtains the color RAW image captured by the color camera, and obtains the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor, and then fuses the above-mentioned color RAW image with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera to obtain the target color image, so that the terminal device 100 can fuse the color RAW image captured by the color camera with the RAW image captured by one of the black and white camera, the ultra-wide-angle camera and the telephoto camera according to the current zoom factor of the camera to obtain the target color image, so that in the scenario where the user adjusts the zoom factor, the detail effect of the image at different focal lengths can be improved, the overall clarity of the image can be improved, and the user experience can be improved.

[0118] Figure 4 This is a flowchart of an image processing method for a terminal provided in another embodiment of the present application. This embodiment illustrates the image fusion method when the zoom factor in Table 1 is [0.4x, 0.9x).

[0119] like Figure 4 As shown, this application Figure 2 In the illustrated embodiment, step 203 may include:

[0120] Step 401, if the current zoom factor is less than 1 focal length, after acquiring the ultra-wide-angle RAW image, the ultra-wide-angle RAW image is processed through the Bayer domain processing algorithm link to obtain an ultra-wide-angle YUV image; and the color RAW image is processed to obtain a color YUV image.

[0121] For details, see Figure 4 In step 401, after the ultra-wide-angle RAW image and the color RAW image are processed respectively by the Bayer domain processing algorithm link, a YUV domain enhancement algorithm, such as a multi-frame super-resolution (SR) algorithm, can be used to improve the clarity of each image while reducing noise.

[0122] Step 402: Merge the ultra-wide-angle YUV image with the color YUV image to obtain a target color image.

[0123] Specifically, the ultra-wide-angle YUV image and the color YUV image are merged to obtain the target color image as follows: the sizes of the ultra-wide-angle YUV image and the color YUV image are aligned; from the ultra-wide-angle YUV image after the size alignment and the color YUV image after the size alignment, the area to be merged in the ultra-wide-angle YUV image and the area to be merged in the color YUV image are obtained; the pixels in the area to be merged in the ultra-wide-angle YUV image are mapped with the pixels in the area to be merged in the color YUV image to obtain the correspondence between the pixels; error detection is performed on the above correspondence, and the erroneous correspondence is corrected; then, the clear pixels in the area to be merged in the ultra-wide-angle YUV image and the area to be merged in the color YUV image are merged to the corresponding blurred pixels.

[0124] Furthermore, after step 201, the following steps may also be included:

[0125] Step 403: If the current zoom factor is less than 1 focal length, then during the preview process, an ultra-wide-angle RAW image captured by the ultra-wide-angle camera is obtained.

[0126] Step 404: Process the ultra-wide-angle RAW image collected during the preview process through the image processing front end.

[0127] Step 405 , displaying the processed ultra-wide-angle RAW image during the preview process.

[0128] This embodiment can achieve the fusion of the color RAW image captured by the color camera and the ultra-wide-angle RAW image captured by the ultra-wide-angle camera when the zoom ratio is less than 1 times the focal length, so as to obtain the target color image, thereby improving the image quality obtained by shooting in the scenario where the user adjusts the zoom ratio and improving the user experience.

[0129] Figure 5 This is a flowchart of an image processing method for a terminal provided in yet another embodiment of the present application. This embodiment describes the image fusion method when the zoom factor in Table 1 is [1x, 1.8x].

[0130] like Figure 5 As shown, this application Figure 2 In the illustrated embodiment, step 203 may include:

[0131] Step 501, if the current zoom factor is greater than or equal to 1 times the focal length and less than or equal to N times the focal length, after acquiring the black and white RAW image, the black and white RAW image is processed through the Bayer domain processing algorithm link to obtain a black and white YUV image; and the color RAW image is processed in the RAW domain to obtain a color YUV image.

[0132] For details, see Figure 5 In step 501, after the black-and-white RAW image and the color RAW image are processed respectively by the Bayer domain processing algorithm link, a YUV domain enhancement algorithm, such as a multi-frame SR algorithm, can be used to improve the clarity of each image and reduce noise.

[0133] Step 502: merge the black-and-white YUV image with the color YUV image to obtain a target color image.

[0134] Specifically, the black-and-white YUV image and the color YUV image are merged to obtain the target color image as follows: the sizes of the color YUV image and the black-and-white YUV image are aligned; the area to be merged in the color YUV image and the area to be merged in the black-and-white YUV image are obtained from the color YUV image after the size is aligned and the black-and-white YUV image after the size is aligned; pixels in the area to be merged in the color YUV image are mapped to pixels in the area to be merged in the black-and-white YUV image to obtain a correspondence between the pixels; error detection is performed on the above correspondence, and the erroneous correspondence is corrected; finally, the clear pixels in the area to be merged in the color YUV image and the area to be merged in the black-and-white YUV image are merged to the corresponding blurred pixels.

[0135] This embodiment can achieve the fusion of the color RAW image captured by the color camera and the black-and-white RAW image captured by the black-and-white camera to obtain a target color image when the zoom ratio is greater than or equal to 1 times the focal length and less than or equal to N times the focal length. This can improve the image quality obtained by shooting in a scenario where the user adjusts the zoom ratio, thereby improving the user experience.

[0136] Figure 6 This is a flowchart of an image processing method for a terminal provided in yet another embodiment of the present application. This embodiment describes the image fusion method when the zoom factor in Table 1 is (1.8x, 3.5x).

[0137] like Figure 6 As shown, this application Figure 2 In the illustrated embodiment, step 203 may include:

[0138] Step 601, if the current zoom factor is greater than N times the focal length, after acquiring the telephoto RAW image, the telephoto RAW image is processed through the Bayer domain processing algorithm link to obtain a telephoto YUV image; and the color RAW image is processed to obtain a color YUV image.

[0139] For details, see Figure 6 In step 601, after the telephoto RAW image and the color RAW image are processed respectively by the Bayer domain processing algorithm link, a YUV domain enhancement algorithm, such as a multi-frame SR algorithm, can be used to improve the clarity of each image while reducing noise.

[0140] Step 602: Fusing the telephoto YUV image with the color YUV image to obtain a target color image.

[0141] Specifically, the telephoto YUV image and the color YUV image are merged to obtain the target color image as follows: aligning the sizes of the telephoto YUV image and the color YUV image; obtaining the area to be merged in the telephoto YUV image and the area to be merged in the color YUV image from the aligned telephoto YUV image and the aligned color YUV image; mapping the pixels in the area to be merged in the telephoto YUV image with the pixels in the area to be merged in the color YUV image to obtain the correspondence between the pixels; performing error detection on the correspondence and correcting the erroneous correspondence; and then, fusing the clear pixels in the area to be merged in the telephoto YUV image and the area to be merged in the color YUV image to the corresponding blurred pixels.

[0142] Furthermore, after step 201, the following steps may also be included:

[0143] Step 603: If the current zoom factor is greater than N times the focal length, a color RAW image captured by the color camera is obtained during the preview process.

[0144] Step 604: Process the color RAW image collected during the preview process through the image processing front end.

[0145] Step 605: Display the processed color RAW image in the above preview process.

[0146] This embodiment can achieve the fusion of the color RAW image captured by the color camera and the telephoto RAW image captured by the telephoto camera when the zoom ratio is greater than N times the focal length, so as to obtain the target color image, thereby improving the image quality obtained by shooting in the scenario where the user adjusts the zoom ratio and improving the user experience.

[0147] In addition, it should be noted that this application Figures 4 to 6In the illustrated embodiment, the color RAW image is processed to obtain a color YUV image by: linearizing and bad pixel correction processing is performed on the color RAW image through a Bayer domain processing algorithm link, and RAW domain processing is performed on the color RAW image after the linearization and bad pixel correction processing to obtain a color YUV image.

[0148] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0149] It is understandable that, in order to implement the above functions, the terminal device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0150] This embodiment can divide the terminal device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0151] Figure 7 A schematic diagram of the structure of a terminal device provided in another embodiment of the present application, in which each functional module is divided according to each function, Figure 7 A possible schematic diagram of the composition of the terminal device 700 involved in the above embodiment is shown. Figure 7 As shown, the terminal device 700 may include: a receiving unit 701, a processing unit 702 and a sending unit 703;

[0152] The processing unit 702 can be used to support the terminal device 700 to implement the present application. Figure 2 to Figure 6 The technical solution described in the illustrated embodiment.

[0153] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0154] The terminal device 700 provided in this embodiment is used to execute the above-mentioned terminal image processing method, and thus can achieve the same effect as the above-mentioned method.

[0155] It should be understood that the terminal device 700 may correspond to Figure 1 The terminal device 100 shown in FIG. The functions of the receiving unit 701 and the sending unit 703 can be Figure 1 The processor 110, antenna 1 and mobile communication module 150 in the terminal device 100 shown in the figure, and / or, are implemented by the processor 110, antenna 2 and wireless communication module 160; the function of the processing unit 702 can be implemented by Figure 1 The processor 110 in the terminal device 100 is shown to be implemented.

[0156] In the case of adopting an integrated unit, the terminal device 700 may include a processing module, a storage module and a communication module.

[0157] The processing module can be used to control and manage the actions of the terminal device 700, for example, it can be used to support the terminal device 700 to execute the steps executed by the above-mentioned receiving unit 701, processing unit 702 and sending unit 703. The storage module can be used to support the terminal device 700 to store program codes and data, etc. The communication module can be used to support the communication between the terminal device 700 and other devices.

[0158] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0159] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 700 involved in this embodiment can be a Figure 1 Device of the structure shown.

[0160] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 2 to Figure 5 The method provided by the illustrated embodiment.

[0161] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 2 to Figure 5The method provided by the illustrated embodiment.

[0162] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0163] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0166] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is [0.4x, 0.9x) and the equivalent focal length is 11-23 mm: The first image is an image captured by the ultra-wide-angle camera based on a binning mode; In a scenario where the sensitivity ISO is 200≤≤2000, the target color image is an image acquired by performing dual-channel RAW domain multi-frame fusion or YUV domain fusion of an image acquired by the ultra-wide-angle camera based on a binning mode and an image acquired by the color camera based on a Stagger HDR mode; In a macro photography scene, a high-brightness scene or a low-light scene, the target color image is an image acquired by the ultra-wide-angle camera based on a binning mode, and is obtained by performing single-channel RAW domain multi-frame fusion.

2. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom factor is [0.9x, 1x) and the equivalent focal length is 23-27 mm: The first image is an image captured by the color camera based on the Stagger HDR mode; In the full scene, the target color image is an image obtained by performing single-channel RAW domain multi-frame fusion on the image collected by the color camera based on the Stagger HDR mode.

3. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is [1x, 1.8x] and the equivalent focal length is 27-48 mm: The first image is an image captured by the color camera based on the Stagger HDR mode; In the full scene, the target color image is an image acquired by the color camera based on the Stagger HDR mode and processed by the artificial intelligence AI RAW and an image acquired by the black and white camera and processed by the hardware multi-frame noise reduction technology MFNR, and the four-frame image fusion is performed; or In some non-motion scenes, the target color image is the dual-channel restart and stop flow of the color camera and the black and white camera. After a pause of 500-1000ms, the image acquired by the color camera based on the binning mode and the image acquired by the black and white camera based on the 64M single frame mode are fused.

4. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is (1.8x, 3.5x) and the equivalent focal length is 48-95 mm: The first image is an image captured by the color camera based on the Stagger HDR mode; In the scenario of ISO≤3200, the target color image is an image obtained by performing RAW domain multi-frame FOV fusion of the image collected by the color camera based on the Quad mode and the image collected by the telephoto camera based on the binning mode after the color camera restarts and stops streaming for 400ms; In a scenario where ISO>3200, the target color image is an image acquired by the color camera based on the Stagger HDR mode, and is obtained by performing RAW domain multi-frame FOV fusion.

5. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is [3.5x, 7x) and the equivalent focal length is 95-190 mm: In a bright light scene, the first image is an image captured by the telephoto camera based on a binning mode; the target color image is an image captured by the telephoto camera based on a binning mode, and is obtained by performing single-channel Tele RAW domain multi-frame fusion; In a dark scene or a scene with a shooting distance less than 1200 cm, the first image is an image captured by the color camera based on the Stagger HDR mode; the target color image is an image captured by the color camera based on the Stagger HDR mode, and is obtained by performing multi-frame fusion of a single main camera RAW domain.

6. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is [7x, 10x) and the equivalent focal length is 190-270 mm: In a high-brightness scene, the first image is an image captured by the telephoto camera based on the remosaic crop mode; the target color image is an image obtained by performing Tele RAW domain multi-frame fusion on the image captured based on the Quad mode after the telephoto camera restarts and stops streaming for 400 milliseconds; In a medium-brightness scene, the first image is an image captured by the telephoto camera based on a binning mode; the target color image is an image captured by the telephoto camera based on a binning mode, and obtained by performing Tele RAW domain multi-frame fusion; In a dark scene or a scene with a shooting distance less than 1200 cm, the first image is an image captured by the color camera based on the Stagger HDR mode; the target color image is an image captured by the color camera based on the Stagger HDR mode, and is obtained by performing multi-frame fusion of the main camera RAW domain.

7. An image processing method, characterized in that: Applied to a terminal device, the terminal device includes a black and white camera, a color camera, an ultra-wide-angle camera, and a telephoto camera, and the method includes: In response to a camera function execution instruction, obtaining a current zoom factor; Display a preview interface, wherein the preview interface includes a first image; In response to a shooting instruction, obtaining a target color image; When the zoom ratio is greater than 10 times the focal length and the equivalent focal length is greater than 270 mm: In a non-dark light scene, the first image is an image captured by the telephoto camera based on the remosaic crop mode; the target color image is an image obtained by performing Tele RAW domain multi-frame fusion on the image captured based on the Quad mode after the telephoto camera restarts and stops streaming for 400 milliseconds; In a dark light scene, the first image is an image captured by the telephoto camera based on a binning mode; the target color image is an image captured by the telephoto camera based on a binning mode, and is obtained by performing Tele RAW domain multi-frame fusion.

8. A terminal device, characterized in that: It includes a black and white camera, a color camera, an ultra-wide-angle camera and a telephoto camera; one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer program is executed by the terminal device, the terminal device executes the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 7.

Images